Common pool for methods as a way to solve common names in different packages

[tshort]

DataFrames exports many methods that use common names, like cut, by, in, index,matrix, range, and with. If another package comes along and defines and exports their own methods of the same name, the user can no longer just type the method name and expect it to work. They'll have to preface the function with the module name.

@johnmyleswhite describes the issue well here:

https://groups.google.com/d/msg/julia-stats/zHFIOdQp664/dvohHm54Tm8J

@JeffBezanson's two cents:

https://groups.google.com/d/msg/julia-stats/zHFIOdQp664/I9Vu68prr-UJ

Maybe we could have a new keyword called shared that is something like import, but it uses a module called Common as the pool of common method definitions. Another feature that would be nice is to keep Common.fun1 separate from MyModule.fun1 in case there are overlaps. That might complicate things too much, though.

[pao]

Crazier idea: share to a namespace (represented by a metapackage?) that semantically-equivalent packages could share. That gives some granularity to the sharing mechanism. Possibly either too crazy or overkill.

[nolta]

This might be off-topic, but the hierarchical module idea is interesting. Instead of

module A
  # implicit 'using Base'
  import Base.x, Base.y
end

baremodule B
   ...
end

you could write something like

module A <: Base
  import x, y
end

module B
  ...
end

[StefanKarpinski]

Using the <: subtype symbol is a non-starter, but syntax aside, it's certainly an interesting idea.

[JeffBezanson]

It is an interesting idea, but having a near-obligatory <: Base puts us back in the using Base days that nobody misses.

[pao]

If @nolta's proposal was meant as a response to my comment, my crazy idea isn't quite the same. I'm wondering if instead of a single common namespace pool, we could have an arbitrarily large number of namespace pools which would emerge by domain: instead of export foo you might export_to Finance foo, which would in turn refer to a sort of "virtual module" which multiple modules could share and could be used as a unit.

[toivoh]

I also like the idea of common namespace pools, but I think that the solution is primarily about how to reach consensus about what a given generic function means, not about adding new language features. If you use a common name without prior consensus, you are bound to run into trouble sooner or later.

Consider two modules A and B that both want to provide methods for the same generic function f.
Here is the way that I see it:

• Ideally, the generic function f corresponds to a concept. (Plus, expectations on how f should be invoked.)
• If A and B both want to provide methods to f, they need to reach consensus on what f means.
• One way to reach consensus is for A and B to depend on a common definition of f, in some module.
• If A and B do have consensus on the meaning of f, that actually implies a common dependence on that consensus. (And so they might as well encode it as a common dependence.)

If B depends on A, then the common definition is naturally A.f. If neither depends on the other, I argue that they should create a module C to house f, that both will depend upon. There must be some term for this kind of pattern in object oriented design, but I'm not sure of what it is.

I see the consensus process something like this:

• Packages are developed.
• At some point, someone realizes that function fa in module A stands for the same concept as function fb in module B.
• The creators of A and B, and perhaps also others, talk about this, and agree upon a common definition f.
• They put f somewhere before A and B in the dependency chain, probably as early as they have mandate for.

This way, common concepts will gradually merge into common generic functions, percolating down the dependency graph towards Base itself.

The immediate need seems to be somewhere to house common generic functions that don't belong in Base itself, since Base will not provide any methods for them. I think that a good start would be a Concepts module, either in a separate Concepts package, or as Base.Concepts. Since the concepts module might not even provide any kind of implementation, it seems vital to me to document there which concept the function is meant to embody, and how.

Some language support will also be helpful:

• A way to create a generic function without adding any method.
• Don't warn if an imported binding becomes shadowed by another binding to the same generic function/object.

[pao]

Thanks, that explains things much better, though the focus on a Concepts package/module is roughly equivalent to the original proposal in that we're back to a flat namespace for those names. I accidentally think I previewed an answer which requires no new language features and is immediately available. It's pretty much that, but it can be done with a domain-specific package that provides an empty implementation that can be imported from.

The difference between that and my previous comment is the formality of the relationship.

[johnmyleswhite]

The one concern I have with these proposals is discoverability: suppose that I fumble around and develop a function called X. How long will it be before I realize that my X is called Y in the Concepts1235 package?

[pao]

If you are calling what everyone else is calling "Y" "X", there is no discoverability except by discussion with your peers. I'm not sure how any of these proposals, even the simplest name pool one, can fix that.

[tshort]

Hopefully, common users of packages will alert both maintainers of similar
methods with different names. Common users can also alert maintainers about
definition conflicts, or more likely warnings about method ambiguities.

[toivoh]

Sorry for not giving credit where credit is due. I didn't see my post as a new proposal, but as trying to flesh some of the ideas that came before.

About discoverability: I also don't think that this is worse than anything else that we could come up with. (But please write if you come up with ways that work better in this respect.) I didn't intend that you should create the concepts to sit idly in the dark until someone would realize that they should implement them; a concept wouldn't be added until there were at least two uses. So you would at least have the chance to realize that you wanted the same concept as one of those. Hopefully the common concepts should also become documented sooner or later.

Also, I think that the same concept could very well be imported from the point of common definition into dependent concepts modules. This would be natural if a concept was first created in one context, and then became more widely adopted. By importing appropriate concepts into more specific concepts modules (if there would be several), it might be easier to find the concept you are looking for when looking in the place where you think it would be.

[JeffBezanson]

This is definitely a real problem, but triage thinks we won't implement this specific solution. The leading candidate solution for this kind of thing is method merging, and/or to continue to make packages like StatsBase as a home for shared names.