Reconsider overloading Base.join?

[KristofferC]

Base.join has a clear docstring which is to join an array of strings with delimiters. This seems like a different operation than what the join on DataFrames provides. Since the operation is different, it shouldn't share the namespace (DataFrames.join shouldn't extend Base.join)

[bkamins]

Indeed this is not very clean.

The problem we have is that the operation we do in DataFrames.jl is universally called join. For sure we do not want to force users to write DataFrames.join.

I think we could do two things:

• leave the situation as is (knowing it is not ideal, but for years people used it and never complained);
• maybe change the name to jointables and make it live in DataAPI.jl or something similar?

I am not sure what would be better. Also adding @piever to the discussion as JuliaDB.jl also extends Base.join for exactly the same purpose as DataFrames.jl does.

[ararslan]

FWIW I don't see an issue with the current terminology/extension. join(df1, df2) is so natural that I doubt there would be any confusion, and even if there is, people can just check the docstring for join to read about the method for DataFrames.

[ararslan]

(I'm also not a fan of the idea that if someone has used a word as a function name to mean a particular thing, that no one else can use that word to mean anything else. It's just a word, that's what documentation is for.)

[KristofferC]

(I'm also not a fan of the idea that if someone has used a word as a function name to mean a particular thing, that no one else can use that word to mean anything else. It's just a word, that's what documentation is for.)

You seem to be confused about how generic programming works. Someone creates a function in a namespace. That function has a certain meaning (which is indicated by the docstring) envisioned by the author who created the function. For example, Base.push!(a, b) means to add the element b to the collection a. It does not mean that b gives a a physical push (like in a game). Other types can extend (overload) this function by creating methods Base.push(::MyType, ...) =. The meaning of the function is still the same though. This is important so that when you e.g. see untyped code like

function func(x, a)
    for i in foo(x)
        push!(a, i)
    end
end

you have some clue what it does and what the meaning of it is.

If a game want to use push! to actually mean that a player gives another player a push that is done by creating a new function push! (that does not extend Base.push!). This function has it's own docstring. People reading code can from looking at the namespace of push! know which function is used and code makes sense.

This is not unique to Julia, for example from the C++ code guidelines (https://google.github.io/styleguide/cppguide.html#Function_Overloading):

Use overloaded functions (including constructors) only if a reader looking at a call site can get a good idea of what is happening without having to first figure out exactly which overload is being called.

In other words, you shouldn't need to know which method of Base.push! that gets called to know sort of what the functionality of the code is. That is why you can give docstrings to function at all. To convey their meaning because they are more than "a name".

[ararslan]

You seem to be confused about how generic programming works.

Perhaps I didn't convey my point accurately but this is pretty rude.

I don't disagree with most of the points you've made. My point is that there are some operations with names that are sufficiently generic that they mean different things in different contexts anyway, like join. No one is going to be reading tabular data manipulation code and see join(df1, df2) and assume that df1 is a vector of strings and df2 is a separator. The reuse of the name is of very little practical consequence, and requiring a module qualification is just annoying for the user.

[KristofferC]

My point is that there are some operations with names that are sufficiently generic that they mean different things in different contexts anyway, like join.

Sure, but those things should then not extend the same function.

However, I do think that perhaps

• leave the situation as is (knowing it is not ideal, but for years people used it and never complained);

might be the best thing from a pragmatic point of view.

[pdeffebach]

We recently decided that a new flatten function for DataFrames would not be an extension of Iterators.flatten and instead a flatten should live in DataAPI.

So there is precedent for this. Making join in DataAPI seems natural in this case.

[bkamins]

The difference is that flatten is not in Base, so in normal workflows you do not have to write DataAPI.flatten but just flatten in the code. While join is in Base, so you would always have to write DataFrames.join when you would want to do a join of two data frames.

[KristofferC]

While join is in Base, so you would always have to write DataFrames.join when you would want to do a join of two data frames.

Unless you du using DataFrames: join I guess?

If only there was a way to namespace things based on the object :P

[bkamins]

This will not solve a problem:

julia> join(["1", "2"])
"12"

julia> module A
       join(::Float64) = 0.0
       end
Main.A

julia> using Main.A: join
WARNING: ignoring conflicting import of A.join into Main

and if you reverse the order of calls you get:

julia> module A
       join(::Float64) = 0.0
       end
Main.A

julia> using Main.A: join

julia> join(["1", "2"])
ERROR: MethodError: no method matching join(::Array{String,1})

[pdeffebach]

requiring people to use DataFrames.join would be frustrating. I would bet most users (or, desired audience) have never encountered a namespace error before.

[piever]

While I agree that from a purist's point of view Base.join should not be overloaded for some very different behavior, I would tend to agree that having to qualify it with the package name would be very annoying. OTOH I would find an exported function with a different name, like datajoin or jointables as proposed by @bkamins, less annoying.

The only thing I wanted to point out, though, is that eventually it'd be nice to be able to define fallbacks for data manipulations for all table structures (things that respect the Tables.jl interface). With that in mind, it is tricky that one of these operations is an extended Base function, because then it would be impossible to extend join to arbitrary tables (no common supertype). In the case of other Base functions (e.g. map or filter) that are also useful on tables, this is less of an issue as the Base fallback already does a sensible thing, e.g. map(f, Tables.rows(sometable)) would return the correct table.

On a related note, JuliaDB uses broadcast(merge, t1, t2) as a synonym (with some differences) of join, with the idea that primary keys are indices of the table, so broadcasting should make a join (and merge merges the rows, which are named tuples, so that the output has columns from both inputs), but this is very specific to keyed tables and may not make sense for DataFrames.

[CameronBieganek]

How about splitting join into separate functions like innerjoin and leftjoin? This solves the overloading problem and would be consistent with both SQL and dplyr.