move DataFrame type definition to lightweight package

[piever]

See discussion starting from this comment. The key issue is that some CSV reading packages like to have a DataFrame as a default output: it is probably the most popular tabular data format in Julia and is preferrable to the "typed alternatives" when the data is wide (NamedTuple based alternatives have poor performance when the number of columns is in the thousands). In practice, both CSV.jl and TableReader.jl have a dependency on DataFrames which they only use so that they can output a DataFrame. Unfortunately, this causes a slowdown when loading those packages due to the complexity of the code of DataFrames. It also makes those packages less appealing as dependencies for tabular data packages that rely on alternative formats (like IndexedTables, StructArrays or TypedTables).

A proposed solution is to split the DataFrame definition and constructors to a DataFramesBase packages which would have basically zero dependencies and would be very fast to load, so that CSV and TableReader can just depend on DataFramesBase. DataFrames would then reexport DataFramesBase.DataFrame so this change would not affect users.

As an added bonus, I suspect this may make it more appealing to port IndexedTables from their current "mutable table" structure (a dictionary of columns) to just using a DataFrame: converting IndexedTable to DataFrame, replacing a few columns in various ways and converting back to IndexedTable is a useful workflow.

[KristofferC]

Methods that overload Base functions and other extend functions from other packages would probably need to move to the new lightweight package as well (type piracy).

[bkamins]

I am OK to go this way, but could you be more specific what you expect that can be stripped?

I am asking, because I expect that most of the things will have to stay anyway. The majority of dependencies are needed by various DataFrame constructors.

[quinnj]

Yes, I'm in favor of both cleaning up DataFrames dependency (I think the StatsBase could be inside a Requires.jl block, for example) and moving the core type definition + constructor to a separate package (DataFrameTypes.jl?), I guess we'd need AbstractDataFrame and Index as well, but that doesn't seem too bad.

[JeffBezanson]

The majority of dependencies are needed by various DataFrame constructors.

I think it's fine to have some constructors defined in DataFramesBase and some in DataFrames, if that helps.

[nalimilan]

I see the reasoning, but wouldn't it be weird if CSV parsers returned a DataFrame object without providing most functions that you can call on it? That sounds confusing. I'd rather have CSV parsers use the Tables.jl interface to return a custom iterable like CSV.File, so that they no longer depend on DataFrames (they don't even need to depend on Tables.jl as they just need to return an iterable). Users would have to call DataFrame(CSV.File(path)) instead of CSV(path), which I kind of regret, but at least it would be clear and fully generic.

We could probably get rid of a few dependencies like CategoricalArrays and PooledArrays by using Requires.jl, but apparently it currently incurs a significant slowdown at load time.

[bkamins]

We could probably get rid of a few dependencies like CategoricalArrays and PooledArrays

Actually most of the dependencies can be lazily loaded and most of them will be never loaded in a normal workflow. Maybe this is the way to go?.

[nalimilan]

One issue is that AFAIK we cannot load CategoricalArrays automatically when calling categorical!.

[JeffBezanson]

wouldn't it be weird if CSV parsers returned a DataFrame object without providing most functions that you can call on it?

Depends how we define "most" :) I think it is highly valuable to have a standard data structure like Array for this. Huge numbers of functions over the julia ecosystem can work on Arrays, but you only need a handful to get started, and DataFrame should be similar. Perhaps it needs a few more functions. But not a huge number.

I hate to say this, but categorical! is not a crucial tabular data operation. You can analyze data without it. A DataFramesBase that lacks it and some other things could still be perfectly useful.

[nalimilan]

categorical! and other data management functions could live in a different package. But you also evoked constructors, and I was thinking about indexing, views, printing... where do you put the limit?

EDIT: Also we rely on Tables.allocatecolumn for essential things such as vcat. Since Tables.jl is slow to load due to Requires.jl, we would also have to drop vcat from the basic package. At that point it would be quite limited...

[JeffBezanson]

I'd be happy to pick a place to draw the line. We do that in API design all the time.

I also humbly submit that if something as byzantine as Requires.jl is needed to concatenate tables, we are doing something wrong.

[bkamins]

Let me write down the DataFrames.jl dependency list and where they are used (if I have missed something please add here):

• CategoricalArrays: constructor, categorical array handling, split-apply-combine
• DataStreams: dependency will be removed soon by @quinnj
• IteratorInterfaceExtensions: constructor
• Missings: used in functions handling missing values
• PooledArrays: split-apply-combine
• Reexport, Compat - standard utility usage
• SortingAlgorithms - sorting
• StatsBase: describe method extends function defined there
• Tables: constructor
• TableTraits: constructor
• WeakRefStrings: dependency will be removed soon by @quinnj

[piever]

but wouldn't it be weird if CSV parsers returned a DataFrame object without providing most functions that you can call on it?

I am a bit confused by this point. I imagine most users do using DataFrames, CSV anyway as CSV does not reexport DataFrames functionality (RDatasets seems to be the odd one out here in that it also reexports DataFrames). Returning a DataFrame without having to load a lot of code is to me a good compromise as:

• Users of DataFrames keep doing using DataFrames, CSV or using DataFrames, TableReader and are completely unaffacted by this
• Users of TypedTables or IndexedTables can use TableReader just by doing TypedTable(TableReader.readcsv(df)) or table(TableReader.readcsv(df)) and no longer have to wait the extra load time (or dependencies, or potential, even though very unlikely, breakage) of the largish DataFrames codebase.

OTOH I appreciate that splitting the package without type piracy may be tricky. I imagine that the things to take out of DataFramesBase for sure are the functions defined in DataFrames that are not imported from elsewhere (groupby, stack, unstack, categorical!) which hopefully gets rid of the PooledArrays extension.

I'm not sure that IteratorInterfaceExtensions and TableTraits constructors are needed as direct dependencies now that Tables "incorporates them". Sorting is a bit trickier though: I've had similar difficulties with StructArrays: it was difficult to define efficient sorting by multiple columns without extra dependencies (and in the end I needed PooledArrays).

[nalimilan]

@piever even if functions are usually not reexported, methods are implicitly (e.g. getindex). So to avoid weird MethodErrors about getindex we would have to move all definitions of methods defined in packages other than DataFrames to the new lightweight package.

Do we actually know what makes loading slow? Is it DataFrames per se, one of more dependencies, or the fact that Tables.jl uses Requires? I think we should experiment with this. For example, Tables.allocatecolumn is arguably so generic that it could leave out of Tables.jl (it's actually a variant of similar but taking an entry instead of an array), which would allow getting rid of its use of Requires.jl.

Users of TypedTables or IndexedTables can use TableReader just by doing TypedTable(TableReader.readcsv(df)) or table(TableReader.readcsv(df)) and no longer have to wait the extra load time (or dependencies, or potential, even though very unlikely, breakage) of the largish DataFrames codebase.

This was precisely the goal of defining a generic interface in Tables.jl. In an ideal world I would like CSV and TableReaders to return a DataFrame by default, but also provide a way to create any table without creating a DataFrame first. The latter is possible using CSV.File, but unfortunately we don't have a mechanism currently to have CSV.read return a DataFrame but only load DataFrames when that function is called.

Therefore an argument for not splitting DataFrames is that it will annoy @JeffBezanson so much that he will implement a way to avoid that (e.g. that's possible in R, though I guess it may be more difficult in Julia). :-) In the meantime, as I suggested above, I would just require people to write DataFrame(CSV.File(...)) and leave DataFrames alone.