Is there a Tables.jl equivalent of `DataFrames.jl`'s `view(df, rows, :)`

[xiaodaigh]

Does Tables.jl define an interface for creating views? I think df[rows, :] is DataFrames.jl syntax. Wondering what the Table's equivalent is

[ablaom]

I'll let the experts answer this one. However, if you are using MLJ (which I suspect is the source of your question) you can use the methods MLJBase.selectrows, MLJBase.selectcols and MLJ.select for accessing arbitrary tables by index (query the docstrings for details). However, the table returned is not generally of the same type because some tables can only be used as a source and not a sink (eg, CSV file). Furthermore, the returned table is not in general a view; the amount of copying that goes on, and the speed, depends on the particular table being accessed, as I understand it.

Alternatively, you may want to just convert your incoming tabular data to a matrix with Tables.matrix or MLJ.matrix (essentially the same). This is the more common pattern.

I think there is a place for a more specialised tabular interface for "in-memory" tables that supports random row access, sparsity in rows, and views, but I am not aware of one.

[xiaodaigh]

Thanks @ablaom I like tighter integration with MLJ, but I think there are things that there are always more specialised things I want to do with JLBoost.jl than is readily available in MLJ. I think having view is quite important for performance. I benchmarked JLBoost vs XGBoost.jl and for small datasets, the timing is small but is around 20x (it's like 0.05 vs 1s). I think want to open up JLBoost to MLJ but I wish to provided specialised support for tables that has views.

I think Tables.jl should provide some view support? Tables.rowviewaccess(::T) = true/false type of interface similiar to row and columns?

Also, at some point, I want to make MLJJLBoost.jl a separate package so I keep the dependencies to a minimum.

[joshday]

I'm not sure views make sense in Tables, which is based on iteration. The data may not "exist" yet.

[xiaodaigh]

If views don't make sense for tables then we can close it. But I think somewhere in the data ecosystem there should be an interface that supports views for tables that can. Also if Tables.jl supports only the iteration case then TableIterator.jl or. Something like that would be clearer

[nalimilan]

Getting a view could work for DataFrames and NamedTuple{<:Vector} at least. Maybe there could be a trait as you suggested, and a Tables.view function (as we cannot overload Base.view for NamedTuple).

[quinnj]

This has been brought up a couple of times in various contexts; I think a bunch of us could brainstorm an InMemoryTable interface, which could be implemented by various in-memory tables w/o dependent packages needing to commit to a specific representation.

Personally, I've love the amount of thought, work, and design that has gone into the current DataFrames interface, and I think a lot of it could be abstracted into a kind of TableIndexing.jl interface package. That would include the column selecting API, cell/row/column indexing, etc.

As for this current issue for Tables.jl: Tables.jl is meant as a very generic interface to allow integration of any kind of table-like data source/sink, not just in-memory. I'm closing for now, but happy to keep the discussion for a Tables.jl-interface subset interface going here or elsewhere.

[xiaodaigh]

think a bunch of us could brainstorm an InMemoryTable interface, which could be implemented by various in-memory tables

I have an idea about how to support view for JDF.JDFFile and I think I can emulate the DataFrames approach by lazy loading from disk. So I would love another subset to Tables.jl. I like

TableIndexing.jl

idea, because I can just hook JDF into that. Currently, JDF.jl implements a subset of df[rows, cols] so it's syntax compatible with DataFrames.jl. Would love an official light-weight interface to enable more collaboration and innovation!

[juliohm]

Fully support the idea of a TableIndexing.jl interface to solve once and for all this issue of trying to take views of tables and not getting what is expected. Supporting a general df[rows,cols] would be great.

Is this discussion continued somewhere else? Anyone started a package with this idea?

[nalimilan]

I don't think anybody has worked on this. I guess you could take inspiration from what DataFrames does.