Roadmap: pandas SparseDataFrame may be deprecated

[justmarkham]

I noticed in the new roadmap that the second item on the list relates to "Pandas DataFrames and SparseDataFrames". I wanted to mention that according to an August 2018 talk by @datapythonista (who is a pandas core contributor), SparseDataFrames will be deprecated. Here are the slides from the talk (in notebook form).

I mentioned this to @amueller on Twitter, and he suggested opening an issue to document this.

This PR seems to be the most up-to-date discussion among the pandas team about deprecation.

Hope this is helpful!

[justmarkham]

Additional information, that I just noticed on Twitter from @jorisvandenbossche (another pandas core contributor):

Small clarification: for now, DataFrames with sparse data will not be deprecated, but we are considering to deprecate the SparseDataFrame subclass (because a normal DataFrame can also hold sparse data) https://mail.python.org/pipermail/pandas-dev/2018-November/000855.html

[amueller]

Thanks for the input.
So it looks like they might

Deprecate SparseDataFrame in favor of a DataFrame holding sparse arrays

So maybe we should change it to "DataFrames for dense and sparse data" to accommodate what ever solution they are going for

[TomAugspurger]

Just to note: the memory usage of the old SparseDataFrame and the "new" (it's been possible for a while) DataFrame where some of the columns are sparse is identical.

And, I don't think either are that good for very wide matrices. Each column is stored independently as two arrays: One array for the "sparse index" (the positions where the value is explicitly) and the "sparse values" (the value at each sparse index position).

[jnothman]

That's essentially equivalent to csc which is still much better than dense. Yes, update the wording to say that we need to directly support DataFrame sparsity

[mitar]

So if one would pass into scikit-learn method which accepts "sparse array" as input a DataFrame with SparseSeries, would that work or explode at the moment?

[jnothman]

Check? I assume it would work but inefficiently by making dense, unless pandas raises an error when converting to array

[jorisvandenbossche]

A check_array on a (sparse) DataFrame will convert it to a numpy array (__array__), which for a sparse DataFrame means it gets densified, and memory usage will increase.
You can convert your sparse DataFrame to a scipy sparse matrix, which is probably the best approach for now if you want to keep it sparse.

[mitar]

So for me the beauty of using DataFrame is that I can have both string columns and numerical columns and throw it at random forest and it just works.

Now, with some columns sparse, we cannot just convert it to scipy sparse matrix, because the latter does not support object dtype.

What would be the ideal implementation here to support this properly in scikit-learn?

[jnothman]

Well at the moment I wouldn't have thought that throwing categorical dtypes at our random forest "just works" either.

[amueller]

I agree that we should make it a scipy sparse matrix. Should densifying be considered a bug? also cc @thomasjpfan as a potential todo.

[mitar]

So currently sparse Pandas DataFrames are supported only if all columns are sparse. What about supporting mixed case where dense columns could be converted to sparse ones?

[thomasjpfan]

What kind of behavior would be reasonable for the mixed case?

[mitar]

Making dense columns into sparse columns, I would say, and then everything into sparse?

[jnothman]

Choosing whichever has a smaller memory representation?

[mitar]

That would be ideal, but it might be hard to know in advance?

[TomAugspurger]

I don't think trying to guess here is a good idea. I would direct the user to process the dense and sparse subsets independently using a ColumnTrasformer. Or if they really want them processed together then the user should convert the dense to sparse ahead of time.