Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

concat([sparse]) should be a Series / DataFrame #25702

Closed
TomAugspurger opened this issue Mar 13, 2019 · 3 comments

Comments

@TomAugspurger
Copy link
Contributor

commented Mar 13, 2019

I messed up the logic somewhere along the way.

In [11]: df = pd.DataFrame({"A": pd.SparseArray([0, 1])})

In [12]: type(pd.concat([df, df]))
Out[12]: pandas.core.sparse.frame.SparseDataFrame

That should be a DataFrame with sparse values.

When should concat return a SparseSeries/Frame?

a.) When any of the inputs are SparseSeries / SparseDataFrame
b.) When all of the inputs are SparseSeries/Frame.

Option b is probably least disruptive. Implementing it isn't a pain.

@TomAugspurger TomAugspurger added this to the 0.25.0 milestone Mar 13, 2019

@WillAyd

This comment has been minimized.

Copy link
Member

commented Mar 13, 2019

What's the downside to option a? I admittedly don't use these objects a lot but to cast a vote in there that seems more intuitive to me

@TomAugspurger

This comment has been minimized.

Copy link
Contributor Author

commented Mar 13, 2019

@TomAugspurger

This comment has been minimized.

Copy link
Contributor Author

commented Mar 13, 2019

Note that this only affects DataFrame. concat(List[Series[Sparse]]) has the desired behavior of returning a Series with sparse values.

In [4]: type(pd.concat([pd.Series(pd.SparseArray([0, 1]))] * 2))
Out[4]: pandas.core.series.Series

TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Mar 13, 2019

API: concat on sparse values
API breaking change to `concat(List[DataFrame[Sparse]])` to
return a DataFrame with sparse values, rather than a SparseDataFrame.

Doing an outright break, rather than deprecation, because I have a
followup PR deprecating SparseDataFrame. We return this internally in
a few places (e.g. get_dummies on all-sparse data).

Closes pandas-dev#25702

@TomAugspurger TomAugspurger referenced this issue Mar 13, 2019

Merged

API: concat on sparse values #25719

0 of 4 tasks complete

TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Mar 15, 2019

Squashed commit of the following:
commit 96508ca
Author: Tom Augspurger <tom.w.augspurger@gmail.com>
Date:   Wed Mar 13 08:23:21 2019 -0500

    API: concat on sparse values

    API breaking change to `concat(List[DataFrame[Sparse]])` to
    return a DataFrame with sparse values, rather than a SparseDataFrame.

    Doing an outright break, rather than deprecation, because I have a
    followup PR deprecating SparseDataFrame. We return this internally in
    a few places (e.g. get_dummies on all-sparse data).

    Closes pandas-dev#25702

jreback added a commit that referenced this issue Mar 19, 2019

API: concat on sparse values (#25719)
API breaking change to `concat(List[DataFrame[Sparse]])` to
return a DataFrame with sparse values, rather than a SparseDataFrame.

Doing an outright break, rather than deprecation, because I have a
followup PR deprecating SparseDataFrame. We return this internally in
a few places (e.g. get_dummies on all-sparse data).

Closes #25702
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.