BUG: Sparse concat may fill fill_value with NaN

[sinhrks]

• closes concat erroneously sets series to NaN #9765

• tests added / passed

• passes git diff upstream/master | flake8 --diff

• whatsnew entry

  This also added tests related to concat() on Sparse dataframe returns strange results #12174 and fixed concat(axis=1) issue.

# on current master
dense1 = pd.DataFrame({'A': [1, 2, 3, np.nan],
                    'B': [0, 0, 0, 0],
                    'C': [np.nan, np.nan, np.nan, np.nan],
                    'D': [1, 2, 3, 4]})
dense2 = pd.DataFrame({'E': [1, 2, 3, np.nan],
                    'F': [0, 0, 0, 0],
                    'G': [np.nan, np.nan, np.nan, np.nan],
                    'H': [1, 2, 3, 4]})
sparse1 = dense1.to_sparse()
sparse2 = dense2.to_sparse()
pd.concat([sparse2, sparse1], axis=1)
# AttributeError: 'int' object has no attribute 'ravel'

One point to be discussed is the logic for return type. Currently, SparseDataFrame is returned only when all blocks are all sparse. Because SparseDataFrame can't work properly if dense block is contained.

Thus, dense and sparse concat with axis=0 resunts in SparseDataFrame, and axis=1 results in normal DataFrame.

[jreback]

I think we could return SparseDataFrame if any blocks are sparse.

[sinhrks]

OK. Let me clarify the detail. When returning SparseDataFrame if any block is sparse:

• Should dense blocks be converted to sparse?
• Or keep dense blocks as it is (allow mixture of sparse blocks and dense blocks)?
  • If so, slicing dense block(s) should be normal Series / DataFrame, rather than SparseSeries / SparseDataFrame?

[jreback]

In an ideal world we wouldn't even have SparseDataFrame, just `SparseSeries``. but since we do I wouldn't coerce anything, e.g. leave sparse as sparse and dense as dense. In general we DO preserve these kinds so things will tend to propogate anyhow.

[sinhrks]

we wouldn't even have SparseDataFrame

Indeed...

Fixed the return type logic and now green.

[jreback]

thanks!