Frequent Patterns Sparse Support

[rasbt]

Attempt to add support for Pandas 1.0 in frequent pattern mining functions.

It looks like pandas 1.0 had quite a revamp regarding sparse data frames. I attempted to use the migration guide at https://pandas.pydata.org/pandas-docs/stable/user_guide/sparse.html#migrating but it's a bit tricky, because they don't have fill values anymore etc.

@dbarbier and @DanielMorales9 , do you have experience with porting code to pandas 1.0? I would really appreciate your feedback for how to handle these cases.

[dbarbier]

I believe that e3ff3a7 fixes all issues with pandas 1.0, except failures with test_sparse_with_zero. But this is a because pandas 1.0 does not handle sparse dataframes with zeros, see pandas-dev/pandas#29814. You could either remove this test, or discard it with pandas >= 1.0.

[pep8speaks]

Hello @rasbt! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-02-24 20:39:52 UTC

[rasbt]

Thanks for the feedback! To prevent the codebase becoming more and more cluttered, I thought that even dropping SparseDataFrame support entirely may be a good idea (in favor of the new sparse data handling in Pandas >= 1.0). I added a check valid_input_check() (in fpcommon.py) to let users now that we only support the "new" way.

Thanks your points, most of the issues are now addressed. One thing though is that in some tests, we check columns and dtypes of the columns. However, when doing sth like

dfs = df.astype(pd.SparseDtype("int", np.nan)).sparse.to_coo()

this will produce a sparse scipy matrix, which doesn't have dtypes and columns. So we'd just need to do

dfs = df.astype(pd.SparseDtype("int", np.nan))

I guess. But is this now really "sparse" or do we need to do sth extra?

[dbarbier]

Thanks for the feedback! To prevent the codebase becoming more and more cluttered, I thought that even dropping SparseDataFrame support entirely may be a good idea (in favor of the new sparse data handling in Pandas >= 1.0). I added a check valid_input_check() (in fpcommon.py) to let users now that we only support the "new" way.

Good idea, you can then remove if hasattr(df, "to_coo") blocks which become useless.

With dfs = df.astype(pd.SparseDtype(...)), all columns become sparse, yes.

[rasbt]

Good idea, you can then remove if hasattr(df, "to_coo") blocks which become useless.

Ah yes, thanks, will do!

dfs = df.astype(pd.SparseDtype(...))

Thanks! I was a bit suspicious for some reason ...

[rasbt]

I think I addressed everything now. Will make a new "bugfix" version so that there is a version that works for people who installed the latest pandas version. Thanks for your help!

[dbarbier]

You should have kept

   all_bools = ((df.dtypes == pd.SparseDtype(bool)) |
                        (df.dtypes == bool)).all()

[rasbt]

not sure why I deleted this! Now, after adding it back, for some reason

(df.dtypes == pd.SparseDtype(bool))

produces a deprecation warning for some reason.

DeprecationWarning: elementwise comparison failed; this will raise an error in the future.
res_values = op(left, right)

Can't reproduce this issue outside the code though.

[dbarbier]

Indeed, that's weird; I do not find how to avoid this warning.

[dbarbier]

This seems to work; is_bool_dtype had been introduced in pandas 0.19, so there is no need to check version:

# Fast path: if all columns are boolean, there is nothing to checks
all_bools = df.dtypes.apply(pd.api.types.is_bool_dtype).all()

[rasbt]

Awesome, that works! Thanks a lot!