add warning for pandas sparse Dataframe in check_array #16021

rushabh-v · 2020-01-04T09:22:53Z

rushabh-v · 2020-01-04T13:12:37Z

The added test fails and gives ValueError: shape mismatch: value array of shape (2,) could not be broadcast to indexing result of shape (1,10) on only Linux py35_conda_openblas. And this error is raised by pandas. Any hints why it only fails on Linux py35_conda_openblas?
logs here

rushabh-v · 2020-01-04T14:07:05Z

The added test fails and gives ValueError: shape mismatch: value array of shape (2,) could not be broadcast to indexing result of shape (1,10) on only Linux py35_conda_openblas. And this error is raised by pandas. Any hints why it only fails on Linux py35_conda_openblas?
logs here

Linux pylatest_pip_openblas_pandas passes this test.

…into patch-3

rushabh-v · 2020-01-05T07:28:56Z

This bug was fixed in numpy/numpy#5710.
But the Linux py35_conda_openblas is using numpy version 1.11.0

sklearn/utils/validation.py

jnothman · 2020-01-08T05:59:13Z

Is there a pd 0.23 equivalent of pd.api.types.is_sparse, @TomAugspurger?

I suppose I'm fine with having this feature limited to recent Pandas

TomAugspurger · 2020-01-08T13:04:57Z

I'm not sure if there's a method that'll work that far back.

…

On Tue, Jan 7, 2020 at 11:59 PM Joel Nothman ***@***.***> wrote: Is there a pd 0.23 equivalent of pd.api.types.is_sparse, @TomAugspurger <https://github.com/TomAugspurger>? I suppose I'm fine with having this feature limited to recent Pandas — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#16021?email_source=notifications&email_token=AAKAOISJNKI73DJ2C2RHMBLQ4VTTFA5CNFSM4KCVGE42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEILI25A#issuecomment-571903348>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOIXX3P7QG4G5PFO4GGDQ4VTTFANCNFSM4KCVGE4Q> .

rushabh-v · 2020-01-08T17:55:01Z

Can somebody please guide me to get rid of these failing codecov/patch checks?

sklearn/linear_model/tests/test_base.py

thomasjpfan

Thank you for the PR @rushabh-v !

sklearn/linear_model/tests/test_base.py

sklearn/utils/validation.py

thomasjpfan

Thank you for the PR @rushabh-v !

jnothman

Otherwise lgtm

jnothman · 2020-01-13T20:03:15Z

sklearn/utils/validation.py

+            from pandas.api.types import is_sparse
+            if array.dtypes.apply(is_sparse).any():
+                warnings.warn(
+                    "pandas Dataframe having sparse columns found."


Suggested change

"pandas Dataframe having sparse columns found."

"pandas.DataFrame having sparse columns found."

jnothman · 2020-01-13T20:04:03Z

sklearn/utils/validation.py

+            if array.dtypes.apply(is_sparse).any():
+                warnings.warn(
+                    "pandas Dataframe having sparse columns found."
+                    "It will be inflated automatically."


Is "inflated" the formal term? We may use "densified" elsewhere

I have used "densified" there now.

rushabh-v · 2020-01-20T12:35:12Z

@jnothman Can you review it again for the changes you requested, please?

jnothman · 2020-01-20T20:02:57Z

I don't think another review from me us needed. Please await another core developer's review. Thanks

rth

A few comments, otherwise LGTM thanks!

rth · 2020-01-27T08:32:23Z

doc/whats_new/v0.23.rst

@@ -131,3 +131,6 @@ Changelog

 - |Enhancement| improve error message in :func:`utils.validation.column_or_1d`.
  :pr:`15926` by :user:`Loïc Estève <lesteve>`.
+- |Enhancement| add warning in :func:`utils.validation.check_array` for
+  pandas sparse Dataframe.


Suggested change

pandas sparse Dataframe.

pandas sparse DataFrame.

rth · 2020-01-27T08:33:55Z

sklearn/linear_model/tests/test_base.py

+    pd = pytest.importorskip('pandas')
+    # restrict the pd versions < '0.24.0' as they have a bug in is_sparse func
+    if LooseVersion(pd.__version__) < '0.24.0':
+        return


Suggested change

return

pytest.skip(reason="pandas 0.24+ required.")

rth · 2020-01-27T08:35:56Z

sklearn/utils/validation.py

+            from pandas.api.types import is_sparse
+            if array.dtypes.apply(is_sparse).any():
+                warnings.warn(
+                    "pandas.DataFrame having sparse columns found."


Suggested change

"pandas.DataFrame having sparse columns found."

"pandas.DataFrame with sparse columns found."

rth · 2020-01-27T08:36:19Z

sklearn/utils/validation.py

+            if array.dtypes.apply(is_sparse).any():
+                warnings.warn(
+                    "pandas.DataFrame having sparse columns found."
+                    "It will be densified automatically."


Suggested change

"It will be densified automatically."

"It will be converted to a dense numpy array."

Made all the changes

rth

Thanks!

…arn#16021)

rushabh-v added 5 commits January 4, 2020 14:47

add warning for pandas sparse dataframe

72dfd79

add whatsnew

fd44a1b

add test

ecafcab

fix linting error

285df52

remove trailing whitespace

ed338f1

rushabh-v added 2 commits January 5, 2020 01:16

Merge branch 'master' of https://github.com/scikit-learn/scikit-learn …

9930ad5

…into patch-3

edit test

1577110

rushabh-v added 2 commits January 6, 2020 09:50

edit test

03a9bae

add pandas version restriction for test

e35a006

TomAugspurger reviewed Jan 6, 2020

View reviewed changes

sklearn/utils/validation.py Outdated Show resolved Hide resolved

use pandas function is_sparse

f686f61

rushabh-v force-pushed the patch-3 branch from 8e867e9 to f686f61 Compare January 7, 2020 18:02

rushabh-v added 2 commits January 8, 2020 20:19

restrict pd version < '0.24.0'

4929f2b

edit warning

1571b83

jnothman reviewed Jan 8, 2020

View reviewed changes

sklearn/linear_model/tests/test_base.py Outdated Show resolved Hide resolved

use LooseVersion

6dfacab

thomasjpfan reviewed Jan 9, 2020

View reviewed changes

rushabh-v added 3 commits January 10, 2020 00:42

make requested changes

c5c04c3

solve linting error

b51d6a3

remove extra imports

a89d844

rushabh-v requested review from thomasjpfan and jnothman January 10, 2020 18:17

jnothman approved these changes Jan 13, 2020

View reviewed changes

make requested changes

87ec92f

rushabh-v requested a review from jnothman January 14, 2020 03:15

rushabh-v requested review from qinhanmin2014, amueller, rth and thomasjpfan and removed request for thomasjpfan January 25, 2020 11:20

rth reviewed Jan 27, 2020

View reviewed changes

rushabh-v added 3 commits January 27, 2020 14:45

make requested changes

605a50d

resolve pytest arg error

734dcef

change msg in test

4a50539

rth approved these changes Jan 27, 2020

View reviewed changes

rth merged commit c2ede74 into scikit-learn:master Jan 27, 2020

rushabh-v deleted the patch-3 branch January 27, 2020 15:32

thomasjpfan pushed a commit to thomasjpfan/scikit-learn that referenced this pull request Feb 22, 2020

ENH add warning for pandas sparse Dataframe in check_array (scikit-le…

89eadc5

…arn#16021)

panpiort8 pushed a commit to panpiort8/scikit-learn that referenced this pull request Mar 3, 2020

ENH add warning for pandas sparse Dataframe in check_array (scikit-le…

a878a6f

…arn#16021)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add warning for pandas sparse Dataframe in check_array #16021

add warning for pandas sparse Dataframe in check_array #16021

rushabh-v commented Jan 4, 2020

rushabh-v commented Jan 4, 2020 •

edited

Loading

rushabh-v commented Jan 4, 2020 •

edited

Loading

rushabh-v commented Jan 5, 2020

jnothman commented Jan 8, 2020

TomAugspurger commented Jan 8, 2020 via email

rushabh-v commented Jan 8, 2020 •

edited

Loading

thomasjpfan left a comment

thomasjpfan left a comment

jnothman left a comment

jnothman Jan 13, 2020

rushabh-v Jan 17, 2020

jnothman Jan 13, 2020

rushabh-v Jan 17, 2020 •

edited

Loading

rushabh-v commented Jan 20, 2020

jnothman commented Jan 20, 2020

rth left a comment

rth Jan 27, 2020

rth Jan 27, 2020

rth Jan 27, 2020

rth Jan 27, 2020

rushabh-v Jan 27, 2020

rth left a comment

	"pandas Dataframe having sparse columns found."
	"pandas.DataFrame having sparse columns found."

	"pandas.DataFrame having sparse columns found."
	"pandas.DataFrame with sparse columns found."

	"It will be densified automatically."
	"It will be converted to a dense numpy array."

add warning for pandas sparse Dataframe in check_array #16021

add warning for pandas sparse Dataframe in check_array #16021

Conversation

rushabh-v commented Jan 4, 2020

rushabh-v commented Jan 4, 2020 • edited Loading

rushabh-v commented Jan 4, 2020 • edited Loading

rushabh-v commented Jan 5, 2020

jnothman commented Jan 8, 2020

TomAugspurger commented Jan 8, 2020 via email

rushabh-v commented Jan 8, 2020 • edited Loading

thomasjpfan left a comment

Choose a reason for hiding this comment

thomasjpfan left a comment

Choose a reason for hiding this comment

jnothman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rushabh-v Jan 17, 2020 • edited Loading

Choose a reason for hiding this comment

rushabh-v commented Jan 20, 2020

jnothman commented Jan 20, 2020

rth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rth left a comment

Choose a reason for hiding this comment

rushabh-v commented Jan 4, 2020 •

edited

Loading

rushabh-v commented Jan 4, 2020 •

edited

Loading

rushabh-v commented Jan 8, 2020 •

edited

Loading

rushabh-v Jan 17, 2020 •

edited

Loading