DEPS: Add warning if pyarrow is not installed #56896

lithomas1 · 2024-01-15T21:26:56Z

closes #xxxx (Replace xxxx with the GitHub issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

~~Not sure how to add a test for this - importing our warning asserters would cause the warning the be raised! 🤣 ~~

EDIT: Nvm, figured it out.

(I've checked by hand that the warning is both raised when you import pandas directly and import a submodule of pandas e.g. pandas.arrays.
I also double checked that the warning only occurs once and not on every import.)

datapythonista

Looks good, added some minor comments/suggestions.

datapythonista · 2024-01-16T08:14:30Z

pandas/__init__.py

+# Don't allow users to use pandas.os
+del os


Is there a reason to not run this this at the end with the other modules? Seems like it's easier to maintain if there is all the clean up at the end or next to the import, but no big deal

No, not really.

Personally I like to keep the del closed to where it's being used, but no strong preference.

I think os is not being used in the code before beibg closed. As said, not important, but to me personally it makes sense to open and close around the block where it is being used, or open and close at the beginning and the end if it's an import that is used in several places. Now os is being opened at the beginning, and closed in the middle, in a kind of random place if I'm not wrong. No huge difference, but I think it'll help maintain the code if closed at the end (or we move the import and del around where it is being used).

I put the os delete with the deletion of warnings/pa_version_under10p1.

pandas/__init__.py

datapythonista · 2024-01-17T03:17:03Z

pandas/__init__.py

+    from pandas.compat._optional import VERSIONS
+
+    try:
+        import pyarrow  # noqa: F401


Should we add an else block to del pyarrow so pandas.pyarrow is not created?

lithomas1 · 2024-01-17T15:45:07Z

also cc @pandas-dev/pandas-core for feedback on the warning message itself.

jorisvandenbossche · 2024-01-17T15:50:03Z

pandas/__init__.py

+
+    warnings.warn(
+        f"""
+Pyarrow will become a required dependency of pandas in the next major release of pandas,


Suggested change

Pyarrow will become a required dependency of pandas in the next major release of pandas,

Pyarrow will become a required dependency of pandas in the next major release (pandas 3.0),

I think we can be explicit about what this next major release will be

Is the next major release also the next minor release, i.e., 2.2 then 3.0? I noticed that the dev wheels recently switched to reporting 3.0.0.dev0 recently.

Is the next major release also the next minor release, i.e., 2.2 then 3.0? I noticed that the dev wheels recently switched to reporting 3.0.0.dev0 recently.

No, 2.2 is coming out sometime this week.
But yes, 3.0.0 will be the release after that.

jorisvandenbossche · 2024-01-17T15:56:51Z

pandas/__init__.py

+please provide us feedback at https://github.com/pandas-dev/pandas/issues/54466
+        """,
+        DeprecationWarning,
+        stacklevel=2,


Do we want to make this stacklevel=1?

Because this code is top-level and not inside a function or method, it one less than the typical stacklevel of 2 we use in a direct function/method.
Setting it as 1 would avoid that it triggers a warning when you call a function that has an inline import pandas (not sure how important this is, though)

The warning doesn't pop up in the REPL when I import pandas if I do stacklevel=1.

Hmm, OK forget this. I did test it, but should have done something wrong because now I can't reproduce getting the warning with plain import and stacklevel 1

jorisvandenbossche · 2024-01-17T15:58:21Z

pandas/__init__.py

+    warnings.warn(
+        f"""
+Pyarrow will become a required dependency of pandas in the next major release of pandas,
+(to allow more performant data types and better interoperability with other libraries)


I would maybe explicitly mention the new default string dtype, since that's still the only one that is used by default that requires pyarrow

pandas/__init__.py

Dr-Irv

I'm good with it. Will let others make the final decision.

datapythonista

Very nice @lithomas1, looks perfect

pandas/tests/test_common.py

mroeschke

Looks like ci/deps/actions-311-numpydev.yaml looks like it needs pyarrow otherwise LGTM

…o add-pyarrow-warning

…talled

lithomas1 · 2024-01-19T17:13:05Z

Merging for now, happy to address any followups in another PR.

…ot installed) (#56963) Backport PR #56896: DEPS: Add warning if pyarrow is not installed Co-authored-by: Thomas Li <47963215+lithomas1@users.noreply.github.com>

ml-evs · 2024-01-22T11:03:50Z

Sorry to comment here rather than an issue but this feels like quite a niche gotcha -- just a heads up that the warning added here starts with a newline which makes it annoying to filter out (as filterwarnings always matches the start of the message).

e.g., neither

"ignore:Pyarrow will become a required dependency of pandas.*:DeprecationWarning"

nor

"ignore:.*Pyarrow will become a required dependency of pandas.*:DeprecationWarning"

nor

"ignore:.*pandas.*:DeprecationWarning"

will match the message, and what is required is

"ignore:\\nPyarrow will become a required dependency of pandas.*:DeprecationWarning"

I guess changing it now might cause more pain, but I suspect you may get a few more questions on this before v3 (unless I am singularly confused by this -- a quick GH search shows a few PRs that also took a while to get this correct in other packages)!

…)" This reverts commit 5c2a407

* Revert "DEPS: Add warning if pyarrow is not installed (#56896)" This reverts commit 5c2a407 * Add whatsnew * Update * Update doc/source/whatsnew/v2.2.1.rst Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> --------- Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com>

* Revert "DEPS: Add warning if pyarrow is not installed (pandas-dev#56896)" This reverts commit 5c2a407 * Add whatsnew * Update * Update doc/source/whatsnew/v2.2.1.rst Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> --------- Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> (cherry picked from commit 4ed67ac)

* add warning * DEPS: Add warning if pyarrow is not installed * formatting * Update __init__.py * adjustments * adjustments * updates * address code review * Update __init__.py * add pyarrow to npdev build * ignore non numpy related deprecationwarnings/futurewarning * ignore non numpy related deprecationwarnings/futurewarning * Update actions-311-numpydev.yaml

* Revert "DEPS: Add warning if pyarrow is not installed (pandas-dev#56896)" This reverts commit 5c2a407 * Add whatsnew * Update * Update doc/source/whatsnew/v2.2.1.rst Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com> --------- Co-authored-by: Matthew Roeschke <10647082+mroeschke@users.noreply.github.com>

lithomas1 added 4 commits January 15, 2024 12:07

add warning

bb520d1

DEPS: Add warning if pyarrow is not installed

a3b3dc3

formatting

944c288

Update __init__.py

613eac7

datapythonista reviewed Jan 16, 2024

View reviewed changes

mroeschke reviewed Jan 16, 2024

View reviewed changes

pandas/__init__.py Outdated Show resolved Hide resolved

mroeschke added Dependencies Required and optional dependencies Warnings Warnings that appear or should be added to pandas labels Jan 16, 2024

lithomas1 added 2 commits January 16, 2024 16:41

adjustments

21509c5

adjustments

c6ae03d

datapythonista reviewed Jan 17, 2024

View reviewed changes

updates

7b48501

jorisvandenbossche reviewed Jan 17, 2024

View reviewed changes

Dr-Irv reviewed Jan 17, 2024

View reviewed changes

pandas/__init__.py Outdated Show resolved Hide resolved

lithomas1 added 2 commits January 17, 2024 10:18

address code review

199fb7c

Update __init__.py

4637a96

Dr-Irv approved these changes Jan 17, 2024

View reviewed changes

datapythonista approved these changes Jan 18, 2024

View reviewed changes

mroeschke reviewed Jan 18, 2024

View reviewed changes

pandas/tests/test_common.py Outdated Show resolved Hide resolved

mroeschke reviewed Jan 18, 2024

View reviewed changes

lithomas1 added 5 commits January 18, 2024 16:21

add pyarrow to npdev build

aa48e19

Merge branch 'add-pyarrow-warning' of github.com:lithomas1/pandas int…

9999fe3

…o add-pyarrow-warning

ignore non numpy related deprecationwarnings/futurewarning

3aa3ba9

ignore non numpy related deprecationwarnings/futurewarning

0074b03

Update actions-311-numpydev.yaml

cc2debf

lithomas1 added this to the 2.2 milestone Jan 19, 2024

lithomas1 merged commit 5c2a407 into pandas-dev:main Jan 19, 2024
49 of 50 checks passed

lithomas1 deleted the add-pyarrow-warning branch January 19, 2024 17:12

meeseeksmachine pushed a commit to meeseeksmachine/pandas that referenced this pull request Jan 19, 2024

Backport PR pandas-dev#56896: DEPS: Add warning if pyarrow is not ins…

dd4615a

…talled

meeseeksmachine mentioned this pull request Jan 19, 2024

Backport PR #56896 on branch 2.2.x (DEPS: Add warning if pyarrow is not installed) #56963

Merged

phofl added a commit to phofl/pandas that referenced this pull request Feb 21, 2024

Revert "DEPS: Add warning if pyarrow is not installed (pandas-dev#56896…

b4fe3fc

…)" This reverts commit 5c2a407

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DEPS: Add warning if pyarrow is not installed #56896

DEPS: Add warning if pyarrow is not installed #56896

lithomas1 commented Jan 15, 2024 •

edited

datapythonista left a comment

datapythonista Jan 16, 2024

lithomas1 Jan 17, 2024

datapythonista Jan 17, 2024

lithomas1 Jan 17, 2024

datapythonista Jan 17, 2024

lithomas1 Jan 17, 2024

lithomas1 commented Jan 17, 2024

jorisvandenbossche Jan 17, 2024 •

edited

bashtage Jan 17, 2024 •

edited

lithomas1 Jan 17, 2024

jorisvandenbossche Jan 17, 2024

lithomas1 Jan 17, 2024

jorisvandenbossche Jan 17, 2024

jorisvandenbossche Jan 17, 2024

lithomas1 Jan 17, 2024

Dr-Irv left a comment

datapythonista left a comment

mroeschke left a comment

lithomas1 commented Jan 19, 2024

ml-evs commented Jan 22, 2024 •

edited

	Pyarrow will become a required dependency of pandas in the next major release of pandas,
	Pyarrow will become a required dependency of pandas in the next major release (pandas 3.0),

DEPS: Add warning if pyarrow is not installed #56896

DEPS: Add warning if pyarrow is not installed #56896

Conversation

lithomas1 commented Jan 15, 2024 • edited

datapythonista left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lithomas1 commented Jan 17, 2024

jorisvandenbossche Jan 17, 2024 • edited

Choose a reason for hiding this comment

bashtage Jan 17, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Dr-Irv left a comment

Choose a reason for hiding this comment

datapythonista left a comment

Choose a reason for hiding this comment

mroeschke left a comment

Choose a reason for hiding this comment

lithomas1 commented Jan 19, 2024

ml-evs commented Jan 22, 2024 • edited

lithomas1 commented Jan 15, 2024 •

edited

jorisvandenbossche Jan 17, 2024 •

edited

bashtage Jan 17, 2024 •

edited

ml-evs commented Jan 22, 2024 •

edited