Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ARROW-7950: [Python] Determine + test minimal pandas version + raise error when pandas is too old #6992

Closed
wants to merge 6 commits into from
Closed

ARROW-7950: [Python] Determine + test minimal pandas version + raise error when pandas is too old #6992

wants to merge 6 commits into from
+51 −30

Conversation

jorisvandenbossche
Copy link
Member

No description provided.

@jorisvandenbossche
Copy link
Member Author

This is still WIP (depending on which pandas version we choose, we can clean up some things in the pandas-shim.pxi), but:

  • I defined the minimal pandas version now as pandas 0.23 (released 2 years ago). But, I also ran tests with pandas 0.22 and pandas 0.21, and tests are still passing for those. So we could also set this minimal version at 0.21 (the mailing list thread that triggered this issue reported an error on pandas 0.20). But personally, I think 0.23 is more than fine.
  • I now raise an error when the pandas version is too low. But, since it is actually working (mostly) for lower versions as well, we could also only give a warning instead.

@github-actions
Copy link

https://issues.apache.org/jira/browse/ARROW-7950

pitrou
pitrou reviewed Apr 22, 2020
View reviewed changes
python/pyarrow/pandas-shim.pxi
"installed".format(self._version)
)
else:
return
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we perhaps emit a warning here? I don't think that users expect their Pandas installation to be silently ignored.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's a good idea

pitrou
pitrou reviewed Apr 22, 2020
View reviewed changes
python/pyarrow/tests/test_pandas.py Outdated
@@ -2685,8 +2685,8 @@ class A:
'a': pd.period_range('2000-01-01', periods=20),
})

expected_msg = 'Conversion failed for column a with type period'
with pytest.raises(TypeError, match=expected_msg):
expected_msg = 'Conversion failed for column a with type period|object'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean (period|object)? (parentheses included)

@jorisvandenbossche
Copy link
Member Author

cc @wesm @xhochy @BryanCutler are you fine with

  1. a hard required minimal pandas version? (meaning: we don't use the pandas integration if an older version is installed, instead of trying to use it and potentially erroring/failing in certain cases)
  2. if ok with 1, pandas 0.23 as minimal version? (released 2 years ago, but our tests still pass with eg 0.21 as well)

@wesm
Copy link
Member

wesm commented Apr 23, 2020

I'm OK with this. The maintenance burden of supporting several years' worth of pandas releases seems like a lot to bear. If there are parties which are affected by this they should contribute to the maintenance (either monetarily or with their time)

@BryanCutler
Copy link
Member

Sounds good to me. FWIW, Spark also has a minimum Pandas version set at 0.23.2.

wesm
wesm approved these changes Apr 23, 2020
View reviewed changes
Copy link
Member

@wesm wesm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@wesm
Copy link
Member

wesm commented Apr 23, 2020

The Appveyor failure is unrelated

@wesm
Copy link
Member

wesm commented Apr 23, 2020

Actually I'll hold off on merging this to confirm that @jorisvandenbossche has done everything that he planned

@xhochy
Copy link
Member

xhochy commented Apr 23, 2020

👍

2 years ago released pandas version still sounds very generous. People who cannot upgrade from that to a newer version will probably have the same problems with pyarrow updates.

@jorisvandenbossche
Copy link
Member Author

I further cleaned up the shim to remove if/else checks we no longer need, so should be ready now.

@wesm
Copy link
Member

wesm commented Apr 23, 2020

Sweet thanks, merging now

@wesm wesm closed this in fa60ea6 Apr 23, 2020
@jorisvandenbossche jorisvandenbossche deleted the ARROW-7950-pandas-version branch April 24, 2020 09:55
@asfimport asfimport mentioned this pull request Apr 23, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@pitrou pitrou pitrou left review comments

@wesm wesm wesm approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@jorisvandenbossche @wesm @BryanCutler @xhochy @pitrou