GH-48808: [Python] Drop support for Pandas < 2.0.3#50444
Conversation
|
|
github-actions
Bot
added
Component: Python
awaiting committer review
raulcd
changed the title
|
@github-actions crossbow submit -g python |
|
Revision: e2ae2d8 Submitted crossbow builds: ursacomputing/crossbow @ actions-e1266493a7
|
|
@github-actions crossbow submit test-conda-python-3.11-pandas-2.0.0-numpy-1.23.2 |
|
Revision: 419d344 Submitted crossbow builds: ursacomputing/crossbow @ actions-40780d2205
|
raulcd
requested review from
AlenkaF,
assignUser,
jonkeane,
kou,
pitrou and
rok
as code owners
There was a problem hiding this comment.
Pull request overview
This PR updates PyArrow’s Python/Pandas integration to drop support for pandas versions older than 2.0.0 by raising the minimum-version gate in the pandas shim, removing legacy version conditionals/skips throughout the test suite, and updating CI/docs to reflect the new baseline.
Changes:
- Enforce pandas >= 2.0.0 in the pandas import shim and update the documented minimum pandas version.
- Remove pandas<2-specific coercion paths and version-conditional logic across pandas-related conversions/tests.
- Update CI task matrices to run the pinned “minimum pandas” job against pandas 2.0.0 instead of 1.5.2.
Reviewed changes
Copilot reviewed 13 out of 14 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| python/pyarrow/types.pxi | Removes pandas-v1-specific temporal coercion from dtype mapping logic. |
| python/pyarrow/tests/test_schema.py | Drops pandas-version branching in dtype expectations. |
| python/pyarrow/tests/test_pandas.py | Removes numerous pandas<2 code paths/skips and updates expectations for non-ns units. |
| python/pyarrow/tests/test_compute.py | Removes old pandas<1.1 conditional for isocalendar extraction (always uses modern path). |
| python/pyarrow/tests/parquet/test_pandas.py | Removes skip for pandas versions below a now-unsupported threshold. |
| python/pyarrow/tests/interchange/test_conversion.py | Removes pandas<2-era skips/expected-behavior branching for interchange tests. |
| python/pyarrow/table.pxi | Removes pandas-v1 default that forced temporal nanosecond coercion. |
| python/pyarrow/src/arrow/python/arrow_to_pandas.cc | Minor comment whitespace fix. |
| python/pyarrow/pandas-shim.pxi | Raises minimum supported pandas version check to 2.0.0 and removes is_v1. |
| python/pyarrow/pandas_compat.py | Stops forcing datetimetz unit to ns for pandas v1. |
| python/pyarrow/array.pxi | Removes pandas-v1 default that forced temporal nanosecond coercion for array-like conversions. |
| docs/source/python/install.rst | Updates documented optional dependency minimum to pandas 2.0.0. |
| dev/tasks/tasks.yml | Updates the “minimum pandas” integration-test entry to pandas 2.0.0. |
| .github/workflows/python.yml | Updates the pinned CI job from pandas 1.5.2 to pandas 2.0.0 (and renames job label accordingly). |
github-actions
Bot
added
awaiting changes
| // DateOffset objects do not add nanoseconds component to pd.Timestamp. | ||
| // as of Pandas 1.3.3 | ||
| // as of Pandas 1.3.3 | ||
| // (https://github.com/pandas-dev/pandas/issues/43892). |
There was a problem hiding this comment.
It looks like pandas-dev/pandas#43892 was fixed in 1.4, can this piece of code be simplified? (though I have no idea how :))
There was a problem hiding this comment.
I think it could be simplified to something like:
diff --git a/python/pyarrow/src/arrow/python/arrow_to_pandas.cc b/python/pyarrow/src/arrow/python/arrow_to_pandas.cc
index 348d352a04..581da12577 100644
--- a/python/pyarrow/src/arrow/python/arrow_to_pandas.cc
+++ b/python/pyarrow/src/arrow/python/arrow_to_pandas.cc
@@ -1292,24 +1292,10 @@ struct ObjectWriterVisitor {
auto to_date_offset = [&](const MonthDayNanoIntervalType::MonthDayNanos& interval,
PyObject** out) {
ARROW_DCHECK(internal::BorrowPandasDataOffsetType() != nullptr);
- // DateOffset objects do not add nanoseconds component to pd.Timestamp.
- // as of Pandas 1.3.3
- // (https://github.com/pandas-dev/pandas/issues/43892).
- // So convert microseconds and remainder to preserve data
- // but give users more expected results.
- int64_t microseconds = interval.nanoseconds / 1000;
- int64_t nanoseconds;
- if (interval.nanoseconds >= 0) {
- nanoseconds = interval.nanoseconds % 1000;
- } else {
- nanoseconds = -((-interval.nanoseconds) % 1000);
- }
PyDict_SetItemString(kwargs.obj(), "months", PyLong_FromLong(interval.months));
PyDict_SetItemString(kwargs.obj(), "days", PyLong_FromLong(interval.days));
- PyDict_SetItemString(kwargs.obj(), "microseconds",
- PyLong_FromLongLong(microseconds));
- PyDict_SetItemString(kwargs.obj(), "nanoseconds", PyLong_FromLongLong(nanoseconds));
+ PyDict_SetItemString(kwargs.obj(), "nanoseconds", PyLong_FromLongLong(interval.nanoseconds));
*out =
PyObject_Call(internal::BorrowPandasDataOffsetType(), args.obj(), kwargs.obj());
RETURN_IF_PYERROR();but I think this is a small change on the UX, nowadays we don't roundtrip hours/minutes on DateOffset we show microseconds and nanoseconds. With this change we will be consistent and we would show, Month/Day/Nano instead of Month/Day/Microseconds/Nano. I think this is worth it but maybe we should do it as a separate issue from this one around dropping older pandas supprt and discuss there whether the UX change is worth?
Are you ok if I move this to its own issue?
There was a problem hiding this comment.
Yes, it's ok to me. Also should probably ping @jorisvandenbossche for advice :)
There was a problem hiding this comment.
I created an issue to track this separately:
raulcd
changed the title
github-actions
Bot
added
awaiting change review
|
@github-actions crossbow submit test-conda-python-3.11-pandas-2* |
|
Revision: 5dad2a2 Submitted crossbow builds: ursacomputing/crossbow @ actions-8ccbf18024
|
|
@github-actions crossbow submit pandas |
|
Let's ensure that all Pandas-enabling builds pass before merging. |
|
Revision: 5dad2a2 Submitted crossbow builds: ursacomputing/crossbow @ actions-9768a39d8c
|
Rationale for this change
Pandas 2.0.0 was released on April 3rd 2023.
The last version of Pandas 1 was Pandas 1.5.3 and it was released on January 19th 2023. That version only supports up to Python 3.11.
We are currently supporting up to Pandas 1.5.2 (first version supporting 3.11).
On the issue we discussed to bump our Pandas support to 2.0.0 but given 2.0.3 is the latest in the 2.0.x, was released on June 2023, only supports up to Python 3.11 and bumping to it allows use to improve even further some of the conditionals we are bumping to Pandas >= 2.0.3
What changes are included in this PR?
Update minimal support check to Version(2.0.3) and remove all test skips, conditionals and scenarios where we are checking for Pandas < 2.0.3.
We found a couple of minor things
Are these changes tested?
Yes via CI
Are there any user-facing changes?
Yes, Pandas < 2 won't be supported.