Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Python] Failing tests because pandas.Index can now store all numeric dtypes (not only 64bit versions) #34404

Closed
jorisvandenbossche opened this issue Mar 1, 2023 · 0 comments · Fixed by #34498
Closed

[Python] Failing tests because pandas.Index can now store all numeric dtypes (not only 64bit versions) #34404

jorisvandenbossche opened this issue Mar 1, 2023 · 0 comments · Fixed by #34498
Assignees
@AlenkaF
Labels
Component: Python Type: test
Milestone
12.0.0

Comments

@jorisvandenbossche
Copy link
Member

We have several failing tests in the nightly build (https://github.com/ursacomputing/crossbow/actions/runs/4277727973/jobs/7446784501) because of a change in pandas 2.0: the Index can now store all numeric dtypes, and not just int64/uint64/float64, see https://pandas.pydata.org/docs/dev/whatsnew/v2.0.0.html#index-can-now-hold-numpy-numeric-dtypes.

Failing tests because of this:

FAILED python/pyarrow/tests/test_pandas.py::test_table_from_pandas_schema_index_columns - AssertionError: DataFrame.index are different
FAILED python/pyarrow/tests/parquet/test_dataset.py::test_read_partitioned_directory[False] - AssertionError: Attributes of DataFrame.iloc[:, 2] (column name="foo") are different
FAILED python/pyarrow/tests/parquet/test_dataset.py::test_read_partitioned_directory_s3fs[False] - AssertionError: Attributes of DataFrame.iloc[:, 2] (column name="foo") are different

I think all those cases are where now an int32 dtype is preserved, while before it would have been cast to int64 by pandas. But the expected result still uses int64, causing the test failures.
@jorisvandenbossche jorisvandenbossche added this to the 12.0.0 milestone Mar 1, 2023
@AlenkaF AlenkaF self-assigned this Mar 2, 2023
@github-actions github-actions bot mentioned this issue Mar 8, 2023
Merged
jorisvandenbossche pushed a commit that referenced this issue Mar 10, 2023
@AlenkaF
GH-34404: [Python] Failing tests because pandas.Index can now store a…
71f3c56 
…ll numeric dtypes (not only 64bit versions) (#34498)

### Rationale for this change
Several failing tests in the nightly build (https://github.com/ursacomputing/crossbow/actions/runs/4277727973/jobs/7446784501) 

### What changes are included in this PR?
Due to change in supported dtypes for Index in pandas, the tests expecting `int64`and not `int32` are failing with dev version of pandas. The failing tests are updated to match the new pandas behaviour.
* Closes: #34404

Authored-by: Alenka Frim <frim.alenka@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
@noloerino noloerino mentioned this issue May 3, 2023
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AlenkaF AlenkaF

Labels
Component: Python Type: test
Projects
None yet
Milestone
12.0.0
2 participants
@jorisvandenbossche @AlenkaF