You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The test_categorize_info tests the info display, which includes the memory size of the dataframe. When the dataframe is using pyarrow string dtype, this test is failing with latest pyarrow.
Checking locally, it seems that with pyarrow main, the StringArray that gets constructed under the hood no longer has a validity bitmap allocated (for a case without nulls, so this is a fine optimization), while with pyarrow 14.0, the bitmap is present. This results in a change in .nbytes, making the test fail. This will need to be fixed downstream in dask.
The text was updated successfully, but these errors were encountered:
…ping test_categorize_info (#39029)
The test requires an downstream fix in dask (because of a valid change in Arrow), until then temporarily skipping this test (see the issue for more details).
* Closes: #39028
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
…y skipping test_categorize_info (apache#39029)
The test requires an downstream fix in dask (because of a valid change in Arrow), until then temporarily skipping this test (see the issue for more details).
* Closes: apache#39028
Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
See for example https://github.com/ursacomputing/crossbow/actions/runs/7054010641/job/19202136552
The
test_categorize_info
tests the info display, which includes the memory size of the dataframe. When the dataframe is using pyarrow string dtype, this test is failing with latest pyarrow.Checking locally, it seems that with pyarrow main, the StringArray that gets constructed under the hood no longer has a validity bitmap allocated (for a case without nulls, so this is a fine optimization), while with pyarrow 14.0, the bitmap is present. This results in a change in
.nbytes
, making the test fail. This will need to be fixed downstream in dask.The text was updated successfully, but these errors were encountered: