Improved support for `pyarrow` strings #10000

j-bennet · 2023-02-24T23:55:05Z

Part of #9946.

Upstream issues:

BUG: pd.concat fails with GroupBy.head() and pd.StringDtype["pyarrow"] pandas-dev/pandas#51734
BUG: converting a string[pyarrow] column to category triggers an error in to_parquet pandas-dev/pandas#51752
Writing HDF5 with StringDtype columns fails with AttributeError: 'StringArray' object has no attribute 'size' pandas-dev/pandas#31199
[Python] to_parquet fails with a category field backed by pyarrow string apache/arrow#34449 (closed) is a duplicate of the following issue:
- [Python] array() errors if pandas categorical column has dictionary as string not object apache/arrow#33727
- PR: GH-33727: [Python] array() errors if pandas categorical column has dictionary as string not object apache/arrow#34289
[C++] String concatenate aggregate kernel apache/arrow#28454 (related to sum aggregation)

Issues to follow up in Dask:

Tests added / passed
Passes pre-commit run --all-files

dask/array/core.py

dask/array/utils.py

dask/utils_test.py

jrbourbeau

dask/partd#64 should fix some of the shuffle-related serialization errors. You might try temporarily pointing CI environments at that PR to see what tests start passing

j-bennet · 2023-03-02T22:21:18Z

dask/partd#64 should fix some of the shuffle-related serialization errors. You might try temporarily pointing CI environments at that PR to see what tests start passing

@jrbourbeau Thanks, I added this to install script. I couldn't add it to the conda env file because of version conflict - when installed from git, partd version is 1.0, and we have 1.2.0 in requirements.

conftest.py

dask/bag/tests/test_bag.py

dask/dataframe/io/tests/test_csv.py

dask/dataframe/io/tests/test_demo.py

dask/dataframe/io/tests/test_parquet.py

dask/dataframe/tests/test_arithmetics_reduction.py

dask/dataframe/tests/test_groupby.py

dask/dataframe/utils.py

j-bennet · 2023-03-06T18:15:14Z

This PR is using a couple of different ways to handle pyarrow string failures.

For some tests, using pyarrow strings is undesirable or unnecessary:
- sometimes we explicitly need object dtype, for example: to store lists and explode
- tests that check for a specific number of graph layers
- tests that don't have upstream support for pyarrow strings, and it's not clear when support will be added (storing to hdf, spark compatibility)

These tests use a fixture called disable_pyarrow_strings, defined in conftest.py: @pytest.mark.usefixtures("disable_pyarrow_strings"). Mostly there's also a comment saying why.

Tests that have corresponding issues opened in upstream. Marked with pytest.mark.xfail(pyarrow_strings_enabled(), reason="[link to issue]".
Tests that fail with pyarrow strings, but there's no corresponding upstream issue:
- fastparquet tests are expected to fail with pyarrow strings enabled in config, because we raise a warning
- some reductions are not implemented for pyarrow strings in upstream, such as all, sum and mean.

Those are marked with @xfail_with_pyarrow_strings, a shared mark defined in dask.tests.__init__.py.

dask/bag/tests/test_bag.py

continuous_integration/scripts/install.sh

dask/dataframe/io/tests/test_parquet.py

dask/dataframe/tests/test_dataframe.py

dask/dataframe/tests/test_groupby.py

dask/dataframe/tests/test_rolling.py

j-bennet · 2023-03-07T05:11:16Z

I simplified how the failures are handled. We have two different types now:

Marked with xfail_with_pyarrow_strings. These break with pyarrow strings. Some may be dask problems, some may be fixed upstream, we may have to get back to them.
Marked with skip_with_pyarrow_strings. These don't make sense with pyarrow strings and there's no point to run them.

…d continue-on-error.

…rrow-ci-failures

…ssing.

…ltiprocessing." This reverts commit cd91854.

…rrow-ci-failures

jrbourbeau

Thanks for all your work here @j-bennet!

jrbourbeau · 2023-03-08T18:29:25Z

.github/workflows/tests.yml

-        # pyarrow dtypes currently fail, so we allow continuing on error for that specific build
-        # TODO: Remove the `continue-on-error` line below once tests are all passing
-        continue-on-error: ${{ matrix.extra == 'pyarrow' }}


mrocklin · 2023-03-08T21:46:46Z

🎉

j-bennet marked this pull request as draft February 24, 2023 23:55

github-actions bot added dataframe io labels Feb 24, 2023

j-bennet changed the title ~~Tmporarily turn on pyarrow strings to look for CI failures.~~ Temporarily turn on pyarrow strings to look for CI failures. Feb 24, 2023

j-bennet added the upstream label Feb 24, 2023

github-actions bot added the array label Mar 1, 2023

j-bennet force-pushed the j-bennet/pyarrow-ci-failures branch from 14201f7 to 37b6d81 Compare March 1, 2023 17:07

jrbourbeau reviewed Mar 1, 2023

View reviewed changes

dask/array/core.py Outdated Show resolved Hide resolved

dask/array/utils.py Outdated Show resolved Hide resolved

dask/utils_test.py Outdated Show resolved Hide resolved

github-actions bot removed the array label Mar 1, 2023

jrbourbeau reviewed Mar 2, 2023

View reviewed changes

j-bennet force-pushed the j-bennet/pyarrow-ci-failures branch from e96e2bf to 7e65678 Compare March 2, 2023 21:42

jrbourbeau reviewed Mar 3, 2023

View reviewed changes

j-bennet force-pushed the j-bennet/pyarrow-ci-failures branch 4 times, most recently from 889a4fb to 43a6d96 Compare March 5, 2023 00:09

j-bennet changed the title ~~Temporarily turn on pyarrow strings to look for CI failures.~~ Initial support for pyarrow strings Mar 6, 2023

j-bennet force-pushed the j-bennet/pyarrow-ci-failures branch from 2b5eed4 to 6ca705b Compare March 6, 2023 17:43

jrbourbeau reviewed Mar 6, 2023

View reviewed changes

j-bennet mentioned this pull request Mar 6, 2023

[pyarrow strings] consistently handle string[pyarrow] in MultiIndex #10025

Closed

j-bennet force-pushed the j-bennet/pyarrow-ci-failures branch 3 times, most recently from 4c98ed1 to 400bb23 Compare March 7, 2023 04:49

j-bennet mentioned this pull request Mar 7, 2023

[pyarrow strings] Follow-up on test failures #10029

Open

5 tasks

j-bennet marked this pull request as ready for review March 7, 2023 18:09

j-bennet added 2 commits March 7, 2023 10:51

Tmporarily turn on pyarrow strings to look for CI failures.

778ab1d

test-upstream

2af891e

j-bennet added 9 commits March 7, 2023 10:51

Fix for DataFrame.categorize().

ae93e82

Fix for test_abs.

991dd15

This only fails with numeric_only=None.

9cb5ac7

Fix test_categorical.py::test_categorize.

4269462

Review feedback.

9d4d2e7

Fix for dask/dataframe/tests/test_extensions.py::test_reduction.

b9a7c65

No need for objects with decimal dtypes. Fix for get_dummies.

13b99a4

Xfail the last batch of tests. Remove the change to install script an…

3c36f56

…d continue-on-error.

See if we still need this importorskip.

b295ef9

j-bennet force-pushed the j-bennet/pyarrow-ci-failures branch from 1c1b692 to b295ef9 Compare March 7, 2023 18:51

j-bennet and others added 4 commits March 7, 2023 11:26

Xfail melt. Add related issue links.

80fef5c

Move markers to conftest.py

2bad4af

Merge branch 'main' of https://github.com/dask/dask into j-bennet/pya…

8d84fc3

…rrow-ci-failures

xfail new test_filter_nulls test

3d47e10

j-bennet mentioned this pull request Mar 7, 2023

Raise an error with dataframe.convert_string=True and pandas<2.0 #10033

Merged

2 tasks

j-bennet and others added 5 commits March 7, 2023 14:52

Fix for dask/dataframe/tests/test_groupby.py::test_groupby_multiproce…

cd91854

…ssing.

Revert "Fix for dask/dataframe/tests/test_groupby.py::test_groupby_mu…

8032b00

…ltiprocessing." This reverts commit cd91854.

Minor csv update

2aac112

Merge branch 'main' of https://github.com/dask/dask into j-bennet/pya…

a093cf7

…rrow-ci-failures

Merge branch 'main' of https://github.com/dask/dask into j-bennet/pya…

8e6667a

…rrow-ci-failures

jrbourbeau changed the title ~~Initial support for pyarrow strings~~ Improved support for pyarrow strings Mar 8, 2023

jrbourbeau approved these changes Mar 8, 2023

View reviewed changes

jrbourbeau merged commit 15ba4b3 into dask:main Mar 8, 2023

j-bennet deleted the j-bennet/pyarrow-ci-failures branch March 8, 2023 22:13

jrbourbeau mentioned this pull request Mar 10, 2023

Fix test_str_accessor with pyarrow strings active #10048

Merged

j-bennet mentioned this pull request Mar 10, 2023

[pyarrow strings] Fix test_set_index_on_empty #10049

Closed

2 tasks

jrbourbeau mentioned this pull request Mar 10, 2023

Fix test_values with pyarrow strings active #10050

Merged

This was referenced Mar 11, 2023

Fix test_melt with pyarrow strings active #10052

Merged

Fix test_set_index_on_empty with pyarrow strings active #10054

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improved support for `pyarrow` strings #10000

Improved support for `pyarrow` strings #10000

j-bennet commented Feb 24, 2023 •

edited

Loading

jrbourbeau left a comment

j-bennet commented Mar 2, 2023

j-bennet commented Mar 6, 2023 •

edited

Loading

j-bennet commented Mar 7, 2023

jrbourbeau left a comment

jrbourbeau Mar 8, 2023

mrocklin commented Mar 8, 2023

Improved support for pyarrow strings #10000

Improved support for pyarrow strings #10000

Conversation

j-bennet commented Feb 24, 2023 • edited Loading

jrbourbeau left a comment

Choose a reason for hiding this comment

j-bennet commented Mar 2, 2023

j-bennet commented Mar 6, 2023 • edited Loading

j-bennet commented Mar 7, 2023

jrbourbeau left a comment

Choose a reason for hiding this comment

jrbourbeau Mar 8, 2023

Choose a reason for hiding this comment

mrocklin commented Mar 8, 2023

Improved support for `pyarrow` strings #10000

Improved support for `pyarrow` strings #10000

j-bennet commented Feb 24, 2023 •

edited

Loading

j-bennet commented Mar 6, 2023 •

edited

Loading