upstream fixes #10549

graingert · 2023-10-08T11:03:36Z

Closes #xxxx
Tests added / passed
Passes pre-commit run --all-files

graingert · 2023-10-11T11:07:00Z

hmm these tests are failing every time in CI but not locally for me

FAILED dask/dataframe/tests/test_multi.py::test_concat_categorical[True-False-True] - AssertionError: Series are different

Series values are different (26.66667 %)
[index]: [0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4]
[left]:  ['a', 'a', 'b', 'b', 'b', ..., 'b', 'b', 'a', 'c', 'c']
Length: 15
Categories (3, object): ['a', 'b', 'c']
[right]: ['a', 'a', 'b', 'b', 'c', ..., 'b', 'b', 'a', 'c', 'c']
Length: 15
Categories (3, object): ['a', 'b', 'c']
At positional index 4, first diff: b != c
FAILED dask/dataframe/tests/test_multi.py::test_concat_categorical[False-False-True] - AssertionError: Series are different

Series values are different (26.66667 %)
[index]: [0, 0, 0, 1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4]
[left]:  ['a', 'a', 'b', 'b', 'b', ..., 'b', 'b', 'a', 'c', 'c']
Length: 15
Categories (3, object): ['a', 'b', 'c']
[right]: ['a', 'a', 'b', 'b', 'c', ..., 'b', 'b', 'a', 'c', 'c']
Length: 15
Categories (3, object): ['a', 'b', 'c']
At positional index 4, first diff: b != c

eg:

dask/dataframe/tests/test_multi.py::test_concat_categorical[True-True-False] PASSED                                                                         [ 16%]
dask/dataframe/tests/test_multi.py::test_concat_categorical[True-False-True] PASSED                                                                         [ 33%]
dask/dataframe/tests/test_multi.py::test_concat_categorical[True-False-False] PASSED                                                                        [ 50%]
dask/dataframe/tests/test_multi.py::test_concat_categorical[False-True-False] PASSED                                                                        [ 66%]
dask/dataframe/tests/test_multi.py::test_concat_categorical[False-False-True] PASSED                                                                        [ 83%]
dask/dataframe/tests/test_multi.py::test_concat_categorical[False-False-False] PASSED                                                                       [100%]

all pass for me locally

graingert · 2023-10-11T11:45:27Z

dask/tests/test_sizeof.py::test_sparse_matrix is failing due to scipy/scipy#18929

graingert · 2023-10-11T12:32:53Z

diffing my environment gets:

--- ci.txt	2023-10-11 13:28:30.957665048 +0100
+++ mine.txt	2023-10-11 13:26:42.778166612 +0100
@@ -1,4 +1,4 @@
-# packages in environment at /usr/share/miniconda3/envs/test-environment:
+# packages in environment at /home/graingert/mambaforge/envs/test-environment-upstream:
 #
 # Name                    Version                   Build  Channel
 _libgcc_mutex             0.1                 conda_forge    conda-forge
@@ -67,7 +67,7 @@
 cryptography              41.0.4          py310h75e40e8_0    conda-forge
 curl                      8.4.0                hca28451_0    conda-forge
 cytoolz                   0.12.2          py310h2372a71_1    conda-forge
-dask                      0+untagged.1.ge3a25a3.dirty          pypi_0    pypi
+dask                      2023.9.3+13.g010b3099.dirty          pypi_0    pypi
 dav1d                     1.2.1                hd590300_0    conda-forge
 debugpy                   1.8.0           py310hc6cd4ac_1    conda-forge
 decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
@@ -220,7 +220,7 @@
 nbformat                  5.9.2              pyhd8ed1ab_0    conda-forge
 ncurses                   6.4                  hcb278e6_0    conda-forge
 ndindex                   1.7                      pypi_0    pypi
-nest-asyncio              1.5.7              pyhd8ed1ab_0    conda-forge
+nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
 networkx                  3.1                pyhd8ed1ab_0    conda-forge
 nodeenv                   1.8.0              pyhd8ed1ab_0    conda-forge
 nomkl                     1.0                  h5ca1d4c_0    conda-forge
@@ -271,6 +271,7 @@
 pyspark                   3.5.0              pyhd8ed1ab_0    conda-forge
 pytest                    7.4.2              pyhd8ed1ab_0    conda-forge
 pytest-cov                4.1.0              pyhd8ed1ab_0    conda-forge
 pytest-reportlog          0.1.2              pyhd8ed1ab_0    conda-forge
 pytest-rerunfailures      12.0               pyhd8ed1ab_0    conda-forge
 pytest-timeout            2.2.0              pyhd8ed1ab_0    conda-forge

phofl · 2023-10-11T17:35:28Z

Using the newest Dask release and pandas main I get the following:

Traceback (most recent call last):
  File "/Users/patrick/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/232.8660.197/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevd.py", line 1500, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Users/patrick/Library/Application Support/JetBrains/Toolbox/apps/PyCharm-P/ch-0/232.8660.197/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/patrick/Library/Application Support/JetBrains/PyCharm2023.2/scratches/scratch.py", line 552, in <module>
    res = check_and_return(dframes, frames, join)
  File "/Users/patrick/Library/Application Support/JetBrains/PyCharm2023.2/scratches/scratch.py", line 539, in check_and_return
    assert_eq(res, sol)
  File "/Users/patrick/mambaforge/envs/pandas-dev/lib/python3.10/site-packages/dask/dataframe/utils.py", line 602, in assert_eq
    tm.assert_frame_equal(
  File "/Users/patrick/PycharmProjects/pandas/pandas/_testing/asserters.py", line 1210, in assert_frame_equal
    assert_series_equal(
  File "/Users/patrick/PycharmProjects/pandas/pandas/_testing/asserters.py", line 919, in assert_series_equal
    assert_attr_equal("dtype", left, right, obj=f"Attributes of {obj}")
  File "/Users/patrick/PycharmProjects/pandas/pandas/_testing/asserters.py", line 414, in assert_attr_equal
    raise_assert_detail(obj, msg, left_attr, right_attr)
  File "/Users/patrick/PycharmProjects/pandas/pandas/_testing/asserters.py", line 599, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: Attributes of DataFrame.iloc[:, 0] (column name="w") are different

Attribute "dtype" are different
[left]:  CategoricalDtype(categories=['x', 'y', 'z'], ordered=False, categories_dtype=object)
[right]: string[pyarrow]

…st-upstream]

…ease use 'BY' instead of 'BA'

…test-upstream]

graingert · 2023-10-18T09:14:45Z

I've added to the issue here #10558

phofl

2 small comments, otherwise lgtm

continuous_integration/scripts/install.sh

dask/dataframe/tests/test_multi.py

phofl · 2023-10-18T12:29:10Z

pyproject.toml

@@ -148,6 +148,8 @@ filterwarnings = [
    "ignore:'H' is deprecated and will be removed in a future version. Please use 'h' instead of 'H':FutureWarning",
    'ignore:DataFrameGroupBy\.apply operated on the grouping columns\. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation\. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning\.:FutureWarning',
    "ignore:'BA' is deprecated and will be removed in a future version. Please use 'BY' instead of 'BA':FutureWarning",
+    'ignore:numpy\.core.* is deprecated and has been renamed to numpy\._core\.*:DeprecationWarning', # needed for pandas to import
+    'ignore:Passing a BlockManager to DataFrame is deprecated and will raise in a future version. Use public APIs instead:DeprecationWarning',


This is something we have to address over in partd, can you open an issue there?

issue opened here dask/partd#77

phofl · 2023-10-18T12:29:29Z

pyproject.toml

@@ -150,7 +150,7 @@ filterwarnings = [
    "ignore:'BA' is deprecated and will be removed in a future version. Please use 'BY' instead of 'BA':FutureWarning",
    'ignore:numpy\.core.* is deprecated and has been renamed to numpy\._core\.*:DeprecationWarning', # needed for pandas to import
    'ignore:Passing a BlockManager to DataFrame is deprecated and will raise in a future version. Use public APIs instead:DeprecationWarning',
-
+    'ignore:The previous implementation of stack is deprecated and will be removed in a future version of pandas\.:FutureWarning',


Could you open a follow up issue to test the new implementation?

opened here #10568

phofl · 2023-10-18T12:29:45Z

2 non blocking comments

feel free to merge when ci is through

graingert · 2023-10-18T12:30:58Z

@phofl I don't have commit-bit on dask/dask so I'll need you to merge

phofl · 2023-10-18T12:31:54Z

thx @graingert

graingert · 2023-10-18T13:57:56Z

pyproject.toml

+    "ignore:'A' is deprecated and will be removed in a future version. Please use 'Y' instead of 'A':FutureWarning",
+    "ignore:'A-JUN' is deprecated and will be removed in a future version. Please use 'Y-JUN' instead of 'A-JUN':FutureWarning",
+    "ignore:'H' is deprecated and will be removed in a future version. Please use 'h' instead of 'H':FutureWarning",
+    'ignore:DataFrameGroupBy\.apply operated on the grouping columns\. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation\. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning\.:FutureWarning',


this needs an issue

issue here: #10573

graingert · 2023-10-18T13:58:02Z

pyproject.toml

@@ -139,6 +139,18 @@ filterwarnings = [
    # https://pandas.pydata.org/docs/dev/whatsnew/v1.5.0.html#using-group-keys-with-transformers-in-groupby-apply
    "ignore:Not prepending group keys:FutureWarning",
    "ignore:.*:dask.tests.warning_aliases.RemovedIn20Warning",
+    "ignore:When grouping with a length-1 list-like, you will need to pass a length-1 tuple to get_group in a future version of pandas:FutureWarning",


this needs an issue

issue here: #10572

jrbourbeau and others added 6 commits September 27, 2023 10:52

Upstream compat [test-upstream]

a11485e

More

4d7ad7b

[test-upstream]

3f42549

Even more [test-upstream]

5301ae4

Merge branch 'main' of github.com:dask/dask into upstream-fix

db951ad

add warnings

4c18106

github-actions bot added dataframe array labels Oct 8, 2023

graingert marked this pull request as draft October 9, 2023 14:14

graingert added 3 commits October 11, 2023 11:28

Merge branch 'main' of github.com:dask/dask into upstream-fix-2

2d9987b

add pytables from my fork [test-upstream]

2e7195e

one more pandas warning [test-upstream]

010b309

work around scipy gh-18929 dok_matrix._dict change [test-upstream]

0201f75

graingert mentioned this pull request Oct 12, 2023

test_concat_categorical fails on pandas >= 2.2.0.dev0 or 3.12 #10558

Open

jrbourbeau mentioned this pull request Oct 12, 2023

Release 2023.10.0 dask/community#350

Closed

8 tasks

graingert added 2 commits October 14, 2023 11:35

Merge branch 'main' of github.com:dask/dask into upstream-fix

4d67096

skip test_concat_categorical[division=True] on pandas>=2.2.0.dev0 [te…

29ed5c2

…st-upstream]

graingert force-pushed the upstream-fix branch from 7341677 to 29ed5c2 Compare October 14, 2023 10:36

graingert added 3 commits October 17, 2023 13:58

ignore 'BA' is deprecated and will be removed in a future version. Pl…

d2beca5

…ease use 'BY' instead of 'BA'

Merge branch 'main' of github.com:dask/dask into upstream-fix

ee4a776

replace A with Y in test_map_freq_to_period_start on pandas >=2.2.0 […

4b0a641

…test-upstream]

graingert marked this pull request as ready for review October 17, 2023 14:12

phofl reviewed Oct 18, 2023

View reviewed changes

continuous_integration/scripts/install.sh Outdated Show resolved Hide resolved

dask/dataframe/tests/test_multi.py Outdated Show resolved Hide resolved

graingert added 2 commits October 18, 2023 10:30

xfail strict test_concat_categorical

c7fae04

use specific commit for PyTables numpy-2 support [test-upstream]

45fadd9

graingert force-pushed the upstream-fix branch from ee3a41c to 45fadd9 Compare October 18, 2023 09:36

phofl approved these changes Oct 18, 2023

View reviewed changes

graingert added 4 commits October 18, 2023 11:18

avoid deprecated numpy.core on numpy 2 [test-upstream]

9b85d04

add more warings [test-upstream]

cfa233f

move imports to top [test-upstream]

1e1cf42

ignore future_stack deprecation warning [test-upstream]

cb445ec

phofl reviewed Oct 18, 2023

View reviewed changes

phofl merged commit 51210a6 into dask:main Oct 18, 2023
24 checks passed

graingert deleted the upstream-fix branch October 18, 2023 12:32

graingert commented Oct 18, 2023

View reviewed changes

This was referenced Apr 25, 2024

[WIP] Upstream fixes #10539

Open

NumPy 2.0 support #11066

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upstream fixes #10549

upstream fixes #10549

graingert commented Oct 8, 2023

graingert commented Oct 11, 2023 •

edited

graingert commented Oct 11, 2023

graingert commented Oct 11, 2023

phofl commented Oct 11, 2023

graingert commented Oct 18, 2023

phofl left a comment

phofl Oct 18, 2023

graingert Oct 19, 2023

phofl Oct 18, 2023

graingert Oct 18, 2023

phofl commented Oct 18, 2023

graingert commented Oct 18, 2023

phofl commented Oct 18, 2023

graingert Oct 18, 2023

graingert Oct 18, 2023

graingert Oct 18, 2023

graingert Oct 18, 2023

upstream fixes #10549

upstream fixes #10549

Conversation

graingert commented Oct 8, 2023

graingert commented Oct 11, 2023 • edited

graingert commented Oct 11, 2023

graingert commented Oct 11, 2023

phofl commented Oct 11, 2023

graingert commented Oct 18, 2023

phofl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phofl commented Oct 18, 2023

graingert commented Oct 18, 2023

phofl commented Oct 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

graingert commented Oct 11, 2023 •

edited