Integrate dask-expr and make CI happy

[milesgranger]

First commit is only skipping tests that fail currently w/o dask-expr, can be moved to it's own PR if desired. It skips tests that fail from mostly deprecations/future warnings, and some other things due to outdated code from updated dependencies.

other commits are adding dask-expr logic and marking tests that currently fail using skipif dask-expr is enabled.

[milesgranger]

@phofl

traceback

___________________________ TestPolynomialFeatures.test_df_transform_index[True] ___________________________

self = <tests.preprocessing.test_data.TestPolynomialFeatures object at 0x7f0c5bf00bb0>, daskify = True

    @pytest.mark.parametrize("daskify", [pytest.param(True, marks=mark), False])
    def test_df_transform_index(self, daskify):
        frame = copy(df)
        if not daskify:
            frame = frame.compute()
        frame = frame.sample(frac=1.0)
        res_df = dpp.PolynomialFeatures(
            preserve_dataframe=True, degree=1
        ).fit_transform(frame)
>       assert_eq_df(res_df.iloc[:, 1:], frame, check_dtype=False)

tests/preprocessing/test_data.py:649:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <dask_expr._indexing.ILocIndexer object at 0x7f0c691fb400>
key = (slice(None, None, None), slice(1, None, None))

    def __getitem__(self, key):
        msg = (
            "'DataFrame.iloc' only supports selecting columns. "
            "It must be used like 'df.iloc[:, column_indexer]'."
        )
        if not isinstance(key, tuple):
            raise NotImplementedError(msg)

        if len(key) > 2:
            raise ValueError("Too many indexers")

        iindexer, cindexer = key

        if iindexer != slice(None):
            raise NotImplementedError(msg)

        if len(self.obj.columns) == len(set(self.obj.columns)):
            col_names = self.obj.columns[cindexer]
            if not is_scalar(col_names):
                col_names = list(col_names)
            return new_collection(Projection(self.obj, col_names))
        else:
>           raise NotImplementedError
E           NotImplementedError

../../mambaforge/envs/dask-ml/lib/python3.10/site-packages/dask_expr/_indexing.py:51: NotImplementedError

[phofl]

does the DataFRame have duplicated columns? Then yes, we don't care for now

[milesgranger]

Ah, okay the output res_df has duplicate columns, but seems expected as that's what the iloc is for, but you're saying b/c of that we don't care right now because it's not implemented.

(Pdb) p frame.columns
Index(['0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12',
       '13', '14', '15', '16', '17', '18', '19'],
      dtype='object')
(Pdb) p res_df.columns
Index(['1', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', '12',
       '13', '14', '15', '16', '17', '18', '19'],
      dtype='object')

[milesgranger]

@phofl

traceback

    def test_categorical(self, categories, transformed, daskify, ordered):
        cat = pd.Series(
            ["a", "b", "a"],
            dtype=pd.api.types.CategoricalDtype(categories=categories, ordered=ordered),
        )
        if daskify:
            cat = dd.from_pandas(cat, npartitions=2)
            transformed = da.from_array(transformed, chunks=(2, 1))
            if daskify == "unknown":
                cat = cat.cat.as_unknown()

        a = dpp.LabelEncoder().fit(cat)

        if daskify != "unknown":
            assert a.dtype_ == cat.dtype

>       np.testing.assert_array_equal(a.classes_, categories)

tests/preprocessing/test_label.py:125:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

args = (<built-in function eq>, array(['b', 'a'], dtype=object), ['a', 'b'])
kwds = {'err_msg': '', 'header': 'Arrays are not equal', 'strict': False, 'verbose': True}

    @wraps(func)
    def inner(*args, **kwds):
        with self._recreate_cm():
>           return func(*args, **kwds)
E           AssertionError:
E           Arrays are not equal
E
E           Mismatched elements: 2 / 2 (100%)
E            x: array(['b', 'a'], dtype=object)
E            y: array(['a', 'b'], dtype='<U1')

../../mambaforge/envs/dask-ml/lib/python3.10/contextlib.py:79: AssertionError

[phofl]

This could be legit, could you look into why the order is not correct?

[milesgranger]

@phofl I kinda assume we can safely ignore this test for dask-expr? Fails b/c we cannot set _meta.

[phofl]

Hm this is not great since it seems like the behaviour actually depends on meta?

[milesgranger]

Is there a 'good' way to modify meta? Otherwise I can try mocking it here.

[phofl]

lets chat about this in the standup

[phofl]

okay lets move ahead then

[phofl]

I can't merge, don't know who can

@fjetter probably?

[TomAugspurger]

I can as well :) Thanks for handling this!