Support latest dask.dataframe with query planning (dask-expr)

[jorisvandenbossche]

The 2024.2.0 release of Dask has deprecated the "legacy" dask.dataframe API. They are building a new API with "Query Planning" based on new high-level expressions (instead of the current HighLevelGraph), which currently lives in the dask-expr package.
It can be enabled through dask.config.set({'dataframe.query-planning': True}).

The upcoming 2024.3.0 release (dask/community#366) plans to enable this by default (but you can still turn it off for now).

Given that dask-geopandas is tightly integrated with dask.dataframe, we will have to update to use the new dask_expr backend.
For released versions of dask-geopandas, users will have to upper pin dask or disable the option for now.

(for reference, a similar issue on the cudf side rapidsai/cudf#15027 (they also have a subclass), and their PR to add basic support: rapidsai/cudf#14805)

[ReptarK]

Hi, any updates on the integration of dask_expr ?

[jorisvandenbossche]

An initial patch supporting dask with query planning was merged in #285. So for now using the latest dask-geopandas from the main branch (not yet released, so something like pip install git+https://github.com/geopandas/dask-geopandas.git) should work with the latest dask releases.

[jorisvandenbossche]

This preliminary support has been released in dask-geopandas 0.4.0 (https://github.com/geopandas/dask-geopandas/releases/tag/v0.4.0)