Fix test_merge_by_index_patterns for pandas 2.0

[j-bennet]

Fixes the following failures in CI with pandas 2.0:

dask/dataframe/tests/test_shuffle.py::test_set_index_overlap_2: AssertionError: DataFrame.columns are different
dask/dataframe/tests/test_multi.py::test_merge_by_index_patterns[tasks-inner]: AssertionError: DataFrame.index are different
dask/dataframe/tests/test_multi.py::test_join_by_index_patterns[disk-inner]: AssertionError: DataFrame.index are different
dask/dataframe/tests/test_multi.py::test_join_by_index_patterns[tasks-inner]: AssertionError: DataFrame.index are different
dask/dataframe/tests/test_multi.py::test_merge_by_multiple_columns[disk-inner]: AssertionError: DataFrame.index are different
dask/dataframe/tests/test_multi.py::test_merge_by_multiple_columns[tasks-inner]: AssertionError: DataFrame.index are different

The failure is related to the following change in 2.0:

https://pandas.pydata.org/docs/dev/whatsnew/v2.0.0.html#empty-dataframes-series-will-now-default-to-have-a-rangeindex

In this test, we're already applying a workaround to set dtype on returned pandas dataframes, because pandas doesn't set it on empty dataframes. But with 2.0, it looks like we needed to apply this in more places. Not sure why this changed.

• Xref ⚠️ Upstream CI failed ⚠️ #9736.
• Xref Properly fix index class and dtype mismatch in empty DataFrame or Series with pandas 2.0 #9957.

• Tests added / passed
• Passes pre-commit run --all-files

[jrbourbeau]

Thanks for your work here @j-bennet. It looks like there are other similar failures (e.g. dask/dataframe/tests/test_multi.py::test_merge_by_multiple_columns, dask/dataframe/tests/test_shuffle.py::test_set_index_overlap_2, etc) that are still happening. Would you like to handle those here, or in a follow-up?

[j-bennet]

It looks like there are other similar failures (e.g. dask/dataframe/tests/test_multi.py::test_merge_by_multiple_columns, dask/dataframe/tests/test_shuffle.py::test_set_index_overlap_2, etc) that are still happening. Would you like to handle those here, or in a follow-up?

Good catch, yes, they look like the same error. I'll include them in the PR.

[j-bennet]

Didn't find a good way to fix dask/dataframe/tests/test_shuffle.py::test_set_index_overlap_2, xfailed for now.

[j-bennet]

The remaining test failures here are flaky tests:

FAILED dask/tests/test_distributed.py::test_blockwise_dataframe_io[False-True-hdf] - AssertionError: DataFrame are different
FAILED dask/dataframe/tests/test_groupby.py::test_dataframe_aggregations_multilevel[cov-disk-1-<lambda>1] - FutureWarning: In a future version, the Index constructor will not infer numeric dtypes when passed object-dtype sequences (matching Series behavior)

[jrbourbeau]

Thanks @j-bennet. Overall this looks reasonable. Are these changes more of a workaround and we should follow-up to make the index match, or is this a good long-term solution?

[j-bennet]

Are these changes more of a workaround and we should follow-up to make the index match, or is this a good long-term solution?

It's a workaround. I created an issue to fix this properly:

• Properly fix index class and dtype mismatch in empty DataFrame or Series with pandas 2.0 #9957

[jrbourbeau]

Great, thanks @j-bennet