Skip to content

reindex fails on Dataset from MultiIndex DataFrame with RuntimeError #10347

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
5 tasks done
csernazs opened this issue May 23, 2025 · 5 comments · May be fixed by #10381
Open
5 tasks done

reindex fails on Dataset from MultiIndex DataFrame with RuntimeError #10347

csernazs opened this issue May 23, 2025 · 5 comments · May be fixed by #10381

Comments

@csernazs
Copy link

What happened?

The reproducer code raises
RuntimeError: Cannot set name on a level of a MultiIndex. Use 'MultiIndex.set_names' instead.
from the last statement.

Traceback:

Traceback (most recent call last):
  File "/tmp/xarray-bug/repro.py", line 53, in <module>
    data2b = data1.reindex(y=base2.y)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/xarray-bug/.venv/lib/python3.12/site-packages/xarray/core/dataset.py", line 3571, in reindex
    return alignment.reindex(
           ^^^^^^^^^^^^^^^^^^
  File "/tmp/xarray-bug/.venv/lib/python3.12/site-packages/xarray/structure/alignment.py", line 974, in reindex
    aligner = Aligner(
              ^^^^^^^^
  File "/tmp/xarray-bug/.venv/lib/python3.12/site-packages/xarray/structure/alignment.py", line 172, in __init__
    self.indexes, self.index_vars = self._normalize_indexes(indexes)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/xarray-bug/.venv/lib/python3.12/site-packages/xarray/structure/alignment.py", line 209, in _normalize_indexes
    pd_idx.name = k
    ^^^^^^^^^^^
  File "/tmp/xarray-bug/.venv/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 1690, in name
    raise RuntimeError(
RuntimeError: Cannot set name on a level of a MultiIndex. Use 'MultiIndex.set_names' instead.

This happens with the latest xarray (2025.4.0). I could bisect the version for this issue, and the last version where it runs fine is 2024.5.0 (and the next version, which is 2024.6.0 breaks it).

Thanks,
Zsolt

What did you expect to happen?

Code run without errors.

Minimal Complete Verifiable Example

import xarray as xr
import pandas as pd
import numpy as np


# Generate two arrays of random numbers
rand1 = np.random.randn(15)
rand2 = np.random.randn(12)

# Create a 5x3 DataArray from rand1 with labeled dimensions and convert it to a Dataset
data1 = xr.DataArray(
    rand1.reshape((5, 3)),
    dims=("x", "y"),
    coords={"x": [1, 2, 3, 4, 5], "y": [1, 2, 3]},
).to_dataset(name="value")

# Create a 3x4 DataArray from rand2 with labeled dimensions and convert it to a Dataset
base1 = xr.DataArray(
    rand2.reshape(3, 4),
    dims=("x", "y"),
    coords={"x": [100, 200, 300], "y": [1, 2, 3, 4]},
).to_dataset(name="base_value")

# Create an equivalent Dataset to data1 using a pandas MultiIndex DataFrame
data2 = xr.Dataset.from_dataframe(
    pd.DataFrame(
        rand1,
        columns=["value"],
        index=pd.MultiIndex.from_product([[1, 2, 3, 4, 5], [1, 2, 3]], names=["x", "y"]),
    )
)

# Create an equivalent Dataset to base1 using a pandas MultiIndex DataFrame
base2 = xr.Dataset.from_dataframe(
    pd.DataFrame(
        rand2,
        columns=["base_value"],
        index=pd.MultiIndex.from_product([[100, 200, 300], [1, 2, 3, 4]], names=["x", "y"]),
    )
)

# Verify that the two ways of constructing the datasets yield the same results
# These succeed
assert data1.equals(data2)
assert base1.equals(base2)

# Reindex `data1` to align its 'y' dimension with the 'y' coordinates of `base1`
data1b = data1.reindex(y=base1.y)  # succeeds

# Reindex `data1` again, but now aligning with `base2` (same structure as `base1`)
# fails with
# RuntimeError: Cannot set name on a level of a MultiIndex. Use 'MultiIndex.set_names' instead.
data2b = data1.reindex(y=base2.y)

MVCE confirmation

  • Minimal example — the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.
  • Complete example — the example is self-contained, including all data and the text of any traceback.
  • Verifiable example — the example copy & pastes into an IPython prompt or Binder notebook, returning the result.
  • New issue — a search of GitHub Issues suggests this is not a duplicate.
  • Recent environment — the issue occurs with the latest version of xarray and its dependencies.

Relevant log output

Anything else we need to know?

No response

Environment

numpy==2.2.6
packaging==25.0
pandas==2.2.3
python-dateutil==2.9.0.post0
pytz==2025.2
six==1.17.0
tzdata==2025.2
xarray==2025.4.0

I'm using Linux on x86-64 with python 3.12.

@csernazs csernazs added bug needs triage Issue that has not been reviewed by xarray team member labels May 23, 2025
@benbovy
Copy link
Member

benbovy commented May 27, 2025

Hmm I don't see any specific change on the Xarray side that would have caused this issue.

However, the pandas behavior is not exactly the one that I would have expected:

>>> import pandas as pd
>>> pd.__version__
'2.2.3'
>>> midx = pd.MultiIndex.from_product([[100, 200, 300], [1, 2, 3, 4]], names=["x", "y"])
>>> y_idx = midx.levels[1]
>>> y_idx
Index([1, 2, 3, 4], dtype='int64', name='y')
>>> y_idx.name = "y"
RuntimeError: Cannot set name on a level of a MultiIndex. Use 'MultiIndex.set_names' instead.

It looks like midx.levels[1] does not return a copy but a view of the MultiIndex. I suspect it wasn't the case before and perhaps the change in behavior is related to pandas's copy on write migration?

Changing the line

pd_idx.name = k
to

if pd_idx.name != k:
    pd_idx = pd_idx.copy()
    pd_idx.name = k

should fix the issue.

@benbovy benbovy added topic-indexing and removed needs triage Issue that has not been reviewed by xarray team member labels May 27, 2025
@csernazs
Copy link
Author

hi @benbovy ,

Thanks for the fix. I confirm that the fix works.

Are you going to fix this in xarray (do you want the fix be merged)?

Looking into pandas side, they also provide set_names() method which works like this:

if pd_idx.name != k:
    pd_idx = pd_idx.set_names(k)

(it returns a copy, similar to your fix)

Thanks again,
Zsolt

@benbovy
Copy link
Member

benbovy commented May 30, 2025

pd_idx.set_names(k) won't work if pd_idx is a MultiIndex. Even for the latter, we set the pd_idx.name attribute such that it corresponds to the dimension of the Xarray coordinate variables created for the multi-index levels.

Are you going to fix this in xarray (do you want the fix be merged)?

Yes it would be nice to fix this in Xarray. Are you willing to submit a PR? I'm happy to review it.

@csernazs
Copy link
Author

Yes, I'm willing to submit the PR soon.

@csernazs
Copy link
Author

I've submitted a PR (#10381) for this.
Note that this is my first PR here, so if you spot anything wrong with it, I'm happy to receive comments / feedback.

Thank you,
Zsolt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants