You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
For a Variable whose ._data is already an in-memory numpy.ndarray, Variable.load() does no useful work — it walks a dispatch chain that always returns the same array unchanged. The cost is paid once per variable inside Dataset.load() / DataArray.load(), on top of (and after) is_chunked_array has already concluded the variable isn't chunked.
isinstance(data, ExplicitlyIndexed | ImplicitToExplicitIndexingAdapter) → False (np.ndarray is neither)
is_duck_array(data) → True → returns data unchanged
So self._data = to_duck_array(self._data) reduces to self._data = self._data. The entire call is dispatch overhead.
Why it matters
Dataset.load() calls Variable.load() per variable. On many-variable datasets — common with open_mfdataset / concat results — this overhead compounds. With #11351 the is_chunked_array portion is near-free, so this dispatch is the remaining cost on the same hot path.
Proposed fix
Add a numpy fast-path at the top of Variable.load() and Variable.load_async():
What
For a
Variablewhose._datais already an in-memorynumpy.ndarray,Variable.load()does no useful work — it walks a dispatch chain that always returns the same array unchanged. The cost is paid once per variable insideDataset.load()/DataArray.load(), on top of (and after)is_chunked_arrayhas already concluded the variable isn't chunked.Code path
Variable.load(xarray/core/variable.py:1022):to_duck_array(xarray/namedarray/pycompat.py:139):For
np.ndarray:is_chunked_array(data)→False(after Speed up Dataset.load for in-memory datasets with many variables #11351 lands, this is now fast)isinstance(data, ExplicitlyIndexed | ImplicitToExplicitIndexingAdapter)→False(np.ndarrayis neither)is_duck_array(data)→True→ returnsdataunchangedSo
self._data = to_duck_array(self._data)reduces toself._data = self._data. The entire call is dispatch overhead.Why it matters
Dataset.load()callsVariable.load()per variable. On many-variable datasets — common withopen_mfdataset/concatresults — this overhead compounds. With #11351 theis_chunked_arrayportion is near-free, so this dispatch is the remaining cost on the same hot path.Proposed fix
Add a numpy fast-path at the top of
Variable.load()andVariable.load_async():(Happy to open a follow-up PR. Filing here so the design can be discussed independently of the implementation.)
[This is Claude Code on behalf of Felix Bumann]