Avoid extra read from disk when creating Pandas Index. #8893
Conversation
| @@ -632,7 +632,7 @@ def from_variables( | |||
| # the checks below. | |||
|
|
|||
| # preserve wrapped pd.Index (if any) | |||
| data = getattr(var._data, "array", var.data) | |||
| data = getattr(var._data, "array") if hasattr(var._data, "array") else var.data | |||
There was a problem hiding this comment.
can you explain what is going on here? Why does var.data trigger an additional load? Where does our cache mechanism come in here?
There was a problem hiding this comment.
var._data contains array(array(array(...))) . We load these on the first access to .data and .values
This is a regression from Anderson's work on indexing I think. Something in the pipeline was reading this earlier and so it was fine at this point. Now we must be more careful about loading and this line was loading it again.
There was a problem hiding this comment.
Am I being slow or do these two lines have exactly the same semantics?
There was a problem hiding this comment.
Yes but else conditions don't get evaluated unless necessary
There was a problem hiding this comment.
Ah sorry, I was being slow
There was a problem hiding this comment.
In penance I add a comment
| data = getattr(var._data, "array") if hasattr(var._data, "array") else var.data | |
| # accessing `.data` can load data from disk, so we only access if needed | |
| data = getattr(var._data, "array") if hasattr(var._data, "array") else var.data |
There was a problem hiding this comment.
Sorry the ternary thing is a bit cute :) The comment makes it better.
There was a problem hiding this comment.
hasattr(obj, name) is currently implemented as something similar to:
try:
getattr(obj, name)
return True
except AttributeError:
return Falseso properties will still be evaluated. If you don't want that, use inspect.getattr_static.
Edit: probably not relevant since _data is a simple attribute. But still good to know!
Co-authored-by: Maximilian Roos <5635139+max-sixty@users.noreply.github.com>
|
I'd like to make sure we only read a variable at most once in our backend test, potentially with some mocking but I won't have time to get to it soon. Thoughts on just merging as-is? |
|
My attempts to add some kind of generic mocking did not succeed. I'll punt that to a future date. |
* Avoid extra read from disk when creating Pandas Index. * Update xarray/core/indexes.py Co-authored-by: Maximilian Roos <5635139+max-sixty@users.noreply.github.com> --------- Co-authored-by: Maximilian Roos <5635139+max-sixty@users.noreply.github.com>
* main: Avoid auto creation of indexes in concat (#8872) Fix benchmark CI (#9013) Avoid extra read from disk when creating Pandas Index. (#8893) Add a benchmark to monitor performance for large dataset indexing (#9012) Zarr: Optimize `region="auto"` detection (#8997) Trigger CI only if code files are modified. (#9006) Fix for ruff 0.4.3 (#9007) Port negative frequency fix for `pandas.date_range` to `cftime_range` (#8999) Bump codecov/codecov-action from 4.3.0 to 4.3.1 in the actions group (#9004) Speed up localize (#8536) Simplify fast path (#9001) Add argument check_dims to assert_allclose to allow transposed inputs (#5733) (#8991) Fix syntax error in test related to cupy (#9000)
* backend-indexing: Trigger CI only if code files are modified. (pydata#9006) Enable explicit use of key tuples (instead of *Indexer objects) in indexing adapters and explicitly indexed arrays (pydata#8870) add `.oindex` and `.vindex` to `BackendArray` (pydata#8885) temporary enable CI triggers on feature branch Avoid auto creation of indexes in concat (pydata#8872) Fix benchmark CI (pydata#9013) Avoid extra read from disk when creating Pandas Index. (pydata#8893) Add a benchmark to monitor performance for large dataset indexing (pydata#9012) Zarr: Optimize `region="auto"` detection (pydata#8997) Trigger CI only if code files are modified. (pydata#9006) Fix for ruff 0.4.3 (pydata#9007) Port negative frequency fix for `pandas.date_range` to `cftime_range` (pydata#8999) Bump codecov/codecov-action from 4.3.0 to 4.3.1 in the actions group (pydata#9004) Speed up localize (pydata#8536) Simplify fast path (pydata#9001) Add argument check_dims to assert_allclose to allow transposed inputs (pydata#5733) (pydata#8991) Fix syntax error in test related to cupy (pydata#9000)
No description provided.