Load selected variables instead of making them virtual #69
Conversation
virtualizarr/xarray.py
Outdated
| @@ -21,6 +21,7 @@ def open_virtual_dataset( | |||
| filepath: str, | |||
| filetype: Optional[str] = None, | |||
| drop_variables: Optional[List[str]] = None, | |||
| load_variables: Optional[List[str]] = None, | |||
There was a problem hiding this comment.
Not sure "load" is the best name for this kwarg, because these variables will still be lazy. But what else is the opposite of "virtual"? "Real" also has incorrect connotations (and "inline" makes no sense here).
There was a problem hiding this comment.
Renamed to "loadable", which is marginally better because it doesn't imply that they have already been loaded.
|
|
||
|
|
||
|
|
||
| def dataset_from_kerchunk_refs( |
There was a problem hiding this comment.
We're now only using this function in internal tests
| # TODO really we probably want a dedicated xarray backend that iterates over all variables only once | ||
| ds = xr.open_dataset(filepath, drop_variables=drop_variables) |
There was a problem hiding this comment.
At some point it might be nice to have a dedicated function / xarray backend that only opens the legacy file once, instead of opening it once with kerchunk and once with with xarray.
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #69 +/- ##
==========================================
- Coverage 90.57% 90.18% -0.40%
==========================================
Files 14 14
Lines 955 998 +43
==========================================
+ Hits 865 900 +35
- Misses 90 98 +8
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Towards implementing case (1) of #62, at least the opening step. We would still need to be able to save these loaded variables out (to kerchunk and zarr ideally).
Mostly works but there is a subtlety with the difference between aVariableand anIndexVariablecausing the test to fail.