loading ECCO v4r4 in zarr format #200
Conversation
all set for v0.2
making sure ecco_catalog branch is up to date
commenting out xarray_kwargs
removed chunk arg and removed xarray_kwargs arg
removed ECCO test data, updated grid for ECCO
correction for k_u
fixing Z dims
testing grid compatibility
adding grid
fixing time_snap to time_midp
dropping k index after swapping to Z for LLC data
All set to load ECCO v4r4 from /sciserver/oceanography/ECCO_v4r4
removing ECCO test data entry
asiddi24
requested review from
malmans2,
Mikejmnez,
renskegelderloos and
ThomasHaine
as code owners
|
Hello @asiddi24! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:
Comment last updated at 2022-02-14 20:47:07 UTC |
|
need to correct for failing tests |
|
What's the status of this PR? Can we fix and merge soon? I'm posting to the Poseidon website about ECCO functionality in OceanSpy+SciServer. |
|
@asiddi24 The error looks pretty easy to fix. It looks like it has to do with an incompatibility between the new line ds = ds.drop_vars(["k_p1", "k_u", "k_l", "k"])when creating the test ospy dataset, and the data (along with .yaml file) used for testing. The error you are getting is ValueError: One or more of the specified variables cannot be found in this dataset(you can see this in lines 11681-11689 of the CI/Build). I think the easier fix would be to update the data for testing (I used |
|
@asiddi24 Is this PR ready for merge? Once the failing tests are fixed (which seems to be a universal issue with all PRs at the moment)? |
ECCO v4r4 catalog entry update
change data directory for ECCOv4r4
|
tests are failing, but for now I'm going to go ahead and merge. This is so that we can test the new oceanography image. |
|
got it |
* Revert "loading ECCO v4r4 in zarr format (#200)" This reverts commit 8561d37. * Update catalog_xarray.yaml changes to catalog * Update datasets_list.yaml update ECCO_v4r4 * Update open_oceandataset.py commented out the dropping of the k variable name * Update datasets_list.yaml reverting back to ECCO * Update open_oceandataset.py fixed the white space Co-authored-by: Ali Hasan Siddiqui <43628137+asiddi24@users.noreply.github.com>
I've added the ECCO v4r4 intake catalog entry and removed the test data. Grid is not added separately since the dataset was stored with the grid added to it in the merged dataset. Added grid arguments though to the catalog entry.
The data loads super quick with the zarr format, granted it's about 145 GB. Following up on #168, there needs to be something of the sorts
xarray_kwargsin thexr.open_zarrfunctionality.All looks good. Will update #168 with changes as well.