Convert `byte_no_cf` and `cf_nasa_4326` to Zarr #321

lnicola · 2022-10-19T06:21:13Z

I agree to follow the project's code of conduct.
I added an entry to CHANGES.md if knowledge of this change could be valuable to users.

Closes #239

I left alldatatypes.nc untouched because it's string arrays are not supported. Unfortunately, I had to disable these tests on pre-3.4 GDAL, where Zarr is not supported.

lnicola · 2022-10-19T06:22:00Z

CC @metasim this works on my computer, but will probably fail on others, depending on the GDAL version and configuration (I used ZSTD).

lnicola · 2022-10-19T06:57:01Z

I also tried HDF5 (NC 4), but crashes like crazy. I can put up another PR if you want to take a look.

metasim · 2022-10-19T11:46:47Z

@lnicola I'll give it a run on my setup today.

metasim · 2022-10-19T12:10:17Z

@lnicola I think I need a custom build of GDAL (hence your comment about ZSTD?)...

Test results on macOS 21.6.0 arm64

failures:

---- raster::mdarray::tests::test_datatype stdout ----
thread 'raster::mdarray::tests::test_datatype' panicked at 'called Result::unwrap() on an Err value: NullPointer { method_name: "GDALGroupOpenMDArray", msg: "Decompressor blosc not handled" }', src/raster/mdarray.rs:1007:14
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

---- raster::mdarray::tests::test_array_names stdout ----
thread 'raster::mdarray::tests::test_array_names' panicked at 'assertion failed: (left == right)
left: ["X", "Y"],
right: ["X", "Y", "byte_no_cf"]', src/raster/mdarray.rs:835:9

---- raster::mdarray::tests::test_dimension_name stdout ----
thread 'raster::mdarray::tests::test_dimension_name' panicked at 'called Result::unwrap() on an Err value: NullPointer { method_name: "GDALGroupOpenMDArray", msg: "Decompressor blosc not handled" }', src/raster/mdarray.rs:904:71

---- raster::mdarray::tests::test_attributes stdout ----
thread 'raster::mdarray::tests::test_attributes' panicked at 'called Result::unwrap() on an Err value: NullPointer { method_name: "GDALGroupOpenMDArray", msg: "Decompressor blosc not handled" }', src/raster/mdarray.rs:1097:14

---- raster::mdarray::tests::test_dimension_size stdout ----
thread 'raster::mdarray::tests::test_dimension_size' panicked at 'called Result::unwrap() on an Err value: NullPointer { method_name: "GDALGroupOpenMDArray", msg: "Decompressor blosc not handled" }', src/raster/mdarray.rs:927:71

---- programs::raster::mdimtranslate::tests::test_build_tiff_from_path stdout ----
thread 'programs::raster::mdimtranslate::tests::test_build_tiff_from_path' panicked at 'called Result::unwrap() on an Err value: NullPointer { method_name: "GDALMultiDimTranslate", msg: "Cannot find array /science/grids/imagingGeometry/lookAngle" }', src/programs/raster/mdimtranslate.rs:260:10

---- raster::mdarray::tests::test_spatial_ref stdout ----
thread 'raster::mdarray::tests::test_spatial_ref' panicked at 'called Result::unwrap() on an Err value: NullPointer { method_name: "GDALGroupOpenMDArray", msg: "Decompressor blosc not handled" }', src/raster/mdarray.rs:1031:14

---- raster::mdarray::tests::test_read_data stdout ----
thread 'raster::mdarray::tests::test_read_data' panicked at 'called Result::unwrap() on an Err value: NullPointer { method_name: "GDALGroupOpenMDArray", msg: "Decompressor blosc not handled" }', src/raster/mdarray.rs:951:14

---- raster::mdarray::tests::test_no_data_value stdout ----
thread 'raster::mdarray::tests::test_no_data_value' panicked at 'called Result::unwrap() on an Err value: NullPointer { method_name: "GDALGroupOpenMDArray", msg: "Decompressor blosc not handled" }', src/raster/mdarray.rs:1055:14

---- raster::mdarray::tests::test_n_elements stdout ----
thread 'raster::mdarray::tests::test_n_elements' panicked at 'called Result::unwrap() on an Err value: NullPointer { method_name: "GDALGroupOpenMDArray", msg: "Decompressor blosc not handled" }', src/raster/mdarray.rs:874:71

---- raster::mdarray::tests::test_n_dimension stdout ----
thread 'raster::mdarray::tests::test_n_dimension' panicked at 'called Result::unwrap() on an Err value: NullPointer { method_name: "GDALGroupOpenMDArray", msg: "Decompressor blosc not handled" }', src/raster/mdarray.rs:855:71

---- raster::mdarray::tests::test_stats stdout ----
thread 'raster::mdarray::tests::test_stats' panicked at 'called Result::unwrap() on an Err value: NullPointer { method_name: "GDALGroupOpenMDArray", msg: "Decompressor blosc not handled" }', src/raster/mdarray.rs:1168:71

---- raster::mdarray::tests::test_unit stdout ----
thread 'raster::mdarray::tests::test_unit' panicked at 'called Result::unwrap() on an Err value: NullPointer { method_name: "GDALGroupOpenMDArray", msg: "Decompressor blosc not handled" }', src/raster/mdarray.rs:1144:14

---- vector::vector_tests::tests::test_features_aliasing_compile_fail stdout ----
thread 'vector::vector_tests::tests::test_features_aliasing_compile_fail' panicked at '1 of 1 tests failed', /Users/sfitch/.cargo/registry/src/github.com-1ecc6299db9ec823/trybuild-1.0.55/src/run.rs:74:13

failures:
programs::raster::mdimtranslate::tests::test_build_tiff_from_path
raster::mdarray::tests::test_array_names
raster::mdarray::tests::test_attributes
raster::mdarray::tests::test_datatype
raster::mdarray::tests::test_dimension_name
raster::mdarray::tests::test_dimension_size
raster::mdarray::tests::test_n_dimension
raster::mdarray::tests::test_n_elements
raster::mdarray::tests::test_no_data_value
raster::mdarray::tests::test_read_data
raster::mdarray::tests::test_spatial_ref
raster::mdarray::tests::test_stats
raster::mdarray::tests::test_unit
vector::vector_tests::tests::test_features_aliasing_compile_fail

test result: FAILED. 184 passed; 14 failed; 0 ignored; 0 measured; 0 filtered out; finished in 3.53s

Versions:

gdal: 3.5.1
netcdf: 4.9.0
hdf5: 1.12.2

GDALVersionInfo {
    RELEASE_NAME: "3.5.1"
    RELEASE_DATE: "20220630"
    VERSION_NUM: "3050100"
    BUILD_INFO {
        OGR_ENABLED: "YES"
        PROJ_BUILD_VERSION: "9.0.1"
        GEOS_ENABLED: "YES"
        PAM_ENABLED: "YES"
        PROJ_RUNTIME_VERSION: "9.0.1"
        GEOS_VERSION: "3.11.0-CAPI-1.17.0"
    }
}

lnicola · 2022-10-19T12:16:21Z

Yeah, it seems so. How did you install GDAL?

metasim · 2022-10-19T15:55:09Z

brew install gdal

edit: deleted config quote because it's misleading

lnicola · 2022-10-19T15:57:08Z

I suppose you could try to install zstd (and recompile gdal), but I'll replace these with DEFLATEd ones when I get to my computer.

metasim · 2022-10-19T15:59:02Z

I'm fine rebuilding as we figure this out, but long term should figure out what the "standard" Linux install of GDAL is for most distros (e.g. rpm/dpkg/apt) is, and if it includes requisite libraries needed for this test.

metasim · 2022-10-19T15:59:58Z

Got pulled off on other things, but started looking into possibilities of adding something to VersionInfo that would provide more details about the drivers and what they are linked against.

lnicola · 2022-10-19T16:02:18Z

You could also check if gdalinfo works on those (needs vsizip).

I'm not sure what you mean by standard install. The tests pass on CI (on supported GDAL versions), and if they don't pass locally, that's still not so bad. The distros I've tested (Arch and the official container) have zstd and also blosc (you need that too, btw).

lnicola · 2022-10-19T16:04:57Z

Actually, we can leave these uncompressed. It's only 50 kb or so.

lnicola · 2022-10-19T17:41:41Z

Can you pull and try again?

metasim · 2022-10-19T18:32:52Z

🕺 🚀

lnicola · 2022-10-19T18:33:37Z

Cool. I suppose we could ignore those tests on GDAL pre-3.4.

metasim · 2022-10-19T18:34:50Z

I suppose we could ignore those tests on GDAL pre-3.4.

Yeh, I was already heading down that route in a draft PR.

lnicola · 2022-10-19T18:40:24Z

I'll try it tomorrow. I'm inclined to conditionally #[ignore] those tests using #[cfg_attr].

metasim · 2022-10-19T18:46:54Z

Supersedes #320

lnicola · 2022-10-21T12:57:55Z

bors r=metasim,jdroenner

bors · 2022-10-21T13:02:35Z

Build succeeded:

lnicola changed the title ~~Convert byte_no_cf and cf_nasa_4326 to Zarr~~ Convert byte_no_cf and cf_nasa_4326 to Zarr Oct 19, 2022

lnicola marked this pull request as draft October 19, 2022 06:22

Convert byte_no_cf and cf_nasa_4326 to Zarr

e6a6b4e

lnicola force-pushed the nc-to-zarr branch from 64dd9a7 to e6a6b4e Compare October 19, 2022 17:41

Ignore Zarr tests on pre-GDAL 3.4

d9b6dea

lnicola force-pushed the nc-to-zarr branch from 4ff9046 to d9b6dea Compare October 20, 2022 06:46

lnicola marked this pull request as ready for review October 20, 2022 06:56

Delete nc fixtures

a61bf52

bors bot merged commit c00ab44 into georust:master Oct 21, 2022

lnicola deleted the nc-to-zarr branch October 21, 2022 13:03

metasim mentioned this pull request Oct 21, 2022

Run NetCDF dependent tests in serial due to crashing on MacOS. #320

Closed

2 tasks

metasim mentioned this pull request Dec 10, 2022

Stack overflow in programs::raster::mdimtranslate::tests::test_build_tiff_from_path #299

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert `byte_no_cf` and `cf_nasa_4326` to Zarr #321

Convert `byte_no_cf` and `cf_nasa_4326` to Zarr #321

lnicola commented Oct 19, 2022 •

edited

lnicola commented Oct 19, 2022 •

edited

lnicola commented Oct 19, 2022

metasim commented Oct 19, 2022

metasim commented Oct 19, 2022

lnicola commented Oct 19, 2022

metasim commented Oct 19, 2022 •

edited

lnicola commented Oct 19, 2022

metasim commented Oct 19, 2022

metasim commented Oct 19, 2022

lnicola commented Oct 19, 2022

lnicola commented Oct 19, 2022

lnicola commented Oct 19, 2022

metasim commented Oct 19, 2022

lnicola commented Oct 19, 2022

metasim commented Oct 19, 2022

lnicola commented Oct 19, 2022

metasim commented Oct 19, 2022

lnicola commented Oct 21, 2022

bors bot commented Oct 21, 2022

Convert byte_no_cf and cf_nasa_4326 to Zarr #321

Convert byte_no_cf and cf_nasa_4326 to Zarr #321

Conversation

lnicola commented Oct 19, 2022 • edited

lnicola commented Oct 19, 2022 • edited

lnicola commented Oct 19, 2022

metasim commented Oct 19, 2022

metasim commented Oct 19, 2022

Versions:

lnicola commented Oct 19, 2022

metasim commented Oct 19, 2022 • edited

lnicola commented Oct 19, 2022

metasim commented Oct 19, 2022

metasim commented Oct 19, 2022

lnicola commented Oct 19, 2022

lnicola commented Oct 19, 2022

lnicola commented Oct 19, 2022

metasim commented Oct 19, 2022

lnicola commented Oct 19, 2022

metasim commented Oct 19, 2022

lnicola commented Oct 19, 2022

metasim commented Oct 19, 2022

lnicola commented Oct 21, 2022

bors bot commented Oct 21, 2022

Convert `byte_no_cf` and `cf_nasa_4326` to Zarr #321

Convert `byte_no_cf` and `cf_nasa_4326` to Zarr #321

lnicola commented Oct 19, 2022 •

edited

lnicola commented Oct 19, 2022 •

edited

metasim commented Oct 19, 2022 •

edited