Replace custom WKT, GeoKey and Zarr plumbing with existing Rust crates?

[robmarkcole]

RustGIS implements WKT parsing, GeoKey handling and Zarr codecs internally. Could these custom implementations be replaced with proj-wkt/proj-core, geotiff-core and zarrs?

[giswqs]

Thanks for the suggestion. Rather than guess, we tested all three crates against RustGIS's code: wasm builds, value-for-value comparisons against data written by zarr-python, pyproj and GDAL, and fuzzing with hostile input. In short: not yet for any of the three, though for different reasons, and the Zarr comparison led to #250.

Zarr → zarrs 0.23.14. It's the most capable of the three. On 7 stores written by zarr-python 3.2 and xarray (v2 and v3, blosc lz4/zstd, zlib, gzip, zstd, sharded), zarrs matched the expected values exactly on all 7, and our reader on 5: we couldn't read bit-shuffled blosc or sharded arrays. It also fits our browser design: it can list the chunk keys a slice needs, we fetch them asynchronously, and it decodes them from memory. Three things stop us switching:

• Its zstd and gzip decoders ignore the chunk's expected size. zstd allocates whatever the frame header claims, and gzip reads to the end. A 512 KB chunk file in a valid 16×16 float32 array made it allocate 16 GiB and abort the process. Only parsing every codec's headers ourselves would catch that, and RustGIS has a hard never-crash rule for files read from disk or the web.
• The default features don't build for wasm32-unknown-unknown. default-features = false does, but still pulls in C zstd and about 140 crates.
• It doesn't support numcodecs' standalone LZ4, which we read today, and it needs Rust 1.91 (we're on 1.90).

So #250 adds sharding and bit-shuffled blosc to our own reader instead. It now reads all 7 stores, plus edge cases: partial shards, missing inner chunks, the index at the start, big-endian data. Our decoders still bound every chunk by its size, so the same bomb is refused in 0.2 ms. If zarrs bounds its decoding by the chunk size, we'd happily look again.

WKT and transforms → proj-wkt / proj-core 0.11. We tested 58 coordinate systems, each as GDAL WKT1, ESRI WKT1, WKT1 without authority tags and WKT2, transforming sample points within each system's area of use and comparing with pyproj (PROJ 9.7.1):

Input	RustGIS: parsed / within 1 m	proj-core: parsed / within 1 m
WKT1 (GDAL)	57 / 45	41 / 37
ESRI .prj	48 / 41	40 / 10
WKT1, no authority	48 / 41	43 / 10
WKT2 2019	0 / 0	21 / 19
EPSG code alone	56 / 44	54 / 46

• proj-core implements Albers, Hotine, sinusoidal and others, but proj-wkt doesn't recognise their WKT1 names.
• It refuses to transform most ESRI .prj strings, because it won't assume a datum shift the WKT doesn't state.
• It gave some silently wrong answers: Web Mercator WKT without an authority came out 39.7 km off, and EPSG:21781 in WKT2 120 km off.
• It reads WKT2, which we can't yet. We're also 78–317 m off on ESRI WKT for datums that need a shift (we assume none) and can't do NAD27 without its grid. So there's work on our side too.
• Neither parser panicked on any of 2,233 truncations, deep nesting or junk input.
• The wasm size would grow from 1.37 MB to 1.87 MB.

We'll watch it. If those gaps close, replacing proj4rs with proj-core could make sense.

GeoKeys → geotiff-core 0.8.1. It's tidy and robust: no unsafe, and no panics on 500,000 random GeoKey directories. But it models a coordinate system only as EPSG codes and citation strings. For files with a user-defined projection, like the NLCD's Albers, it reports EPSG 32767 (GeoTIFF's "user-defined" marker) and none of the parameters needed to place the raster. Our code turns those parameters into a PROJ definition that matched GDAL to 6 decimal places on all 5 user-defined test files. It does handle rotated rasters (a model transformation matrix), which we refuse today, and that's worth fixing on our side.