Skip to content

v0.9.0

Latest

Choose a tag to compare

@github-actions github-actions released this 15 Nov 04:51
· 85 commits to main since this release
5dcf599

Highlights

  • GeoParquet at Arrow speed: GeoParquet reads run in parallel and lazily, and read in place over HTTP instead of downloading the file. The Arrow export hands out the file's record batches without converting them to features. On 12 cores, fiboa's 664,000-polygon es-nc file reads into a pyarrow table in 0.79 s and 761 MB, against 1.56 s and 966 MB through GDAL 3.13's Arrow stream and 2.33 s and 2.0 GB before. A small rectangle of a 533 MB Overture buildings file on S3 fetches 4.9 MB in 5 requests. terrio.pyogrio gains read_arrow and open_arrow, and read_dataframe(..., use_arrow=True) works (#733, #742, #745, #869).
  • Faster remote and threaded reads:
    • A COG window's separate byte runs are fetched in parallel, and long contiguous runs are split into parallel pieces (#687, #741). Remote Zarr keeps up to 32 requests in flight (#686), and remote COG tiles are decoded on the thread pool (#898).
    • A lazy Raster can be read from several Python threads at once, and warp and WarpedVRT release the GIL (#688, #692, #754).
    • A reduced read of a VRT uses its sources' overviews: a 256 x 256 read of a VRT over a 16384 x 16384 COG takes 0.05 s instead of 23 to 28 s (#796).
    • terrio.rio.shutil.copy writes a COG as its source is read, with a smaller memory peak (#797). GeoPackage pages decode only the requested records (#893).
  • terrio-js: terrio-js/zarr reads Zarr arrays over HTTP in a 0.19 MB (Brotli) module, against 2.56 MB for terrio-js/core (#900). Remote reads keep 12 requests in flight in a sliding window and open large-header files in fewer round trips: 128 bands of a TESSERA window take 2.29 s instead of 4.86 s (#871, #872, #873, #876, #1017). toGeoJSON of 90k polygons takes 0.65 s instead of 3.6 s (#875).
  • terrio-crs 0.2.4: standalone GeoTIFF GeoKeys (from_geokeys, to_geokeys) (#981), and PROJ strings and strict WKT 1 exported as PROJ 9.8.1 writes them, also as terrio crs --proj4 and --wkt1-strict and in Python and JavaScript (#958, #974). Also new: Transformer::inverse and slice transforms (#937), axis order, projection parameters and celestial body (#993), and Eckert IV and Natural Earth without the fallback feature (#988).
  • Safer handling of untrusted input: driver allow-lists cover point clouds and fail closed (#751, #829). Datum grids can be confined to their search directories (#781, #895, #1001). TERRIO_HTTP_FOLLOW_CROSS_ORIGIN=NO keeps reads on their origin (#793). Declared pixel and band budgets refuse oversized rasters before allocating (#862). Duplicate ZIP members are refused (#846), and diagnostics redact credentials (#998, #1000, #1002). TERRIO_HTTP_CAINFO adds a CA bundle (#1011).
  • Warps you can tile: warped values no longer depend on output tiles, VRT windows or memory chunks, for every resampler (#847). Pixels across the antimeridian, on the 180° meridian and around the poles take the right source, for area methods and sum too (#870, #917, #933, #959, #962, #972, #980, #1016).
  • Writers: ZLEVEL and ZSTD_LEVEL take effect (#827), and Deflate is much smaller on categorical rasters and repeated rows (#858, #861). Also NBITS for unsigned samples (#860), JPEG 2000 QUALITY and REVERSIBLE (#837), lossy standalone WebP (#836), and complex rasters through every write path (#841).
  • Bindings and CLI: terrio.pyogrio reads every layer of a multi-layer source (#830). Feature ranges hold for GeoPackage and every streamed reader (#795, #857). GeoJSON, TopoJSON and GML no longer drop a CRS silently (#923, #967, #968). Multi-input commands expand * and ? themselves, and terrio ui gains Add matching files and Windows drive navigation (#776, #928).
  • Behavior changes:
    • An empty TERRIO_ALLOWED_DRIVERS allows no driver; unset it to allow all (#751).
    • Native GML byte output is standalone XML, not a ZIP; MapInfo byte output takes .mif.zip or .tab.zip names (#851).
    • Python: terrio.rio COG writers report COG as their driver (#840), and oversized window reads raise ResourceLimitError (a MemoryError) instead of RasterioIOError (#862).
    • Rust API: FlatGeobufInfo has new public fields (#880), transform_bounds samples 23 points per edge (#908), and a missing velocity grid gives MissingGrid (#937).
    • Krovak WKT 1 without axes defaults to south-west, and WKT projected CRSs follow their AXIS nodes (#1015).

See CHANGELOG.md for the full list.

What's Changed

  • docs(playground): update terrio-js to 0.8.0 by @giswqs in #690
  • perf(http): fetch the separate byte runs of a prefetch in parallel by @nkarasiak in #687
  • perf(zarr): keep up to 32 requests in flight on remote stores, whatever the cores by @nkarasiak in #686
  • perf(python): warp a Raster in memory without the GIL by @nkarasiak in #688
  • test(http): skip the fork-during-remote-reads test on macOS by @giswqs in #695
  • fix(http): a failed parallel run no longer waits for a stalled request by @nkarasiak in #697
  • perf(rio): read and warp a WarpedVRT's source window without the GIL by @gmoralessanchez in #692
  • perf(mbtiles): read MBTiles archives in place, each tile by an index lookup by @giswqs in #694
  • perf(tiff): reuse worker decoders across window reads by @icbw in #700
  • perf(geoparquet): decode only the columns of the fields selected by @nkarasiak in #745
  • fix(gdal): build the gdal feature without the default features by @gmoralessanchez in #735
  • perf(http): fetch a long range in parallel pieces by @nkarasiak in #741
  • fix(xarray): the .terrio accessor leaves no reference cycle by @nkarasiak in #746
  • fix(rio): keep PixelIsPoint when terrio.rio writes a GeoTIFF or COG by @gmoralessanchez in #743
  • perf(parquet): parallel, lazy GeoParquet reads, read in place over HTTP by @nkarasiak in #733
  • fix(crs): a polar stereographic CRS projects its own pole, not the opposite one by @gmoralessanchez in #734
  • feat(pyogrio): read_arrow and open_arrow for GeoParquet by @nkarasiak in #742
  • fix(rio): write .jp2 by extension, and every JPEG 2000 driver name as JP2 by @gmoralessanchez in #736
  • fix: read no .ovr sidecar under TERRIO_DISABLE_READDIR_ON_OPEN=EMPTY_DIR by @gmoralessanchez in #744
  • fix(rio): keep the color interpretation of Byte COGs and GeoTIFF updates by @gmoralessanchez in #747
  • fix: refuse a query URL no allowed driver could read before requesting it by @gmoralessanchez in #749
  • fix(vrt): read fractional source windows within SrcRect and DstRect by @gmoralessanchez in #750
  • perf(python): read one lazy Raster from several threads at once by @nkarasiak in #754
  • ci: write the changelog in the release pull request only by @giswqs in #765
  • perf(kml): make a document's layer names unique in linear time by @gmoralessanchez in #763
  • build(deps): drop unused dependencies and trim minimal builds by @BenDyson-Arch in #758
  • fix: fail closed for empty driver lists and narrow calling-thread scopes by @gmoralessanchez in #751
  • fix(cog): spool a COG's tiles next to the output, not in the system temp directory by @giswqs in #770
  • fix: read local GeoJSON and GML under a name of no driver by content by @gmoralessanchez in #760
  • perf(tiff): share one decode-thread budget across concurrent window reads by @nkarasiak in #755
  • build(deps): bump rust from 7e9e2d0 to 44819b2 in the docker group by @dependabot[bot] in #771
  • fix(netcdf): list netCDF-4 variables and groups in GDAL's order by @nkarasiak in #752
  • fix(zonal): open and read raster paths under the caller's options by @gmoralessanchez in #769
  • build(deps): bump the actions group with 4 updates by @dependabot[bot] in #775
  • build(deps): bump the cargo group with 4 updates by @dependabot[bot] in #772
  • perf(warp): sum a coarse chunk's source window a strip of rows at a time by @gmoralessanchez in #768
  • fix: write no GeoJP2 box or world file for a raster without georeferencing by @gmoralessanchez in #753
  • fix(shapefile): a .prj that holds no CRS definition gives no CRS by @gmoralessanchez in #762
  • feat(cli): expand input wildcards and add matching files in terrio ui by @giswqs in #776
  • build(deps): bump lzma-rust2 from 0.24.0 to 0.25.0 by @dependabot[bot] in #774
  • build(deps): bump rstar from 0.13.0 to 0.14.0 by @dependabot[bot] in #773
  • fix(warp): area methods take the source inside edge pixels, whatever the grid by @gmoralessanchez in #756
  • test(cog): resolve the temp dir link in the dangling-link spool test by @giswqs in #777
  • feat(osm): decode PBF blocks in-house instead of osmpbfreader by @BenDyson-Arch in #759
  • ci(docs): isolate checks and serialize Pages publishing by @giswqs in #780
  • fix(rio): multiply affine transforms without changing the warning filters by @gmoralessanchez in #778
  • fix: apply the feature range to files the wbvector fallback reads by @gmoralessanchez in #791
  • fix(gpkg): keep the feature range when reading a GeoPackage by @gmoralessanchez in #795
  • feat(tiff): set the chunk decoding limit with TERRIO_TIFF_MAX_CHUNK_BYTES by @icbw in #799
  • feat(http): keep reads on their origin with TERRIO_HTTP_FOLLOW_CROSS_ORIGIN=NO by @gmoralessanchez in #793
  • fix(crs): confine grid reads to the search paths set from code, and to regular files by @giswqs in #781
  • fix(vrt): read a VRT's reduced resolution from its sources' overviews by @gmoralessanchez in #796
  • perf(rio): write a copy from its source as it is written by @gmoralessanchez in #797
  • fix(kmz): read the first KML entry at an archive's root, refuse duplicates by @gmoralessanchez in #789
  • test(rio): allow Windows spool files and open-file replace in copy tests by @giswqs in #802
  • test(rio): allow Windows spool files and VRT-over-source limit by @giswqs in #806
  • test(python): register GDAL's drivers before rasterio's bundled GDAL starts by @giswqs in #820
  • fix(crs): read Polar Stereographic (variant C), EPSG:2985 and 2986 by @gmoralessanchez in #803
  • fix(pointcloud): read EPT nodes under the calling thread's options by @gmoralessanchez in #804
  • fix(rio): leave a copy's destination as it was when its source fails to read by @gmoralessanchez in #821
  • fix(http): count uploads and POST bodies as sent bytes by @gmoralessanchez in #823
  • build(fuzz): regenerate fuzz/Cargo.lock and check it in CI by @gmoralessanchez in #824
  • fix: honour ZLEVEL for PNG and GeoTIFF Deflate, and ZSTD_LEVEL with libzstd by @gmoralessanchez in #827
  • fix(crs): refuse coordinates beyond the polar Lambert azimuthal disk by @gmoralessanchez in #822
  • Improve performance of sparse reads by @tmillenaar in #828
  • fix(formats): restrict point clouds with set_allowed_drivers by @gmoralessanchez in #829
  • fix(pyogrio): list and read every layer of a source with several by @gmoralessanchez in #830
  • fix(translate): resample a fractional window read at its own size by @gmoralessanchez in #834
  • fix(python): attribute native warnings to public callers by @giswqs in #843
  • fix(ept): enforce driver policy before EPT appends by @giswqs in #839
  • fix(zip): reject archives with duplicate member names by @giswqs in #846
  • fix(crs): refuse non-finite inverse projection inputs by @giswqs in #845
  • fix(tiff): preserve lazy metadata read failures by @giswqs in #838
  • fix(warp): retain values across aligned tiles and read windows by @giswqs in #847
  • Fix vector byte output packaging for GML and MapInfo by @giswqs in #851
  • fix(deflate): retain long matches across periodic raster runs by @giswqs in #858
  • docs(warp): explain nodata bias in overview sum estimates by @giswqs in #835
  • fix(pyogrio): support signed buffers and preserve warning filters by @giswqs in #848
  • fix(webp): honor lossy quality and lossless creation options by @giswqs in #836
  • perf(tiff): keep the spare decoders of all open rasters within one byte budget by @icbw in #856
  • fix(jp2): honor quality layers and reversible encoding options by @giswqs in #837
  • perf(zarr): unshuffle Blosc bits in 8 x 8 blocks by @nkarasiak in #866
  • fix(deflate): code a chunk's repeated rows as copies of the row above by @gmoralessanchez in #861
  • test(jp2): allow one-step 9/7 rounding on macOS arm64 by @giswqs in #877
  • fix(rio): normalize raster driver aliases before writer policy by @giswqs in #840
  • perf(zarr): fill a region's missing chunks by doubling, not one element at a time by @nkarasiak in #867
  • perf(zarr): reorder a window row by row when columns are the last dimension by @nkarasiak in #868
  • perf(geoparquet): decode WKB geometries straight from the column's bytes by @nkarasiak in #869
  • perf(js): read remote vector layers without copying them when no bbox is given by @nkarasiak in #874
  • fix(gdal): retain fractional seconds and explicit UTC zones by @giswqs in #850
  • test(warp): guard complete polar-cap coverage through API and CLI by @giswqs in #859
  • perf(js): make toGeoJSON's geometries in JavaScript, its other text parsed in chunks by @nkarasiak in #875
  • perf(zarr-js): split shard reads to fill the requests in flight by @nkarasiak in #873
  • perf(zarr): ask for the array's zarr.json in the first round at open by @nkarasiak in #872
  • perf(js): grow requests while a remote file opens by @nkarasiak in #876
  • fix(warp): area and sum footprints take the source on both sides of 180° by @gmoralessanchez in #870
  • fix(jp2): preserve channel roles and stored component order by @giswqs in #849
  • fix(flatgeobuf): report declared Z and M in export_schema by @brownag in #880
  • test(http): serialize the fetch statistics tests with the reset test by @brownag in #890
  • fix(shapefile): diagnose unreadable projection sidecars by @giswqs in #853
  • fix(raster): refuse what a raster declares and reads the process cannot hold by @gmoralessanchez in #862
  • docs: clarify projection-sidecar diagnostics by @giswqs in #891
  • fix(test): keep real-data grids independent of PROJ directories by @brownag in #889
  • fix(crs): recognize engineering and long vertical WKT roots by @giswqs in #892
  • fix(kmz): make layer names unique across documents by @giswqs in #854
  • perf(js): keep remote requests in flight in a sliding window by @nkarasiak in #871
  • fix(tiff): retain reduced-resolution first-image masks by @giswqs in #842
  • fix(raster): preserve complex samples through writes and translation by @giswqs in #841
  • fix(shapefile): keep DBF record allocations on the caller by @giswqs in #894
  • perf(cog): copy JPEG tiles and pixel-interleaved chunks without per-sample loops by @nkarasiak in #896
  • fix(crs): refuse special files and confine MCP grid lookup by @giswqs in #895
  • test(terrio): build 90 integration test files as one binary (rebuild 4x faster) by @nkarasiak in #899
  • perf(gpkg): decode only requested feature pages by @giswqs in #893
  • fix(vector): honor source feature ranges across readers by @giswqs in #857
  • fix(crs): retain both components of compound OGC URNs by @giswqs in #852
  • fix(crs): preserve projection parameters and datum exports by @giswqs in #844
  • fix(vector): validate stored CRS and keep dataframe exports usable by @giswqs in #901
  • test(terrio): register streamed vector ranges as a test target by @giswqs in #902
  • feat(tiff): write packed unsigned samples and preserve copy widths by @giswqs in #860
  • perf(cog): decode remote COG tiles on the thread pool by @nkarasiak in #898
  • feat(js): terrio-js/zarr, a Zarr-only WebAssembly module (0.19 MB Brotli) by @nkarasiak in #900
  • fix(lint): satisfy Clippy from Rust 1.100 by @giswqs in #905
  • fix(crs): include interior poles in transformed bounds by @giswqs in #908
  • test(jp2): pin irreversible decoding to fixed Linux references by @giswqs in #909
  • test(http): cover fork lock hand-off on macOS by @giswqs in #916
  • fix(python): validate CRS metadata in feature stream exports by @giswqs in #919
  • fix(pyogrio): interpret mixed naive datetimes as UTC on pandas 2 by @giswqs in #920
  • fix(webp): reject malformed quality values by @giswqs in #925
  • fix(csv): default TSV writers to tab separators by @giswqs in #922
  • fix(rio): retain failed points in one transform batch by @giswqs in #927
  • fix(python): emit warnings without a Python caller frame by @giswqs in #918
  • fix(mif): reject content without a complete header by @giswqs in #926
  • fix(gtiff): retain engineering CRS definitions by @giswqs in #924
  • fix(geojson): name retained authorities and refuse silent CRS loss by @giswqs in #923
  • fix(pyogrio): identify archive and text vector buffers by @giswqs in #929
  • fix(warp): write the polar cap under the chunk-dependent policy by @gmoralessanchez in #917
  • fix(config): resolve option aliases by scope before name by @giswqs in #934
  • fix(warp): return empty arrays for zero-sized regions by @giswqs in #935
  • fix(warp): sample full-turn geographic rasters periodically by @giswqs in #933
  • fix(rio): expose cached TIFF block offsets and sizes by @giswqs in #930
  • terrio-crs: Transformer::inverse and slice transforms by @pick2510 in #937
  • fix(ui): reach other drives from the file browser on Windows by @giswqs in #928
  • fix(csv): preserve values and geometry when column names collide by @giswqs in #951
  • fix(shapefile): preserve dotted basenames and matching sidecars by @giswqs in #949
  • fix(raster): retain stored CRS definitions before EPSG summaries by @giswqs in #954
  • fix(jpeg): validate complete quality values for images and tiles by @giswqs in #955
  • fix(python): preserve successful reads during interpreter shutdown by @giswqs in #956
  • fix(raw): size a window's row list only when it has samples by @nkarasiak in #963
  • fix(warp): keep meridian-corner kernels and sums local by @giswqs in #959
  • terrio-crs: read Paris in degrees in WKT 1 in grads, and ESRI's Behrmann by @pick2510 in #961
  • fix(python): validate stored GeoParquet CRS in Arrow exports by @giswqs in #965
  • fix(raster): open extensionless PNG JPEG and TIFF by content by @giswqs in #964
  • fix(warp): retain meridian sides of polar-edge source windows by @giswqs in #962
  • terrio-crs: write PROJ strings as PROJ 9.8.1 does (SpatialRef::to_proj4) by @pick2510 in #958
  • fix(json): preserve CRS meaning in TopoJSON and GeoJSON by @giswqs in #967
  • fuzz: open ASCII grid, XYZ, Erdas Imagine and HDF5 bytes by @nkarasiak in #957
  • fix(errors): name a missing or unrecognized input once by @nkarasiak in #960
  • fix(vector): retain buffer filenames and refuse layerless archives by @giswqs in #966
  • fix(vector): name GML authorities and warn on CSV CRS loss by @giswqs in #968
  • fix(warp): read regional sources at equivalent longitudes by @giswqs in #972
  • Preserve every declared duplicate vector column by @giswqs in #978
  • Preserve CSV WKT geometry beside coordinate attributes by @giswqs in #975
  • Reject wrong document kinds in XML and GeoJSON sequence readers by @giswqs in #976
  • Replace Shapefile sidecars across extension case variants by @giswqs in #977
  • Fix lossy WebP token alignment for skipped macroblocks by @giswqs in #979
  • terrio-crs: add SpatialRef::to_wkt1_strict, WKT 1 without the PROJ4 extension by @pick2510 in #974
  • terrio-crs: read and write GeoTIFF GeoKeys (SpatialRef::from_geokeys, to_geokeys) by @pick2510 in #981
  • test(crs): run the to_geokeys doctest on a larger stack by @giswqs in #987
  • Fix wider-source footprints needing two placements by @giswqs in #980
  • terrio-crs: Eckert IV and Natural Earth in the default build by @pick2510 in #988
  • terrio-crs: axis order, conversion parameters and celestial body accessors by @pick2510 in #993
  • terrio-crs: drop serde_json's preserve_order, keep PROJJSON in PROJ's key order by @pick2510 in #994
  • Transformer: dropping one at thread exit no longer aborts the process by @pick2510 in #995
  • terrio-crs: datum shifts no longer fold latitudes beyond the poles back by @pick2510 in #996
  • Fix main CI after the untrusted-input hardening merge by @giswqs in #998
  • terrio-crs: refuse PROJ-string parameter values PROJ refuses; read radians and +units as PROJ does by @pick2510 in #997
  • fix: catalog VRT bytes, Python stub all and scheme-less redaction by @giswqs in #999
  • Redact credentials after URL-valued parameters; keep declared VRT bytes in the source map by @giswqs in #1000
  • fix: keep non-credential schemeless queries and refuse unmapped VRT bytes by @giswqs in #1005
  • fix: reject unresolved JSON CRS names and warn for GML by @giswqs in #1009
  • fix: confine catalog VRT bytes and preserve literal paths by @giswqs in #1002
  • fix: identify PROJJSON definitions and accept untyped bases by @giswqs in #1003
  • fix: preserve repeated GeoParquet columns in Arrow exports by @giswqs in #1004
  • fix: retain validated CRS across registered Arrow types by @giswqs in #1007
  • test: compare PROJ parameter differences line by line by @giswqs in #1008
  • fix: expose Python grid confinement and report grid refusals by @giswqs in #1001
  • fix: name archive errors after caller-visible inputs by @giswqs in #1006
  • fix: cancel stalled Zarr requests after a chunk fails by @giswqs in #1013
  • fix: keep polar area resampling inside its local footprint by @giswqs in #1016
  • fix: support additional CA trust for native HTTPS reads by @giswqs in #1011
  • fix: cancel and join parallel range reads before fork by @giswqs in #1014
  • perf(js): fetch long tile runs in parallel pieces by @giswqs in #1017
  • test: connect CONNECT proxy upstream over IPv4 loopback on Windows by @giswqs in #1018
  • test(python): connect the CONNECT test proxy over IPv4 loopback by @giswqs in #1019
  • terrio-crs: projected CRSs given by WKT have the axes of their AXIS nodes by @pick2510 in #1015
  • Release 0.9.0 by @giswqs in #1020

New Contributors

Full Changelog: v0.8.0...v0.9.0

Credits

terrio does not need GDAL to be installed, but it is not independent of GDAL: its commands, options and behavior follow GDAL's, many of its algorithms and drivers are ports of GDAL code, its results are tested against GDAL, PROJ and PDAL, and GDAL is its optional backend. This release ports code from, adapts, or is built on:

  • GDAL (Frank Warmerdam, Even Rouault and hundreds of contributors): the design of terrio's commands and the reference it is tested against. New in this release: GeoTIFF GeoKeys are read and written as GDAL does and checked against GDAL 3.12.1's, GDAL_HTTP_CAINFO, the JPEG 2000 quality convention, and GeoParquet Arrow reads matching GDAL 3.13.3's Arrow stream.
  • PROJ: the map projections and datum transformations. New in this release: Eckert IV and Natural Earth are ported from PROJ 9.8.1's eck4.cpp and natearth.cpp. PROJ strings and strict WKT 1 are written as PROJ 9.8.1 writes them. Axis orders, conversion parameters and the celestial body table come from PROJ 9.8.1's database, and PROJ-string parameters and datum shifts beyond the poles follow PROJ.
  • EPSG Geodetic Parameter Dataset (IOGP), Esri projection engine data and the IAU_2015 definitions PROJ carries: the reference tables behind the new exports, accessors and celestial bodies.
  • ureq by Martin Algesten: New in this release: the rustls connector that adds a CA bundle to the Mozilla roots (MIT).
  • webp-rust by MITH@mmk: New in this release: the lossy VP8 encoder behind lossy WebP writes, with a corrected skip-flag grammar (MIT).
  • pyogrio, rasterio, rioxarray and xarray: the APIs terrio.pyogrio, terrio.rio and terrio.xarray follow. New in this release: read_arrow and open_arrow.
  • zarrita and numcodecs: the reference terrio-js/zarr is measured against.
  • whitebox_next_gen by John Lindsay: the pure-Rust format libraries terrio began on. New in this release: declared header layouts adapted from wbraster 0.2.1.

Thanks to everyone who contributed code to this release: @BenDyson-Arch, @brownag, @gmoralessanchez, @icbw, @nkarasiak, @pick2510 and @tmillenaar.

Copyright and license notices are in NOTICE, and the acknowledgements page lists every project. If terrio is useful to you, please consider sponsoring GDAL.