feat: phase 2 — S1 GRD RTC GeoTIFF ingestion pipeline by emmanuelmathot · Pull Request #146 · EOPF-Explorer/data-model

emmanuelmathot · 2026-03-26T14:39:40Z

Summary

Phase 2 of the S1 GRD RTC implementation: production GeoTIFF → Zarr V3 ingestion pipeline for S1Tiling γ0T RTC outputs.

Delivered files

File	Description
`src/eopf_geozarr/conversion/s1_ingest.py` (~500 lines)	Production ingestion pipeline
`tests/test_s1_rtc_ingest.py` (27 tests)	Full test coverage

Public API

ingest_s1tiling_acquisition(vv, vh, mask, store, orbit) — create-or-append one acquisition
consolidate_s1_store(store, orbit) — post-batch consolidation
discover_s1tiling_acquisitions(input_dir) — file discovery with grouping
extract_geotiff_metadata(path) — rasterio-based metadata extraction

Key design decisions

Convention metadata built from zarr_cm CMO dicts (UUID, schema_url, spec_url)
Private _downsample_2d helper for overview generation (mean downsampling)
structlog for structured logging throughout
CRS + shape validation on append (rejects mismatched acquisitions)
1D spatial coordinate arrays (x, y) at every resolution level via np.linspace
Lowercase dimensions ["y", "x"] / ["time", "y", "x"] required by titiler-eopf

Test results

27/27 passed

Productionise the GeoTIFF → Zarr V3 ingestion pipeline for S1Tiling gamma-naught RTC outputs. New module: src/eopf_geozarr/conversion/s1_ingest.py (~500 lines) - extract_geotiff_metadata(): rasterio-based metadata extraction with tag validation and S1Tiling datetime normalisation - create_s1_store(): Zarr V3 store creation with full GeoZarr conventions (multiscales, proj:, spatial:), sharded arrays, 1D spatial coordinate arrays at every resolution level - ingest_s1tiling_acquisition(): main API — create-or-open store, read GeoTIFFs, generate overviews (average/nearest), write all levels, append coordinate variables. CRS/shape consistency validation on append. - consolidate_s1_store(): post-batch metadata consolidation - discover_s1tiling_acquisitions(): file discovery with grouping Tests: tests/test_s1_rtc_ingest.py (27 tests) - Metadata extraction, store creation, ingestion (create + append), data integrity, xarray roundtrip, CRS/shape mismatch rejection, consolidation, file discovery Uses zarr_cm CMO dicts for convention metadata (not hardcoded UUIDs). Reuses calculate_aligned_chunk_size() from utils.py. Exports wired in conversion/__init__.py. Refs: #139

…ords - Remove private _downsample_2d from s1_ingest.py; use shared downsample_2d_array from utils instead. - Enhance downsample_2d_array: add method='nearest' parameter; use ceil-based block sizes with edge-padding for non-divisible source/target ratios (no longer truncates edge pixels). - Fix spatial coordinate arrays to use pixel-center convention (half-pixel offset from edge origin) per CF/GIS standards. - Update test expectations for improved block averaging behavior.

Copilot

Pull request overview

Implements Phase 2 of the Sentinel-1 GRD RTC (γ0T) GeoTIFF → GeoZarr v3 ingestion pipeline, including store creation, acquisition appends, consolidation, and discovery, with accompanying tests.

Changes:

Added a production ingestion module for S1Tiling RTC GeoTIFFs into sharded Zarr v3 with multiscale overviews and conventions metadata.
Enhanced downsample_2d_array to support padded block-averaging for non-divisible shapes and a "nearest" method.
Added comprehensive ingestion tests and updated existing conversion tests to match the new downsampling behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
`src/eopf_geozarr/conversion/s1_ingest.py`	New S1 RTC ingestion pipeline (metadata extraction, store creation, append logic, consolidation, discovery).
`src/eopf_geozarr/conversion/utils.py`	Updates downsampling behavior and adds a resampling `method` option used by overviews.
`src/eopf_geozarr/conversion/__init__.py`	Exposes the new S1 ingestion functions as part of the conversion public API.
`tests/test_s1_rtc_ingest.py`	New test suite covering metadata extraction, store creation, ingestion/appends, consolidation, and discovery.
`tests/test_conversion.py`	Updates downsampling test expectations for the new padded block-averaging behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…ze orbit_dir - Extract _create_orbit_group helper to deduplicate orbit group creation - Add VH / border_mask alignment validation (CRS, transform, shape) before write - Add spatial_transform comparison on append (not just CRS + shape) - Normalize orbit_dir ASC→ascending / DES→descending in discover_s1tiling_acquisitions - Validate method param in downsample_2d_array, fix block-averaging condition (and→or) - Remove unused compute_multiscales_layout import in tests

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

… cross-orbit validation - Add x/y spatial coordinate arrays to S1RtcOverviewResolutionMembers and S1RtcNativeResolutionMembers (schema + fixture + test) - Use bounded chunk-aligned shard dimensions via calculate_shard_dimension (extracted from geozarr.py to utils.py as shared utility) - Validate CRS/shape/transform against existing orbit groups before creating a new orbit direction group in an existing store - Add schema validation test: ingested store validates against S1RtcRoot

Copilot

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

* feat: Phase 2 — S1 GRD RTC GeoTIFF ingestion pipeline Productionise the GeoTIFF → Zarr V3 ingestion pipeline for S1Tiling gamma-naught RTC outputs. New module: src/eopf_geozarr/conversion/s1_ingest.py (~500 lines) - extract_geotiff_metadata(): rasterio-based metadata extraction with tag validation and S1Tiling datetime normalisation - create_s1_store(): Zarr V3 store creation with full GeoZarr conventions (multiscales, proj:, spatial:), sharded arrays, 1D spatial coordinate arrays at every resolution level - ingest_s1tiling_acquisition(): main API — create-or-open store, read GeoTIFFs, generate overviews (average/nearest), write all levels, append coordinate variables. CRS/shape consistency validation on append. - consolidate_s1_store(): post-batch metadata consolidation - discover_s1tiling_acquisitions(): file discovery with grouping Tests: tests/test_s1_rtc_ingest.py (27 tests) - Metadata extraction, store creation, ingestion (create + append), data integrity, xarray roundtrip, CRS/shape mismatch rejection, consolidation, file discovery Uses zarr_cm CMO dicts for convention metadata (not hardcoded UUIDs). Reuses calculate_aligned_chunk_size() from utils.py. Exports wired in conversion/__init__.py. Refs: #139 * remove plan * refactor: address PR review — consolidate downsample, pixel-center coords - Remove private _downsample_2d from s1_ingest.py; use shared downsample_2d_array from utils instead. - Enhance downsample_2d_array: add method='nearest' parameter; use ceil-based block sizes with edge-padding for non-divisible source/target ratios (no longer truncates edge pixels). - Fix spatial coordinate arrays to use pixel-center convention (half-pixel offset from edge origin) per CF/GIS standards. - Update test expectations for improved block averaging behavior. * refactor: address PR review — extract helper, add validation, normalize orbit_dir - Extract _create_orbit_group helper to deduplicate orbit group creation - Add VH / border_mask alignment validation (CRS, transform, shape) before write - Add spatial_transform comparison on append (not just CRS + shape) - Normalize orbit_dir ASC→ascending / DES→descending in discover_s1tiling_acquisitions - Validate method param in downsample_2d_array, fix block-averaging condition (and→or) - Remove unused compute_multiscales_layout import in tests * refactor: address review round 2 — schema x/y coords, bounded shards, cross-orbit validation - Add x/y spatial coordinate arrays to S1RtcOverviewResolutionMembers and S1RtcNativeResolutionMembers (schema + fixture + test) - Use bounded chunk-aligned shard dimensions via calculate_shard_dimension (extracted from geozarr.py to utils.py as shared utility) - Validate CRS/shape/transform against existing orbit groups before creating a new orbit direction group in an existing store - Add schema validation test: ingested store validates against S1RtcRoot

emmanuelmathot changed the title ~~feat: Phase 2 — S1 GRD RTC GeoTIFF ingestion pipeline~~ feat: phase 2 — S1 GRD RTC GeoTIFF ingestion pipeline Mar 26, 2026

emmanuelmathot force-pushed the feat/s1-rtc-geotiff-ingestion branch from b368033 to 37769c3 Compare March 26, 2026 14:42

emmanuelmathot mentioned this pull request Mar 26, 2026

Sentinel-1 GRD γ0T RTC GeoZarr V3 — Implementation Tracker #139

Open

6 tasks

emmanuelmathot marked this pull request as ready for review March 26, 2026 14:44

remove plan

2221d78

d-v-b reviewed Mar 30, 2026

View reviewed changes

Comment thread src/eopf_geozarr/conversion/s1_ingest.py Outdated

d-v-b reviewed Mar 30, 2026

View reviewed changes

Comment thread src/eopf_geozarr/conversion/s1_ingest.py Outdated

emmanuelmathot requested review from Copilot and d-v-b April 2, 2026 12:59

Copilot AI reviewed Apr 2, 2026

View reviewed changes

emmanuelmathot requested a review from Copilot April 2, 2026 14:17

Copilot started reviewing on behalf of emmanuelmathot April 2, 2026 14:17 View session

Copilot AI reviewed Apr 2, 2026

View reviewed changes

Comment thread src/eopf_geozarr/conversion/s1_ingest.py

Comment thread src/eopf_geozarr/conversion/s1_ingest.py Outdated

Comment thread src/eopf_geozarr/conversion/s1_ingest.py

Comment thread tests/test_s1_rtc_ingest.py

Comment thread tests/test_s1_rtc_ingest.py

emmanuelmathot force-pushed the feat/s1-rtc-geotiff-ingestion branch from c0bc6d0 to d7147a5 Compare April 2, 2026 14:43

emmanuelmathot requested a review from Copilot April 2, 2026 14:44

Copilot started reviewing on behalf of emmanuelmathot April 2, 2026 14:45 View session

Copilot AI reviewed Apr 2, 2026

View reviewed changes

Comment thread src/eopf_geozarr/conversion/s1_ingest.py

emmanuelmathot merged commit d255437 into s1-rtc Apr 2, 2026
4 checks passed

emmanuelmathot deleted the feat/s1-rtc-geotiff-ingestion branch April 2, 2026 14:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: phase 2 — S1 GRD RTC GeoTIFF ingestion pipeline#146

feat: phase 2 — S1 GRD RTC GeoTIFF ingestion pipeline#146
emmanuelmathot merged 5 commits intos1-rtcfrom
feat/s1-rtc-geotiff-ingestion

emmanuelmathot commented Mar 26, 2026

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

emmanuelmathot commented Mar 26, 2026

Summary

Delivered files

Public API

Key design decisions

Test results

Related

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants