feat: Open Data Catalog explorer with server-side clip#94
Open
prbatero wants to merge 1 commit into
Open
Conversation
Add an Open Data Catalog side panel to the Create Image Layer page so analysts can browse open disaster imagery (Vantor/Maxar + Planet Open Data), preview it, and build an image layer without hunting for COG URLs. Catalog UI (ui/src/Components/OpenDataCatalog/): - Dynamic discovery + merge of every event across the Vantor S3 STAC and Planet Source-Cooperative STAC catalogs; framework-free module with a discoverEvents()/fetchEventCatalog() contract (swap-to-backend seam). - Azure Maps footprints + TiTiler COG preview (works with no Azure Maps key); preview tile loading indicator. - Add a scene to pre/post in one click with source-type + capture-date auto-fill; buttons are phase-scoped (pre->Pre, post->Post). - Two-way map<->list selection: click a footprint to scroll/highlight/expand the list row; footprints stay clickable during preview. Server-side clip AOI: - Draw a box that sets a layer-level clipBbox ([w,s,e,n] EPSG:4326); imagery prep clips the pre/post mosaics via gdalwarp -te/-te_srs (GDAL reprojects the bounds). The catalog filters scenes overlapping the AOI and badges scenes that fully cover it, so pre/post share one AOI. - Model ImageLayer.clipBbox + validate_clip_bbox; PutLayer validates/carries it; threaded through the imagery-prep config -> ImageryWorkflow -> mosaic_imagery. Verified end-to-end (output bounds match the drawn box). Supporting fixes: - Allow data.source.coop on the imagery allowlist (UI + hastelib) and route non-S3/Blob allowlisted hosts through the generic HTTP downloader, so Planet imagery downloads at prep time. - Local dev proxy (docker/nginx.conf): TiTiler timeouts + client_max_body_size for chunk uploads. - QUICKSTART: DOCKER_GID must be 0 on macOS Docker Desktop. Azure Maps auth is unchanged (existing anonymous/AAD Client-ID flow). Spec: spec/features/open-data-catalog/. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds an in-app Open Data Catalog explorer on the Create Image Layer flow, letting analysts discover/preview open disaster imagery (Vantor/Maxar + Planet Source Cooperative) and build image layers faster, including an optional server-side clip AOI applied during imagery prep.
Changes:
- Adds new Open Data Catalog UI (panel + list + Azure Maps footprints + TiTiler preview) and wires it into the Create Image Layer form (including scene-to-pre/post “add” + auto-fill).
- Introduces
ImageLayer.clipBboxend-to-end (UI → PutLayer validation → imagery-prep config → GDAL mosaic clip). - Expands imagery URL allowlist to include
data.source.coopand routes non-S3/Blob allowlisted URLs through the generic HTTP downloader; updates local dev proxy settings/docs.
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| ui/src/util/validation.js | Allowlists data.source.coop imagery hosts in UI URL validation. |
| ui/src/Components/OpenDataCatalog/SceneListItem.jsx | New scene row UI with metadata, AOI badge, and phase-scoped add buttons. |
| ui/src/Components/OpenDataCatalog/OpenDataCatalogPanel.jsx | New catalog side panel: discovery, filtering, list↔map selection sync, clip toolbar. |
| ui/src/Components/OpenDataCatalog/OpenDataCatalogMap.jsx | New Azure Maps footprint rendering + TiTiler preview + rectangle clip drawing. |
| ui/src/Components/OpenDataCatalog/openDataCatalog.js | Framework-free STAC discovery/normalization + TiTiler URL helpers + bbox overlap helpers. |
| ui/src/Components/CreateEditImageLayerHelper.js | Adds clipBbox to form state + helper to add catalog scenes into imagery URL arrays. |
| ui/src/Components/CreateEditImageLayerForm.jsx | Adds “Browse Open Data Catalog” entry point; threads clipBbox into PutLayer payload. |
| spec/features/open-data-catalog/README.md | Feature spec overview and decision log for catalog + server-side clip. |
| spec/features/open-data-catalog/design.md | Detailed design: discovery/preview/clip flow and API contract changes. |
| spec/features/open-data-catalog/data-model.md | Documents clipBbox schema change and allowlist additions. |
| spec/features/open-data-catalog/user-stories.md | User stories and acceptance criteria for catalog and clip workflows. |
| spec/features/open-data-catalog/test-plan.md | Test strategy/coverage matrix for allowlist + clip + manual flows. |
| spec/features/open-data-catalog/plan.md | Implementation plan/status across core/API/UI phases. |
| spec/features/open-data-catalog/impact-analysis.md | Impact/risk analysis across UI/API/workflow/security considerations. |
| spec/features/open-data-catalog/rollout.md | Rollout plan, prerequisites, monitoring, rollback notes. |
| QUICKSTART.md | Documents macOS Docker Desktop DOCKER_GID behavior (set to 0). |
| hastelib/tests/core/utils/test_url_allowlist.py | Adds unit tests for data.source.coop allowlist + validate_clip_bbox. |
| hastelib/src/hastegeo/workflows/prepare_imagery.py | Threads clip_bbox into workflow and mosaic creation. |
| hastelib/src/hastegeo/core/utils/url_allowlist.py | Adds source.coop allowlist and validate_clip_bbox validator. |
| hastelib/src/hastegeo/core/utils/imagery.py | Clips mosaics during GDAL warp when clip_bbox is set. |
| hastelib/src/hastegeo/core/utils/downloader.py | Routes non-S3/Blob allowlisted imagery hosts to generic HTTP downloader. |
| hastelib/src/hastegeo/core/processors/imagery.py | Adds clip_bbox into imagery-prep config payload. |
| hastelib/src/hastegeo/core/models/projects.py | Adds ImageLayer.clipBbox field to model. |
| docker/nginx.conf | Increases body size and TiTiler proxy timeouts for local dev. |
| docker/docker-compose.yml | Adds CLEANUP_CONTAINERS env var to aid debugging spawned containers. |
| api/hastefuncapi/function_app.py | Validates clipBbox in PutLayer (400 on invalid bbox). |
Comment on lines
+13
to
+16
| function formatGsd(gsd) { | ||
| if (gsd == null || gsd === "") return null; | ||
| return `${Number(gsd).toFixed(2)} m GSD`; | ||
| } |
Comment on lines
+18
to
+20
| function formatSize(bytes) { | ||
| if (!bytes || !Number.isFinite(Number(bytes))) return null; | ||
| let n = Number(bytes); |
Comment on lines
119
to
122
| // Keep in sync with hastelib/src/hastegeo/core/utils/url_allowlist.py. | ||
| const IMAGERY_URL_ALLOWED_HOST_DESCRIPTION = | ||
| "Azure Blob Storage (*.blob.core.windows.net) or AWS S3 (*.amazonaws.com)"; | ||
| "Azure Blob Storage (*.blob.core.windows.net), AWS S3 (*.amazonaws.com), or Source Cooperative (data.source.coop)"; | ||
|
|
Comment on lines
18
to
22
| ALLOWED_HOST_DESCRIPTION = ( | ||
| "Azure Blob Storage (*.blob.core.windows.net) " | ||
| "or AWS S3 (*.amazonaws.com)" | ||
| "Azure Blob Storage (*.blob.core.windows.net), " | ||
| "AWS S3 (*.amazonaws.com), " | ||
| "or Source Cooperative (data.source.coop)" | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add an Open Data Catalog side panel to the Create Image Layer page so analysts can browse open disaster imagery (Vantor/Maxar + Planet Open Data), preview it, and build an image layer without hunting for COG URLs.
Catalog UI (ui/src/Components/OpenDataCatalog/):
Server-side clip AOI:
Supporting fixes:
Azure Maps auth is unchanged (existing anonymous/AAD Client-ID flow).
Spec: spec/features/open-data-catalog/.