Refactor input sources and main workflow

.dockerignore

@@ -1,5 +1,63 @@
-**/__pycache__/*
-.mypy_cache
+# Pixi environment (will be recreated in container)
+.pixi/
+
+# Python bytecode
+**/__pycache__/
+*.py[cod]
+*$py.class
+*.so
+
+# Distribution / packaging
 build/
-data/
 dist/
+*.egg-info/
+*.egg
+
+# Test and coverage
+.pytest_cache/
+.coverage
+.coverage.*
+htmlcov/
+.tox/
+.nox/
+
+# Type checking caches
+.mypy_cache/
+.ruff_cache/
+
+# Documentation build
+docs/_build/
+site/
+
+# IDE and editor files
+.vscode/
+.idea/
+*.swp
+*~
+
+# OS files
+.DS_Store
+Thumbs.db
+
+# Git
+.git/
+.gitignore
+
+# Test / scratch data (large files)
+tests/data/*.tif
+data/
+cropped/
+
+# Scratch and scripts (development only)
+scratch/
+scripts/
+
+# Old micromamba-era docker scripts (retained in repo for history but
+# not needed inside the pixi-based build)
+docker/
+
+# Docker files (not needed inside container)
+Dockerfile
+Dockerfile.gpu
+docker-compose.yml
+.dockerignore

.github/workflows/test-build-push.yml

@@ -15,41 +15,22 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-latest, macos-14]
-
       fail-fast: true
     runs-on: ${{ matrix.os }}
-    defaults:
-      run:
-        shell: bash -l {0}
     steps:
       - name: Checkout
         uses: actions/checkout@v4
-      - name: Setup environment
-        uses: mamba-org/setup-micromamba@v2
+      - name: Setup pixi
+        uses: prefix-dev/setup-pixi@v0.8.1
         with:
-          environment-file: conda-env.yml
-          environment-name: sweets-env
-          # persist on the same day.
-          cache-environment-key: environment-${{ steps.date.outputs.date }}
-          cache-downloads-key: downloads-${{ steps.date.outputs.date }}
-          generate-run-shell: false
-          # create-args: ${{ matrix.deps.spec }}
-          condarc: |
-            channels:
-              - conda-forge
-      - name: Install
-        run: |
-          python -m pip install --no-deps .
-      - name: Install test dependencies
-        run: |
-          micromamba install -f tests/requirements.txt -c conda-forge
+          environments: test
+          cache: true
       - name: Enable numba boundscheck for better error catching
         run: |
           echo "NUMBA_BOUNDSCHECK=1" >> $GITHUB_ENV
           echo "TQDM_DISABLE=1" >> $GITHUB_ENV
-      - name: Test (with numba boundscheck on)
-        run: |
-          pytest -n0 -vv
+      - name: Test
+        run: pixi run -e test pytest -n0 -vv
 
 
   dockerize: # Based on Mintpy: https://github.com/insarlab/MintPy/blob/5ca554fef324b816f9130feec567e2cf463e41d2/.github/workflows/build-n-publish-to-pypi.yml
@@ -83,7 +64,7 @@ jobs:
         uses: docker/build-push-action@v4
         with:
           context: .
-          file: ./docker/Dockerfile
+          file: Dockerfile
           push: ${{ github.event_name != 'pull_request' }}
           tags: |
             ghcr.io/${{ github.repository }}:${{ env.SWEETS_VERSION }}

.gitignore

@@ -167,3 +167,5 @@ cython_debug/
 # pixi environments
 .pixi
 *.egg-info
+.DS_Store
+*.pdf

.readthedocs.yaml

@@ -7,7 +7,7 @@ mkdocs:
   configuration: mkdocs.yml
 
 conda:
-  environment: conda-env.yml
+  environment: environment.yml
 
 python:
   install:

CHANGELOG.md

@@ -1,7 +1,148 @@
-# Unreleased
+# [0.3.0](https://github.com/opera-adt/sweets/compare/v0.2.0...v0.3.0) - 2026-04-12
 
+**Major changes**
+- **Burst-level downloads.** Sentinel-1 data is now fetched as just the bursts
+  that intersect the AOI via `burst2safe`, instead of full ~250x170 km frames.
+  Closes #23, #85, #88.
+- **OPERA CSLC source.** A second SLC source `OperaCslcSearch` skips
+  burst2safe + COMPASS entirely and pulls pre-made OPERA CSLC HDF5s
+  (and the matching CSLC-STATIC layers) directly from ASF DAAC. Pick
+  it via `sweets config --source opera-cslc`. Locked to OPERA's 5 m x 10 m
+  posting; great for CONUS where OPERA has produced for the AOI.
+- **NISAR GSLC source.** A third source class `NisarGslcSearch` wraps
+  `opera_utils.nisar.run_download` to fetch pre-geocoded NISAR GSLC
+  HDF5s via CMR (L-band, UTM, already geocoded). Pick it via
+  `sweets config --source nisar-gslc --frequency A --polarizations HH`.
+  Skips COMPASS, burst-db, orbits, and geometry stitching; dolphin
+  reads the grid directly from the HDF5. Tropo correction is not yet
+  supported on this path (NISAR GSLCs carry no stitched incidence angle).
+- `Workflow.search` is a `Union[BurstSearch, OperaCslcSearch, NisarGslcSearch]`
+  discriminated by a `kind` field; existing configs without a `kind`
+  default to `safe` for backwards compat.
+- **Tropospheric correction (opt-in).** New `--do-tropo` flag wires the
+  OPERA L4 TROPO-ZENITH workflow from
+  `opera_utils.tropo.create_tropo_corrections_for_stack` into a post-step
+  that runs after dolphin produces unwrapped phase. Outputs land at
+  `dolphin/tropo/tropo_correction_<dt>.tif` and `dolphin/tropo_corrected/<pair>.tropo_corrected.unw.tif`.
+- **dolphin end-to-end.** Phase linking, interferogram network selection,
+  stitching, unwrapping, timeseries inversion and velocity estimation are now
+  delegated to a single `dolphin.workflows.displacement.run` call. The
+  hand-rolled interferogram / stitch / unwrap orchestration was deleted.
+- **`tyro` CLI.** `sweets config`, `sweets run` and `sweets server` are now
+  defined with `tyro` instead of argparse, cutting ~200 lines and giving
+  proper rich help. `sweets run <config_file>` is now positional.
+- **pixi as the primary install.** `pyproject.toml` is reorganized so the
+  `[tool.pixi.*]` sections are the canonical environment definition; an
+  `environment.yml` synced from pixi is provided for non-pixi users.
+- **`s1-reader`, `COMPASS`, `opera-utils` and `dolphin` fork pins.**
+  sweets now installs all four from `scottstanie/<repo>@develop-scott`.
+  The develop-scott branches carry numpy 2 fixes (polyfit scalar in
+  s1-reader, `np.string_`/`np.unicode_` in COMPASS), the new tropo
+  workflow / `search_tropo` CMR client in opera-utils, the
+  GDT_Float16 GTIFF_KWARGS fix in opera-utils' `apply_tropo`, and a
+  `_yaml_model._add_comments` fix in dolphin so Union-of-submodels
+  schemas serialize cleanly. Closes #132.
 
-# [0.2.0](https://github.com/opera-adt/dolphin/compare/v0.2.0...v0.3.0) - 2023-08-23
+**Removed**
+- `sweets.interferogram` (replaced by dolphin's interferogram network).
+- `sweets._missing_data`, `sweets._unzip` (no longer needed — the
+  missing-data filter now lives in `Workflow._apply_missing_data_filter`
+  on top of `opera_utils.missing_data`, and `burst2safe` materializes
+  `.SAFE` directories directly so there is nothing to unzip).
+- `scripts/prep_mintpy.py` (broken with the new layout; mintpy export is
+  TODO via dolphin's existing exporters).
+
+**Added**
+- **Missing-data filter.** New `Workflow._apply_missing_data_filter`
+  wraps `opera_utils.missing_data.get_missing_data_options` to
+  enumerate every `(burst_ids, dates)` subset where every chosen
+  burst has every chosen date, and picks the one that maximizes
+  total CSLC count. Files that aren't in the top option get moved
+  (not deleted) to `<work_dir>/excluded_cslcs/<burst_id>/<date>/` so
+  a debugging user can pull them back. Runs post-COMPASS on
+  BurstSearch (OPERA-style `t071_151230_iw2` naming is what
+  `group_by_burst` expects), and post-download on OperaCslcSearch.
+  No-op on NisarGslcSearch (not burst-organized; `_rank_signatures`
+  handles coverage there). Replaces an earlier simpler
+  `_drop_short_burst_stacks` heuristic, and keeps dolphin from
+  forming a network across partial-coverage bursts — the prior
+  failure mode was an opaque dolphin crash deep inside
+  `interferogram.Network._make_ifg_pairs` when a bbox nicked the
+  edge of a second burst.
+- **`oil_slick` phase colormap.** `sweets.plotting.plot_ifg` now defaults
+  to a cyclic `"oil_slick"` colormap inspired by thin-film interference
+  colors (dark -> violet -> blue -> cyan -> green -> yellow -> orange ->
+  red -> dark), replacing the previous `colorcet.CET_C8` default. The
+  colormap is registered with matplotlib on import so it is also
+  available by name as `cmap="oil_slick"` in any `imshow` / `pcolormesh`
+  call. `sweets.plotting` itself is preserved (the earlier CHANGELOG
+  draft claimed it was removed — that was incorrect; `plot_ifg`,
+  `browse_ifgs`, `browse_arrays` and `plot_area_of_interest` all remain).
+- **Example notebooks, one per source plus a cross-source comparison**,
+  under `docs/`. All four share the same LA AOI + Dec 2025 window
+  so runs can be compared directly:
+  - `example_s1_burst.ipynb` — burst-subset S1 + COMPASS + dolphin
+  - `example_opera_cslc.ipynb` — pre-made OPERA CSLCs from ASF
+  - `example_nisar.ipynb` — NISAR GSLC via CMR, VRT-wrapped for dolphin
+  - `example_compare_sources.ipynb` — loads the longest-baseline
+    timeseries raster from each run and plots them side-by-side with
+    a uniform color scale.
+
+**Fixed**
+- **NISAR VRT wrappers instead of GeoTIFF rewrite.** GDAL's HDF5 driver
+  can't parse NISAR's separate `xCoordinates` / `yCoordinates` grid
+  arrays, so sweets used to rewrite each polarization as a ~19 MB
+  CFloat32 GeoTIFF alongside every 40 MB subset HDF5. That's now a
+  ~1 KB VRT that injects the real SRS + GeoTransform on top of the
+  raw HDF5 subdataset — dolphin opens the VRT natively and the HDF5
+  stays the single source of truth for pixel values. Conversion step
+  dropped from O(n_pixels) to O(1).
+- **NISAR wavelength: read centerFrequency from HDF5 at runtime.**
+  Three-tier resolution in `NisarGslcSearch.wavelength()`:
+  (1) read `/science/LSAR/GSLC/grids/frequency{A,B}/centerFrequency`
+  from the HDF5 — authoritative, distinguishes freqA from freqB
+  automatically, and picks up the exact carrier reported by the
+  processor; (2) if missing, parse the NISAR D-102269 §3.4 filename
+  MODE code and look it up in `dolphin.constants.NISAR_L_MODE_CENTERS_HZ`
+  (Figure 3-1 values — approximate to ~0.8% but strictly better than
+  the generic constant for split modes); (3) fall back to the generic
+  `NISAR_L_WAVELENGTH` / `NISAR_S_WAVELENGTH` from the granule prefix
+  (`NISAR_L*` / `NISAR_S*`), matched to the full-band 77 MHz center.
+  Parallel filename-based auto-detect on the dolphin side for anyone
+  passing NISAR HDF5s directly. Fixes
+  isce-framework/dolphin#704; without it, NISAR timeseries / velocity
+  outputs landed in radians instead of meters.
+- **NISAR signature ranking + fallback.** `NisarGslcSearch.download()`
+  now ranks (frequency, polarization) groups by `(stack size, pol match,
+  freq match)` so a `polarizations` pin always beats a `frequency` pin
+  on ties, and iterates groups in order — if the best group's products
+  all yield empty stubs (AOI inside the bounding polygon but outside
+  the actual grid extent, common on NISAR PR products), sweets falls
+  through to the next signature instead of silently writing zero
+  GeoTIFFs. Raises a clear diagnostic if every signature is empty.
+- **Source-aware DEM bbox.** `Workflow._dem_bbox` used to pad the study
+  bbox by 0.25 deg for every source, but COMPASS geocoding on the
+  BurstSearch path needs DEM coverage for the full IW burst footprint
+  (~20 x 85 km), not just the study area. BurstSearch now pads by 1 deg;
+  NISAR / OPERA-CSLC keep the 0.25 deg buffer. Users can override with
+  a new optional `dem_bbox` field. Water-mask downloads stay on the
+  study-area bbox so the BurstSearch path doesn't waste ASF tile
+  fetches on terrain outside the crop area.
+- **Min-GSLC guard.** `Workflow.run` now raises a clear error before
+  invoking dolphin when fewer than 2 GSLCs survive step 2, instead of
+  letting dolphin fail deep inside `interferogram._make_ifg_pairs` with
+  "No valid ifg list generation method specified".
+- Driver-prefix heuristic for HDF5 vs NETCDF pushed down into
+  `opera_utils.format_nc_filename` and `opera_utils.create_nodata_mask`
+  (on `scottstanie/opera-utils@develop-scott`) and
+  `dolphin.io.format_nc_filename` (on `scottstanie/dolphin@develop-scott`)
+  so the NISAR raw-HDF5 subdataset path works end-to-end.
+- NISAR wavelength auto-detect + corrected `NISAR_L_FREQUENCY` constant
+  landed on `scottstanie/dolphin@develop-scott`
+  (isce-framework/dolphin#704) NISAR wavelength fix
+
+
+# [0.2.0](https://github.com/isce-framework/sweets/compare/v0.2.0...v0.3.0) - 2023-08-23
 
 **Fixed**
 - Geometry/`static layers` file creation from new COMPASS changes

CITATION.cff

@@ -0,0 +1,27 @@
+cff-version: 1.2.0
+message: "If you use this software, please cite it using these metadata."
+title: "sweets: Workflows for generating surface displacement maps using InSAR"
+type: software
+authors:
+  - family-names: Staniewicz
+    given-names: Scott J.
+    orcid: https://orcid.org/0000-0002-3055-5731
+    affiliation: Jet Propulsion Laboratory, California Institute of Technology
+  - family-names: Havazli
+    given-names: Emre
+    orcid: https://orcid.org/0000-0002-1236-7067
+    affiliation: Jet Propulsion Laboratory, California Institute of Technology
+  - family-names: Langemeijer
+    given-names: Jaap
+  - family-names: Dhar
+    given-names: Tisham
+repository-code: "https://github.com/isce-framework/sweets"
+abstract: "sweets is an end-to-end workflow for generating surface displacement time series from Sentinel-1 and NISAR data, wrapping burst download, COMPASS/isce3 geocoded SLC generation, and the dolphin PS/DS InSAR processor."
+keywords:
+  - InSAR
+  - Sentinel-1
+  - NISAR
+  - surface displacement
+  - time series
+  - remote sensing
+license: "BSD-3-Clause OR Apache-2.0"

CLAUDE.md

@@ -0,0 +1,62 @@
+# Python Coding & Editing Guidelines
+
+> **Living document – PRs welcome!**
+> Last updated: 2025‑07‑15
+
+## Table of Contents
+
+1. Philosophy
+1. Docstrings & Comments
+1. Type Hints
+1. Documentation
+
+---
+
+## Philosophy
+
+- **Readability, reproducibility, performance – in that order.**
+- Prefer explicit over implicit; avoid hidden state and global flags.
+- Measure before you optimize (`time.perf_counter`, `line_profiler`).
+- Each module holds a **single responsibility**; keep public APIs minimal.
+
+## Docstrings & Comments
+
+- Style: NumPyDoc.
+- Start with a one‑sentence summary in the imperative mood.
+- Sections: Parameters, Returns, Raises, Examples, References.
+- Use backticks for code or referring to variables (e.g. `xarray.DataArray`).
+- Do not use emojis, or non-unicode characters in comments/print statements.
+- Cite peer‑reviewed papers with DOI links when relevant.
+- Write code that explains itself rather than needs comments.
+- For the inline you do add, explain *why*, not what. For example, *don't* write:
+
+```python
+# open the file
+f = open(filename)
+```
+
+- The comments should be things which are not obvious to a reader with typical background knowledge.
+
+## Tools
+
+- ruff is use for most code maintenance, black for formatting, mypy for type checking, pytest for testing
+- You can run `pre-commit run -a` to run all pre-commit hooks and check for style violations
+
+## Code Style
+
+- Annotate all public functions (PEP 484).
+- Prefer `Protocol` over `ABC`s when only an interface is needed.
+- Validate external inputs via Pydantic models (if existing); otherwise, use `dataclasses`
+- Parse, don't validate, with your dataclasses. Checks should be at the serialization boundaries, not scattered everywhere in the code.
+- If you need to add an ignore, ignore a specific check like # type: ignore[specific]
+- Don't write error handing code or smooth over exceptions/errors unless they are expected as part of control flow.
+- In general, write code that will raise an exception early if something isn't expected.
+- Enforce important expectations with asserts, but raise errors for user-facing problems.
+
+## Documentation
+
+- mkdocs + Jupyter. Hosted on ReadTheDocs.
+- Auto API from type hints.
+- Provide tutorial notebooks covering common workflows.
+- Include examples in docstrings.
+- Add high-level guides for key functionality.