bathymetry: IVERT v0.6.0-beta - #5
Merged
Merged
Conversation
… in prep for full-desktop implementation.
… misc updates. - Move 13 unused/old source files to src/archive/ - Convert print() to logging in icesat2_database_v2.py - Add plot_photon_clouds_v2.py - Update config, icesat2_requests, ivert_output_vector, transform_points, configfile, validate_dem
…ract, and staleness bugs. - Update default photon classes from [1,5,40] to [1,6,40] (ground, land_ice, bathy_floor) throughout validate_dem.py and icesat2_database_v2.py; class 5 does not exist in globato, class 6 is land_ice (ATL06). - Pass photon_classes=classes through from validate_dem_parallel to query_photons so the caller's class selection is actually respected. - Guard against dates=None before building dem_3d_bbox; falls back to full mission span (20180101–20991231). - Guard against omit_bboxes=None at both the call site (validate_dem_parallel) and inside query_photons, preventing a len(None) crash. - Fix height_field staleness: move assignment to after the DEM-bounds photon filter so its shape stays consistent with the filtered photon_df. - Fix query_photons docstring: bbox format was listed as xmin,ymin,xmax,ymax; correct order is xmin,xmax,ymin,ymax. Replace stale cudem CLI hint with globato reference.
…wire validate_dem and validate_dem_collection. - New utils/dem_geom.py: get_dem_reference_frame_from_file, get_dem_reference_frame_from_user_input, get_dem_srs_string, get_wgs84_bounding_box — extracted from icesat2_query.py with no cudem dependency; icesat2_query.py moved to src/archive/. - validate_dem.py: import icesat2_query → import utils.dem_geom; update three call sites accordingly. - validate_dem_collection.py: remove all ivert server-side code (ivert_job_name, ivert_jobs_db, ivert_exporter, clean_ivert_files, is_aws, jobs_database, server_file_export); pass classes= through to validate_dem; fix default classes [1,5,40]→[1,6,40]; drop defunct output_vdatum parameter; fix early-exit plot call to plot_histograms_and_line.
Add src/cli.py with four commands (setup, download, validate, upgrade) using click, matching the existing user-facing interface. Wire upgrade to client_upgrade.upgrade(); leave the other three as stubs to be wired to their respective modules. Add click to install_requires and update the console_scripts entry point to ivert.cli:ivert_cli. Archive client.py.
…s_database.py. VERSION_CLIENT_MIN and the jobs database were holdovers from the client-server architecture. The file no longer exists in the repo and the jobs database module is not used in the offline-only codebase.
…validate helper. - cli.py: add _run_validate() — expands globs, builds class list, branches to validate_dem (single file) or validate_list_of_dems (directory/multi-file) - validate_dem.py: default dem_vertical_datum to None so callers can omit it and let validation read the datum from the DEM metadata header - validate_dem_collection.py: default input_vdatum to None (same); fix band_num not being forwarded to the per-DEM validate_dem calls - archive analyze_validation_results.py
… file. When include_photon_level_validation=True and results_dataframe_file already exists but the photon results file is absent, return None to trigger a full re-run rather than returning early without the photon file. Also include the photon results file in files_to_export and shared_ret_values when it exists.
Accepts a 4-value slash-separated bbox (W/E/S/N default, --wsen for W/S/E/N) or one or more DEM files; reprojects to WGS84 via dem_geom.get_wgs84_bounding_box. Parses dates via IS2Database.convert_date_to_yyyymmdd (dateparser-compatible strings) and classes via slash-separated integers before calling download_new_granules.
Drop driver='GPKG' from geopandas.read_file — pyogrio does not accept driver as an open option (extension-based detection is sufficient). Add logging.basicConfig in the download command so Harmony job progress messages reach the terminal.
- cli.py: change --name default from "DEMs" to None so single-DEM plots fall back to the filename when no name is given; update help text. Add NUMEXPR_MAX_THREADS suppression before NumExpr is imported. - icesat2_database_v2.py: add _normalize_bbox_columns to fix GPKG round-trip of list columns stored as JSON strings; apply on open_gdf and open_gdf_subset. Un-comment filter_query_bbox in download_new_granules. Add delete and rebuild __main__ subcommands. - cuboid_funcs.py: remove debug print statement.
…idate. Both ivert download and ivert validate now accept: -cl/--confidence-level (int 1-4): minimum ATL03 signal confidence to keep. -bc/--bathy-confidence (float 0-1): minimum ATL24 bathy confidence for class-40 photons. At download time the filters are applied in _process_h5_to_nc before saving to NetCDF. At validate time they are passed through _run_validate → validate_dem/validate_list_of_dems → validate_dem_parallel → _fetch_photons → IS2Database.query_photons. Also lowercased all short CLI flags on both commands (-DS→-ds, -DE→-de, -P→-p, -R→-r, -C→-c, -B→-b); -V/--vdatum stays uppercase. Added -v/--verbose to ivert validate.
- Add missing install_requires: blosc2, geopandas, matplotlib, netCDF4, numexpr, pyproj, rasterio, shapely, six, xarray; restore boto3 - Add fetchez, globato, transformez as git dependencies - Remove archived scripts (src/s3.py, src/jobs_database.py) from scripts - Remove archived config files from data_files - Remove generate_granule_lines.py (stub-only, referenced archived module)
client_user_setup.py was deprecated cloud-architecture code; no other active module imported it or boto3.
config/ivert_config.ini is renamed to config/ivert_defaults.ini to make clear it holds shipped defaults, not user state. configfile.py now also reads ~/.ivert/user_config.ini (path from user_configfile in defaults) after loading defaults, overlaying any keys the user has set there. setup.py updated to reference the new filename.
- ivert setup list: shows all user-editable config settings with [default] or [user] source indicators - ivert setup key=value: writes overrides to the user config file, creating it on first use - ivert --config PATH: per-invocation config file override - IVERT_USER_CONFIG env var: session-wide config file override - Config._apply_user_config resolves user config path in priority order: env var > defaults file setting - Fix URL values being wrongly resolved as local paths in configfile.py - Add docs/configfile.md documenting all of the above
… subcommands Replaces top-level `ivert download` with `ivert database` group. Subcommands: download (same logic, updated example in docstring), list (summary table; --all for all fields; --boxes for unique query bboxes), rebuild (recreates .gpkg index from .nc files), delete (removes index files; --all also removes .nc granule files). Classes help text updated to include -1=unclassified and 0=noise.
- Replace setup.py metadata with pyproject.toml (PEP 517/621); setup.py reduced to a minimal shim - Move config/ivert_defaults.ini into src/config/ (ivert package) and load it via importlib.resources, eliminating the deprecated data-files install path - Consolidate deprecated archive files from src/archive/ and config/archive/ into top-level archive/src/ and archive/config/ with a README - Bump version to 0.6.0
- Add --debug flag to ivert CLI to enable DEBUG-level logging - Show X/Y progress counter in "Saved" log messages during granule processing - Log "No valid classified photons retrieved" (with counter) for granules that produce no output, instead of silently skipping them - Sort h5_files alphabetically before processing so granules are handled in chronological order
New module src/vdatum_lookup.py translates common vertical datum names (e.g. 'navd88', 'egm2008', 'mllw') to formal EPSG code strings. It builds its table from transformez.definitions.Datums when available and merges in a curated fallback so well-known names always resolve. The public API is resolve_vdatum(), get_epsg_description(), and list_vdatums(). cli.py: _run_validate now calls resolve_vdatum() and raises a clear error for unrecognised names. Added --list-vdatums flag to ivert validate that prints every recognised name with its EPSG code and description. Updated -V/--vdatum help text to mention short names. Added -o/--outdir option and wired ivert_results_subdir config default. validate_dem.py: fixed latent bug in _fetch_photons where a raw datum string was assigned to dem_vert_ref_frame but get_dem_srs_string expects a pyproj.CRS; now passes through get_dem_reference_frame_from_user_input. validate_dem_collection.py: guard isinstance(dem_list_or_dir, str) before os.path.isdir to avoid TypeError when a list is passed. src/__init__.py: add src/ to sys.path so direct-run imports work alongside the installed CLI. src/utils/__init__.py: alias ivert_utils↔utils in sys.modules to prevent duplicate module loads. config/ivert_defaults.ini: add ivert_results_subdir default.
validate_dem_parallel called _compute_photon_overlap, which called transform_points.transform_points, without ever passing cache_dir. transform_points defaulted to os.getcwd()/transformez_cache, producing the stray transformez_cache/ directory in whatever directory ivert was run from. Fix: read ivert_config.cache_directory once at module load as TRANSFORMEZ_CACHE_DIR and thread it through _compute_photon_overlap to transform_points.transform_points.
Passing len([]) = 0 as xmax to matplotlib PercentFormatter triggers a RuntimeWarning divide-by-zero when the axis tick formatter runs. Guard all five call sites with max(..., 1) so empty plots render without error.
…codes - Summary stats file labelled diff_mean as "(ICESat-2 - DEM)" but the computation has always been DEM - ICESat-2; corrected both occurrences. - argparse __main__ path split classes into strings then discarded the int conversion result, causing isin() to match nothing against int64 class_code column; now builds the integer list directly.
…LI options - Replace haversine along-track with UTM-projected distances (geopandas) for accurate metre-scale geometry; axis label unchanged (km) - Load all ATL03 .h5 photons per beam as class-0 noise background; geoid-correct heights via geophys_corr/geoid interpolation; assign along_track_km to .nc photons by merging on delta_time so both share the same axis - Improve .h5 auto-search: extract bare granule ID (strip _subsetted/bbox suffixes) and glob-match, replacing exact-name lookup - --h5 now accepts an optional path argument (nargs='?'); omitting the path triggers cache search by granule ID - Replace --ylim with --zmin/--zmax (independent elevation bounds) - Add --dmin/--dmax to subset the plot by along-track distance (km) - Add --classes (slash-separated codes, e.g. '1/40/41') to filter which classified photons are shown; '' plots noise background only - Filter non-physical elevations (outside ±1e5 m) before plotting - Add class-42 (Inland Water) style entry
… of dropping When --classes is given, photons outside the selected set are now downgraded to class 0 (noise/grey) rather than removed, so the full photon cloud remains visible in the background.
…l arg - New --h5-only flag skips .nc classifications and plots all ATL03 photons as noise (class 0), useful for inspecting the raw photon cloud - Passing an .h5 file as the positional argument automatically enables h5-only mode; the .h5 is used directly without needing a matching .nc - Positional argument renamed from nc_file to input_file to reflect both cases - Output stem derived from the .h5 filename in h5-only mode
Delta_time alone cannot identify a beam — all six ATL03 beams fire simultaneously and share the same delta_time values. Replace the delta_time-only filter with an inner merge on (delta_time, x, y), which uniquely matches nc photons to the correct h5 beam. This also folds the along_track_km assignment into the same merge, removing the separate lookup step. Per-beam plotting now requires the h5 file; without it the nc fallback plots all photons together as before.
…tput - laser: already present in globato chunks, now included in keep_cols so it survives into the nc file - along_track_m: new static method _h5_along_track_m reads segment_length and dist_ph_along from the h5 geolocation group, computes cumulative along-track distance (m) per photon, and merges into the dataframe on (laser, delta_time, x, y) before writing the nc
…oad h5 without --h5 load_nc crashed with NotImplementedError on the laser column (object dtype) because pandas can't process object arrays via memoryview. Now converts object-dtype arrays to str explicitly. Also fixes silent auto-discovery of the .h5 file when --h5 is not given: background photons should only be included when the flag is explicitly passed.
…ors; DEM resampling at native resolution - --beam → --laser (-b shorthand unchanged) - --dmin/--dmax → --xmin/--xmax - Canopy (class 2): mediumseagreen/alpha=0.5/s=1 → limegreen/alpha=0.8/s=2 - Canopy Top (class 3): darkgreen/alpha=0.5/s=1 → forestgreen/alpha=0.9/s=2; zorder 1→2 - _sample_dem_along_track: interpolate track to evenly-spaced grid at DEM pixel size instead of sampling only at photon locations, eliminating flat gaps in DEM profile
…mation - setup list: bold the Setting header; cyan for setting names; yellow for [user] overrides; dim grey for [default] values - setup reset: new subcommand deletes the user config file (with confirmation prompt; -y/--yes to skip) - database delete: collect files and sizes before deleting, print a per-file breakdown and total, prompt for confirmation (default no); -y/--yes skips the prompt - database size / database delete: replace local _fmt_bytes with sizeof_fmt from utils.sizeof_format - ivert_cli: replace --debug flag with --verbosity LEVEL (debug/info/ warning/error); reads default from config if not specified - validate: drop standalone --verbose flag; verbosity now flows from --verbosity on the top-level ivert_cli group
Adds --ndv VALUE to `ivert validate`. Resolution priority: 1. User-specified --ndv (accepts a number or 'nan') 2. No-data value from the DEM file header 3. Config default (dem_default_ndv, currently -99999.0) Previously the code fell back to "treat all pixels as valid" when the file header had no NDV set; now it falls back to the config default.
Stores 'horizontal_datum' (EPSG:4326) and 'vertical_datum' (EPSG:3855 or EPSG:4979) in each .nc file's metadata and in the GeoPackage index, derived from the configured 'icesat2_vertical_datum' setting. Also adds: - _validate_vertical_datum / _vertical_datum_to_vertical_epsg helpers - get_photon_src_epsg(): builds compound EPSG string from the first database record (e.g. 'EPSG:4326+3855'), falling back to EPSG:4326+4979 - download_new_granules: rejects downloads that would mix datums with records already in the database - Replaces hardcoded water_surface="geoid" in _process_h5_to_nc with the config-driven vertical_datum value
…_default_ndv to NaN
- dem_default_ndv: changed from -99999.0 to NaN so pixels with
IEEE NaN are masked by default
- icesat2_vertical_datum: new setting ('ellipsoid' or 'geoid') controls
the vertical datum photons are stored in (default: 'ellipsoid' / EPSG:4979)
- verbosity: new setting controlling logging output level
(debug/info/warning/error, default: info)
When a DEM has no bathymetry photons (panel B) or no topography photons (panel A), replace the empty histogram with a blank box stating "No bathymetry data was validated." or "No topography data was validated." instead of rendering a histogram with "nan ± nan m".
…ft lookup _apply_vertical_transform now strips "EPSG:" prefixes and compound specifiers (e.g. "4326+4979") from src/dst vertical EPSG strings before building the cache filename and calling transformez.generate_grid(). _decompose_crs also handles 3D geographic CRS (e.g. EPSG:4979) that pyproj does not mark as compound or vertical, returning the CRS itself as both the horizontal and vertical datum identifier.
…ne mask CRS bug Export error formats (validate_dem.py, cli.py, ivert_defaults.ini, validate_dem_collection.py): - New config key export_error_formats (default: tif,gpkg) writes <dem>_errors.<ext> files alongside results; supports tif, gpkg, shp, xyz. - -ef/--export-formats CLI flag overrides the config for a single run. - export_error_results(), _export_errors_vector(), _export_errors_xyz(), _normalize_export_formats(), _error_export_filenames(), _results_cell_centers() added to validate_dem.py. ERROR_EXPORT_FORMATS and _ERROR_EXPORT_FIELDS constants. - Cached-results path regenerates missing _errors.* files; --overwrite deletes them. Misclassification filter (validate_dem.py): - New filter_misclassified_photons() drops cells with large errors driven by mis-classified ICESat-2 photons (false offshore bathy/ground, onshore bathy), using a DEM-aligned coastline mask from coastline_mask.get_or_create_coastline_mask(). - --no_misclassification_filter CLI flag to disable; filter_misclassified kwarg threaded through validate_dem -> validate_dem_parallel -> _write_validation_outputs. - Remove redundant "Discarded N likely-misclassified photons" print; keep only the per-cell count line. Coastline mask CRS fix (coastline_mask.py): - DEMs with a compound CRS (e.g. NAD83 + NAVD88) caused gdal.Warp to treat the binary 0/1 mask pixel values as elevations and apply the NAVD88 geoid shift (~20-26 m in coastal Oregon), producing mask values of 20-26 instead of 0/1. - Fix: call osr.SpatialReference.StripVertical() to strip the vertical component before using the CRS as the warp destination SRS. GPKG geometry fix (validate_dem.py): - Replace ogr.Geometry.AddPoint(x, y) with AddPoint_2D(x, y) in the GPKG/SHP exporter. AddPoint creates a 3D geometry (Z=0) that conflicts with the wkbPoint layer type, producing a GDAL warning. AddPoint_2D creates a true 2D geometry.
…ng blank placeholders When only topo or only bathy data is present, the figure now has 2 panels (A, B) instead of 3. Panel count and figure width scale together; labels are assigned sequentially so the scatter plot always gets the last letter.
Renames the setup group and all associated functions, internal names, and user-facing strings to use 'options' throughout cli.py, docs, and config.
… 429s
Without this, _get_bldg_tree() was called inside yield_chunks() for every
ATL03 granule. On a 429 the cache file was never written, so each granule
independently hammered the GBA server with up to 5 retries each. Now
fetchez.get("gba") runs once per tile before the granule loop; if it
succeeds the cache file satisfies all subsequent per-granule lookups at
zero extra HTTP cost. If it fails (server busy or empty coverage),
use_external_masks=False is passed to every granule so processing
continues with the algorithmic building classifier only.
Rewrites README with capabilities overview, install instructions, and quick start. Adds docs for validate, database, cache, options, and upgrade subcommands. Renames docs/configfile.md to docs/options.md to match the renamed CLI command. Bumps VERSION to 0.6.0-beta.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ivert validate,ivert database,ivert cache,ivert options,ivert upgrade)pyproject.toml; boto3 dependency removed from clientdocs/validate.md,docs/database.md,docs/cache.md,docs/options.md,docs/upgrade.md)Test plan
ivert database download <bbox>fetches and stores granules locallyivert validate <dem.tif>runs end-to-end and produces.h5,.png, and error exportsivert validatewith bathymetric DEM uses ATL24 photons (class code 40/41)ivert options listshows current config;ivert options setpersists a changeivert cache listandivert cache deletework correctlypip install ivertresolves all dependencies includingfetchez,globato,transformez