V1 Production Launch#646
Conversation
Bumps [packaging](https://github.com/pypa/packaging) from 25.0 to 26.0. - [Release notes](https://github.com/pypa/packaging/releases) - [Changelog](https://github.com/pypa/packaging/blob/main/CHANGELOG.rst) - [Commits](pypa/packaging@25.0...26.0) --- updated-dependencies: - dependency-name: packaging dependency-version: '26.0' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [pycparser](https://github.com/eliben/pycparser) from 2.23 to 3.0. - [Release notes](https://github.com/eliben/pycparser/releases) - [Commits](eliben/pycparser@release_v2.23...release_v3.00) --- updated-dependencies: - dependency-name: pycparser dependency-version: '3.0' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [cffi](https://github.com/python-cffi/cffi) from 1.17.1 to 2.0.0. - [Release notes](https://github.com/python-cffi/cffi/releases) - [Commits](python-cffi/cffi@v1.17.1...v2.0.0) --- updated-dependencies: - dependency-name: cffi dependency-version: 2.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
…inventory-csv feature - Move site_name, elevation_ft, elevation_method, and measuring_point_height_ft from required to optional - Replace "both contact_name and contact_organization required" rule with "at least one required" rule - Remove all-or-nothing water level rule; water_level_date_time now required only when depth_to_water_ft is provided, all other water level fields are independent and optional - Add negative scenarios for invalid address_type, state abbreviation, well_hole_status, monitoring_status, and well_pump_type with allowed values specified - Add well_notes, well_measuring_notes, water_notes, historical_notes, well_hole_status, and monitoring_status to optional fields
…inventory-csv feature - removed production tag - removed instances of returning in JSON format - Move site_name, elevation_ft, elevation_method, and measuring_point_height_ft from required to optional - Replace "both contact_name and contact_organization required" rule with "at least one required" rule - Remove all-or-nothing water level rule; water_level_date_time now required only when depth_to_water_ft is provided, all other water level fields are independent and optional - Add negative scenarios for invalid address_type, state abbreviation, well_hole_status, monitoring_status, and well_pump_type with allowed values specified - Add well_notes, well_measuring_notes, water_notes, historical_notes, well_hole_status, and monitoring_status to optional fields
…yle field requirements - Update water-level feature scenarios to mirror well-inventory behavior for requested headers - Keep only base required fields (`field_staff`, `well_name_point_id`, `field_event_date_time`) - Move water-level measurement fields to optional set (`water_level_date_time`, `measuring_person`, `sample_method`, `mp_height`, `level_status`, `depth_to_water_ft`, `data_quality`, `water_level_notes`) - Add timezone-naive ISO 8601 expectation for `field_event_date_time` and `water_level_date_time` - Fix required-field examples and column-order scenario to match new required set - Normalize numeric validation wording to `mp_height` (not `mp_height_ft`)`
feat(tests): relax validation rules and expand enum coverage in well-inventory-csv feature
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…e error extraction
…cleanup feat(tests): add validation error handling for various invalid CSV field values
…elds - Update water-level CSV feature to require lexicon-valid values (when provided) for: - sample_method - level_status - data_quality - Replace free-text acceptance scenario with negative validation scenario for invalid lexicon values - Keep failure expectations consistent: non-zero exit, field/row validation errors, and no imports`
`test(features): enforce lexicon values for water-level descriptor fields
…elated configurations
…with local development setup
…and update related configurations
…mprove database readiness checks
…dependency feat: update test configuration to use specific PostGIS version and improve database readiness checks
feat: add normalized chemistry results materialized view and update related configurations
- Introduced `validation_alias` with `AliasChoices` for selected fields (`well_status`, `sampler`, `measurement_date_time`, `mp_height`) to allow alternate field names. - Ensured alignment with schema validation updates.
- Introduced unit tests for `WellInventoryRow` alias mappings. - Verified correct handling of alias fields like `well_hole_status`, `mp_height_ft`, and others. - Ensured canonical fields take precedence when both alias and canonical values are provided.
… and new fields - Added `flexible_lexicon_validator` to support case-insensitive validation of enum-like fields. - Introduced new fields: `OriginType`, `WellPumpType`, `MonitoringStatus`, among others. - Updated existing fields to use flexible lexicon validation for improved consistency. - Adjusted `WellInventoryRow` optional fields handling and validation rules. - Refined contact field validation logic to require `role` and `type` when other contact details are provided.
…dations - Refined validation error handling to provide more detailed feedback in test assertions. - Adjusted test setup to ensure accurate validation scenarios for contact and water level fields. - Updated contact-related tests to validate new composite field error messages.
kas-water-level-import
…orts Preserve well inventory re-import idempotency while blocking creation of a new water well when the same `Thing.name` already exists in the database. Also keep the water-level importer defensive against ambiguous well lookups so duplicate well names produce row-level validation errors instead of crashing the CLI with `MultipleResultsFound`. This reduces the risk of creating duplicate `Thing` records and makes import failures clearer for operators.
Extract the generic water-well lookup by `Thing.name` into a shared helper and reuse it in the well inventory and water-level CSV importers. Keep importer-specific behavior separate: - water-level import still distinguishes unknown vs ambiguous matches - well-inventory import still applies its own re-import detection before enforcing the duplicate-name check
Bumps [cryptography](https://github.com/pyca/cryptography) from 46.0.6 to 46.0.7. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@46.0.6...46.0.7) --- updated-dependencies: - dependency-name: cryptography dependency-version: 46.0.7 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pdates BDMS-658: add location properties and contact summaries to well responses
Add a new `oco scoped-transfer` command and the supporting scoped transfer service for running legacy transfers by PointID. This adds scoped planning and execution, dependency expansion, family-level summary output, scoped well handling, scoped water-level idempotency, and contact-collision fallback reuse. It also suppresses legacy transfer warnings that are misleading in scoped CLI runs.
…coped transfer CLI and services Add focused tests for scoped transfer CLI output and planning behavior. This covers water-level contact collision reuse, scoped duplicate filtering and scoped log suppression for the targeted transfer workflow.
Introduce a comprehensive guide for the `oco scoped-transfer` CLI command, including examples, common scenarios, troubleshooting tips, and JSON output details. Also, link the guide in `README.md`.
…-cli kas-scoped-transfer-cli
…cate-logic-check NO TICKET fix(importers): prevent duplicate well-name collisions in well inventory and water-level imports
There was a problem hiding this comment.
Pull request overview
This PR packages the “V1 Production Launch” by tightening production/runtime bootstrapping, expanding pygeoapi/OGC outputs, improving transfer & CSV ingestion behavior, and hardening test/CI/deploy workflows to support a production-grade release.
Changes:
- Refactors app initialization (factory-based app creation, lazy admin mounting, request/DB/GCS timing instrumentation).
- Expands data/OGC capabilities (new pygeoapi collections + materialized/view migrations; well details/export payloads; water level CSV schema/validation).
- Updates infra/testing for production (Docker Compose DB init, CI PostGIS service, env handling, transfer parallelism fixes).
Reviewed changes
Copilot reviewed 119 out of 121 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| transfers/well_transfer.py | Lazy-loads external resources; improves parallel transfer safety; adds monitoring_status handling. |
| transfers/transfer.py | Switches get_bool_env import to services.env. |
| transfers/backfill/backfill.py | Switches get_bool_env import to services.env. |
| tests/transfers/test_contact_with_multiple_wells.py | Updates contact dedupe test inputs to match new helper signatures/behavior. |
| tests/test_water_level_csv_schema.py | Adds unit tests for new WaterLevelCsvRow normalization/validation. |
| tests/test_sensor.py | Makes sensor list tests resilient to shared DB state. |
| tests/test_sample.py | Makes sample list tests resilient to shared DB state. |
| tests/test_request_timing.py | Adds tests asserting request lifecycle timing logs. |
| tests/test_pygeoapi_mount.py | Adds tests for pygeoapi app loading/reload behavior. |
| tests/test_location.py | Makes location list test resilient to shared DB state. |
| tests/test_lazy_admin.py | Adds test ensuring admin routes are lazy-loaded on demand. |
| tests/test_asset.py | Adds tests for GCS upload timing, dedupe, and file rewind behavior; minor assertions cleanup. |
| tests/integration/test_alembic_migrations.py | Adds assertion for water elevation matview column contract. |
| tests/features/steps/well-inventory-csv.py | Improves BDD assertions; adds monitoring frequency → monitoring status verification. |
| tests/features/steps/well-inventory-csv-validation-error.py | Makes validation error matching less order/shape dependent; adds new error assertions. |
| tests/features/steps/cli_common.py | Improves failure output for non-zero CLI exit assertions. |
| tests/features/environment.py | Sets default test DB env before imports; switches get_bool_env import. |
| tests/features/data/well-inventory-valid-comma-in-quotes.csv | Updates fixture row quoting and canonical lexicon values. |
| tests/features/data/well-inventory-missing-wl-fields.csv | Updates fixture values for canonical lexicon/status formatting. |
| tests/features/data/well-inventory-missing-required.csv | Updates elevation_method values to canonical lexicon terms. |
| tests/features/data/well-inventory-missing-phone-type.csv | Updates fixture values for canonical lexicon/status formatting. |
| tests/features/data/well-inventory-missing-email-type.csv | Updates fixture values for canonical lexicon/status formatting. |
| tests/features/data/well-inventory-missing-contact-type.csv | Updates fixture values; makes missing contact type scenario clearer. |
| tests/features/data/well-inventory-missing-contact-role.csv | Updates fixture values; makes missing contact role scenario clearer. |
| tests/features/data/well-inventory-missing-address-type.csv | Updates fixture values for canonical lexicon/status formatting. |
| tests/features/data/well-inventory-invalid-utm.csv | Updates fixture values; adds an extra valid row for mixed validity coverage. |
| tests/features/data/well-inventory-invalid-postal-code.csv | Updates fixture values for canonical lexicon/status formatting. |
| tests/features/data/well-inventory-invalid-phone-number.csv | Updates fixture values for canonical lexicon/status formatting. |
| tests/features/data/well-inventory-invalid-partial.csv | Updates fixture values for canonical lexicon/status formatting. |
| tests/features/data/well-inventory-invalid-numeric.csv | Updates elevation_method values to canonical lexicon terms. |
| tests/features/data/well-inventory-invalid-lexicon.csv | Updates fixture values to better align with role/type lexicons. |
| tests/features/data/well-inventory-invalid-email.csv | Updates fixture values for canonical lexicon/status formatting. |
| tests/features/data/well-inventory-invalid-date.csv | Updates elevation_method values to canonical lexicon terms. |
| tests/features/data/well-inventory-invalid-date-format.csv | Updates fixture values for canonical lexicon/status formatting. |
| tests/features/data/well-inventory-invalid-contact-type.csv | Updates fixture values for canonical lexicon/status formatting. |
| tests/features/data/well-inventory-invalid-boolean-value-maybe.csv | Updates fixture values for canonical lexicon/status formatting. |
| tests/features/data/well-inventory-duplicate.csv | Updates duplicate IDs to avoid collisions with other fixtures. |
| tests/features/data/well-inventory-duplicate-columns.csv | Updates fixture values for canonical lexicon/status formatting. |
| tests/conftest.py | Adjusts env loading/host normalization; disposes engine around schema ops; improves fixture cleanup. |
| tests/init.py | Changes dotenv override behavior; normalizes docker hostnames for host-run tests. |
| services/util.py | Removes env helpers moved to services.env. |
| services/thing_helper.py | Adds debug timing logs; adds water well contact eager loads; adds monitoring_status status history; changes rollback behavior. |
| services/sample_helper.py | Adds comprehensive eager loading options; adds get_sample_by_id_with_relationships. |
| services/query_helper.py | Moves to_bool import to services.env. |
| services/observation_helper.py | Refactors transducer observation query logic; adds timing logs for observation pagination. |
| services/ngwmn_helper.py | Tightens SQL column selection; avoids “None” XML text via helper. |
| services/gcs_helper.py | Adds caching, timeouts, stage timing logs, and safer file hashing/rewind behavior. |
| services/env.py | Introduces shared env parsing utilities (to_bool/get_bool_env). |
| services/contact_helper.py | Changes refresh/rollback behavior for commit vs flush flows. |
| services/asset_helper.py | Avoids runtime dependency on google storage types via TYPE_CHECKING forward ref. |
| schemas/well_export.py | Adds response schema for well export payload. |
| schemas/well_details.py | Adds response schema for consolidated well details payload. |
| schemas/water_level_csv.py | Implements WaterLevelCsvRow normalization/validation and bulk upload schema tweaks. |
| schemas/thing.py | Adds monitoring_status to CreateWell; adds WellContactSummaryResponse and WellResponse.contacts. |
| schemas/location.py | Extends GeoJSON properties with county/state/quad_name. |
| schemas/contact.py | Removes default contact_type; makes it required. |
| schedule | Updates Cloud Scheduler warmup instructions for production/staging. |
| README.md | Adds deployment/runtime notes; documents required env vars and Compose DB behavior. |
| pyproject.toml | Updates dependency pins (FastAPI, SQLAlchemy, google libs, etc.). |
| main.py | Switches to factory-based create_api_app entrypoint. |
| entrypoint.sh | Adds strict mode + configurable host/port/module; optional migrations; reload toggle. |
| docker/db/init/01-create-test-db.sql | Initializes ocotilloapi_test DB + PostGIS extension for dev/test DBs in Compose. |
| docker-compose.yml | Switches DB to postgis image; adds init scripts; hardcodes dev DB name; adds pygeoapi env. |
| db/transducer.py | Adds composite index for deployment/parameter/datetime query patterns. |
| db/engine.py | Adds optional DB pool event logging; introduces DB_POOL_TIMEOUT; switches get_bool_env import. |
| core/pygeoapi-config.yml | Adds new OGC collections (water elevation, chemistry, actively monitored wells, project areas). |
| core/permissions.py | Caches JWKS retrieval; adds timeouts; avoids startup-time JWKS fetch. |
| core/lexicon.json | Adds new organizations and note type; updates lexicon terms used across imports. |
| core/initializers.py | Splits API route registration; decomposes middleware; adds lazy admin configuration. |
| core/factory.py | Introduces runtime initialization + app factory; mounts pygeoapi and lazy admin. |
| core/enums.py | Updates MonitoringStatus enum to build from status_value category. |
| cli/service_adapter.py | Improves CSV decoding (BOM); adds progress callback; improves CLI stderr/exit-code rules. |
| cli/README.md | Adds restore/scoped transfer guidance link updates. |
| cli/project_area_import.py | Adds CLI helper to ingest/update project area polygon boundaries. |
| api/thing.py | Adds well details/export endpoints returning consolidated payloads. |
| api/sample.py | Uses relationship-loaded sample retrieval; refactors formatting/handlers. |
| api/asset.py | Adds timing + threadpool for uploads; refactors bucket dependency handling. |
| alembic/versions/t6u7v8w9x0y1_add_project_areas_ogc_view.py | Adds ogc_project_areas view migration. |
| alembic/versions/s4t5u6v7w8x9_drop_unused_well_type_ogc_views.py | Drops unused OGC views; safe name checks; downgrade recreates. |
| alembic/versions/r2s3t4u5v6w7_add_actively_monitored_wells_pygeoapi_materialized_view.py | Adds view for actively monitored wells based on status history + group membership. |
| alembic/versions/p9c0d1e2f3a4_add_transducer_observation_deployment_index.py | Adds DB index migration for transducer observation lookups. |
| alembic/versions/o8b9c0d1e2f3_rebuild_water_elevation_materialized_view.py | Rebuilds water elevation matview with explicit unit handling and indexes. |
| alembic/versions/n7a8b9c0d1e2_fix_water_elevation_units_to_feet.py | Fixes water elevation units to feet; maintains downgrade path. |
| alembic/versions/m6f7a8b9c0d1_add_water_elevation_materialized_view.py | Introduces initial water elevation matview. |
| alembic/versions/d5e6f7a8b9c0_create_pygeoapi_supporting_views.py | Formatting/maintainability improvements to pygeoapi supporting view migration. |
| alembic/versions/c7f8a9b0c1d2_add_thing_id_to_nma_surface_water_data.py | Formatting improvement in alter_column call. |
| alembic/env.py | Switches get_bool_env import to services.env. |
| AGENTS.MD | Adds post-change cleanup/code-analysis guidance. |
| .github/workflows/tests.yml | Moves CI DB to Actions service container; adds explicit test DB creation step; updates actions versions. |
| .github/workflows/jira_codex_pr.yml | Updates pinned setup-uv action ref. |
| .github/workflows/CD_staging.yml | Adds refresh matviews step; renders more App Engine config via envsubst; improves version cleanup logic. |
| .github/workflows/CD_production.yml | Mirrors staging deploy changes for production; fixes job naming/tag message. |
| .github/app.template.yaml | Parameterizes entrypoint and scaling; enables warmup inbound service. |
| .env.example | Updates default POSTGRES_DB to ocotilloapi_dev. |
Comments suppressed due to low confidence (1)
tests/init.py:44
- tests/init.py unconditionally overwrites POSTGRES_PORT to "5432", which defeats the earlier goal of letting explicit shell env vars override
.envand can make it hard to run tests against a non-default mapped port. Consider usingos.environ.setdefault("POSTGRES_PORT", "5432")(or leaving the value untouched) so callers can override it; also the comment about avoiding the production DB port is misleading if you always force 5432.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8cd1103cfd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Bumps [actions/github-script](https://github.com/actions/github-script) from 8 to 9. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@v8...v9) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: '9' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [dependabot/fetch-metadata](https://github.com/dependabot/fetch-metadata) from 2 to 3. - [Release notes](https://github.com/dependabot/fetch-metadata/releases) - [Commits](dependabot/fetch-metadata@v2...v3) --- updated-dependencies: - dependency-name: dependabot/fetch-metadata dependency-version: '3' dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com>
…ctions/staging/actions/github-script-9 build(deps): bump actions/github-script from 8 to 9
…ctions/staging/dependabot/fetch-metadata-3 build(deps): bump dependabot/fetch-metadata from 2 to 3
Bumps [pytest](https://github.com/pytest-dev/pytest) from 9.0.2 to 9.0.3. - [Release notes](https://github.com/pytest-dev/pytest/releases) - [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst) - [Commits](pytest-dev/pytest@9.0.2...9.0.3) --- updated-dependencies: - dependency-name: pytest dependency-version: 9.0.3 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 121 out of 123 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
tests/init.py:44
- The comment says "for safety don't test on the production database port", but this hard-sets
POSTGRES_PORTto5432(the default Postgres prod port). Either update the comment to match the intent, or set a non-prod port here (and ensure CI/docker config aligns).
| from services.gcs_helper import gcs_remove | ||
|
|
||
| asset = simple_get_by_id(session, Asset, asset_id) | ||
| gcs_remove(asset.uri, bucket) |
There was a problem hiding this comment.
This delete endpoint calls gcs_remove(asset.uri, bucket), but asset.uri appears to be a full GCS URL (as returned by the upload endpoint). gcs_remove/bucket.blob(...) expects a blob name; this likely won’t delete anything. Use asset.storage_path (or otherwise extract the blob name from the URI) when removing the object.
| gcs_remove(asset.uri, bucket) | |
| gcs_remove(asset.storage_path, bucket) |
| MonitoringStatus: type[Enum] = build_enum_from_lexicon_category("status_value") | ||
| ParameterName: type[Enum] = build_enum_from_lexicon_category("parameter_name") |
There was a problem hiding this comment.
MonitoringStatus is now built from the status_value lexicon category, which (per lexicon.json) includes values for multiple status types (e.g., Open/Closed, Datalogger suitability, Well Status) in addition to monitoring statuses. This broadens validation so fields typed as MonitoringStatus can accept non-monitoring values. Consider splitting into separate enums per status_type (or introducing a dedicated monitoring_status category) and/or renaming this enum to reflect its broader scope.
| @lru_cache(maxsize=1) | ||
| def get_jwks(): | ||
| if not AUTHENTIK_ISSUER or auth_disabled: | ||
| return {} | ||
|
|
||
| resp = httpx.get(JWKS_URL, timeout=10.0) | ||
| resp.raise_for_status() | ||
| return resp.json() | ||
|
|
||
|
|
||
| def get_public_key(token): | ||
| unverified_header = jwt.get_unverified_header(token) | ||
| for key in jwks["keys"]: | ||
| for key in get_jwks().get("keys", []): | ||
| if key["kid"] == unverified_header["kid"]: | ||
| return RSAAlgorithm.from_jwk(key) | ||
| raise HTTPException(status_code=401, detail="Invalid signing key") |
There was a problem hiding this comment.
get_jwks() is cached indefinitely. If Authentik rotates signing keys, this process will keep using the stale JWKS and start rejecting valid tokens until restart. A common mitigation is: if kid is not found, clear the cache and refetch once before failing (or add a TTL-based cache).
[BDMS-690 & BDMS-689]: Update the columns on the Wells list page to include contact info
V1 Production Launch