Skip to content

V1 Production Launch#646

Merged
jirhiker merged 210 commits into
productionfrom
staging
Apr 13, 2026
Merged

V1 Production Launch#646
jirhiker merged 210 commits into
productionfrom
staging

Conversation

@jirhiker
Copy link
Copy Markdown
Member

V1 Production Launch

dependabot Bot and others added 30 commits February 13, 2026 16:52
Bumps [packaging](https://github.com/pypa/packaging) from 25.0 to 26.0.
- [Release notes](https://github.com/pypa/packaging/releases)
- [Changelog](https://github.com/pypa/packaging/blob/main/CHANGELOG.rst)
- [Commits](pypa/packaging@25.0...26.0)

---
updated-dependencies:
- dependency-name: packaging
  dependency-version: '26.0'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [pycparser](https://github.com/eliben/pycparser) from 2.23 to 3.0.
- [Release notes](https://github.com/eliben/pycparser/releases)
- [Commits](eliben/pycparser@release_v2.23...release_v3.00)

---
updated-dependencies:
- dependency-name: pycparser
  dependency-version: '3.0'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [cffi](https://github.com/python-cffi/cffi) from 1.17.1 to 2.0.0.
- [Release notes](https://github.com/python-cffi/cffi/releases)
- [Commits](python-cffi/cffi@v1.17.1...v2.0.0)

---
updated-dependencies:
- dependency-name: cffi
  dependency-version: 2.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
…inventory-csv feature

- Move site_name, elevation_ft, elevation_method, and measuring_point_height_ft from required to optional
- Replace "both contact_name and contact_organization required" rule with "at least one required" rule
- Remove all-or-nothing water level rule; water_level_date_time now required only when depth_to_water_ft is provided, all other water level fields are independent and optional
- Add negative scenarios for invalid address_type, state abbreviation, well_hole_status, monitoring_status, and well_pump_type with allowed values specified
- Add well_notes, well_measuring_notes, water_notes, historical_notes, well_hole_status, and monitoring_status to optional fields
…inventory-csv feature

- removed production tag
- removed instances of returning in JSON format
- Move site_name, elevation_ft, elevation_method, and measuring_point_height_ft from required to optional
- Replace "both contact_name and contact_organization required" rule with "at least one required" rule
- Remove all-or-nothing water level rule; water_level_date_time now required only when depth_to_water_ft is provided, all other water level fields are independent and optional
- Add negative scenarios for invalid address_type, state abbreviation, well_hole_status, monitoring_status, and well_pump_type with allowed values specified
- Add well_notes, well_measuring_notes, water_notes, historical_notes, well_hole_status, and monitoring_status to optional fields
…yle field requirements

- Update water-level feature scenarios to mirror well-inventory behavior for requested headers
- Keep only base required fields (`field_staff`, `well_name_point_id`, `field_event_date_time`)
- Move water-level measurement fields to optional set (`water_level_date_time`, `measuring_person`, `sample_method`, `mp_height`, `level_status`, `depth_to_water_ft`, `data_quality`, `water_level_notes`)
- Add timezone-naive ISO 8601 expectation for `field_event_date_time` and `water_level_date_time`
- Fix required-field examples and column-order scenario to match new required set
- Normalize numeric validation wording to `mp_height` (not `mp_height_ft`)`
feat(tests): relax validation rules and expand enum coverage in well-inventory-csv feature
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…cleanup

feat(tests): add validation error handling for various invalid CSV field values
…elds

- Update water-level CSV feature to require lexicon-valid values (when provided) for:
  - sample_method
  - level_status
  - data_quality
- Replace free-text acceptance scenario with negative validation scenario for invalid lexicon values
- Keep failure expectations consistent: non-zero exit, field/row validation errors, and no imports`
`test(features): enforce lexicon values for water-level descriptor fields
…dependency

feat: update test configuration to use specific PostGIS version and improve database readiness checks
feat: add normalized chemistry results materialized view and update related configurations
- Introduced `validation_alias` with `AliasChoices` for selected fields (`well_status`, `sampler`, `measurement_date_time`, `mp_height`) to allow alternate field names.
- Ensured alignment with schema validation updates.
- Introduced unit tests for `WellInventoryRow` alias mappings.
- Verified correct handling of alias fields like `well_hole_status`, `mp_height_ft`, and others.
- Ensured canonical fields take precedence when both alias and canonical values are provided.
… and new fields

- Added `flexible_lexicon_validator` to support case-insensitive validation of enum-like fields.
- Introduced new fields: `OriginType`, `WellPumpType`, `MonitoringStatus`, among others.
- Updated existing fields to use flexible lexicon validation for improved consistency.
- Adjusted `WellInventoryRow` optional fields handling and validation rules.
- Refined contact field validation logic to require `role` and `type` when other contact details are provided.
…dations

- Refined validation error handling to provide more detailed feedback in test assertions.
- Adjusted test setup to ensure accurate validation scenarios for contact and water level fields.
- Updated contact-related tests to validate new composite field error messages.
ksmuczynski and others added 10 commits March 30, 2026 22:41
…orts

Preserve well inventory re-import idempotency while blocking creation of a new water well when the same `Thing.name` already exists in the database.

Also keep the water-level importer defensive against ambiguous well lookups so duplicate well names produce row-level validation errors instead of crashing the CLI with `MultipleResultsFound`.

This reduces the risk of creating duplicate `Thing` records and makes import failures clearer for operators.
Extract the generic water-well lookup by `Thing.name` into a shared helper and reuse it in the well inventory and water-level CSV importers.

Keep importer-specific behavior separate:
- water-level import still distinguishes unknown vs ambiguous matches
- well-inventory import still applies its own re-import detection before enforcing the duplicate-name check
Bumps [cryptography](https://github.com/pyca/cryptography) from 46.0.6 to 46.0.7.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst)
- [Commits](pyca/cryptography@46.0.6...46.0.7)

---
updated-dependencies:
- dependency-name: cryptography
  dependency-version: 46.0.7
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…pdates

BDMS-658: add location properties and contact summaries to well responses
Add a new `oco scoped-transfer` command and the supporting scoped transfer service for running legacy transfers by PointID.

This adds scoped planning and execution, dependency expansion, family-level summary output, scoped well handling, scoped water-level idempotency, and contact-collision fallback reuse. It also suppresses legacy transfer warnings that are misleading in scoped CLI runs.
…coped transfer CLI and services

Add focused tests for scoped transfer CLI output and planning behavior.

This covers water-level contact collision reuse, scoped duplicate filtering and scoped log suppression for the targeted transfer workflow.
Introduce a comprehensive guide for the `oco scoped-transfer` CLI command, including examples, common scenarios, troubleshooting tips, and JSON output details. Also, link the guide in `README.md`.
…cate-logic-check

NO TICKET fix(importers): prevent duplicate well-name collisions in well inventory and water-level imports
Copilot AI review requested due to automatic review settings April 13, 2026 15:21
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR packages the “V1 Production Launch” by tightening production/runtime bootstrapping, expanding pygeoapi/OGC outputs, improving transfer & CSV ingestion behavior, and hardening test/CI/deploy workflows to support a production-grade release.

Changes:

  • Refactors app initialization (factory-based app creation, lazy admin mounting, request/DB/GCS timing instrumentation).
  • Expands data/OGC capabilities (new pygeoapi collections + materialized/view migrations; well details/export payloads; water level CSV schema/validation).
  • Updates infra/testing for production (Docker Compose DB init, CI PostGIS service, env handling, transfer parallelism fixes).

Reviewed changes

Copilot reviewed 119 out of 121 changed files in this pull request and generated 6 comments.

Show a summary per file
File Description
transfers/well_transfer.py Lazy-loads external resources; improves parallel transfer safety; adds monitoring_status handling.
transfers/transfer.py Switches get_bool_env import to services.env.
transfers/backfill/backfill.py Switches get_bool_env import to services.env.
tests/transfers/test_contact_with_multiple_wells.py Updates contact dedupe test inputs to match new helper signatures/behavior.
tests/test_water_level_csv_schema.py Adds unit tests for new WaterLevelCsvRow normalization/validation.
tests/test_sensor.py Makes sensor list tests resilient to shared DB state.
tests/test_sample.py Makes sample list tests resilient to shared DB state.
tests/test_request_timing.py Adds tests asserting request lifecycle timing logs.
tests/test_pygeoapi_mount.py Adds tests for pygeoapi app loading/reload behavior.
tests/test_location.py Makes location list test resilient to shared DB state.
tests/test_lazy_admin.py Adds test ensuring admin routes are lazy-loaded on demand.
tests/test_asset.py Adds tests for GCS upload timing, dedupe, and file rewind behavior; minor assertions cleanup.
tests/integration/test_alembic_migrations.py Adds assertion for water elevation matview column contract.
tests/features/steps/well-inventory-csv.py Improves BDD assertions; adds monitoring frequency → monitoring status verification.
tests/features/steps/well-inventory-csv-validation-error.py Makes validation error matching less order/shape dependent; adds new error assertions.
tests/features/steps/cli_common.py Improves failure output for non-zero CLI exit assertions.
tests/features/environment.py Sets default test DB env before imports; switches get_bool_env import.
tests/features/data/well-inventory-valid-comma-in-quotes.csv Updates fixture row quoting and canonical lexicon values.
tests/features/data/well-inventory-missing-wl-fields.csv Updates fixture values for canonical lexicon/status formatting.
tests/features/data/well-inventory-missing-required.csv Updates elevation_method values to canonical lexicon terms.
tests/features/data/well-inventory-missing-phone-type.csv Updates fixture values for canonical lexicon/status formatting.
tests/features/data/well-inventory-missing-email-type.csv Updates fixture values for canonical lexicon/status formatting.
tests/features/data/well-inventory-missing-contact-type.csv Updates fixture values; makes missing contact type scenario clearer.
tests/features/data/well-inventory-missing-contact-role.csv Updates fixture values; makes missing contact role scenario clearer.
tests/features/data/well-inventory-missing-address-type.csv Updates fixture values for canonical lexicon/status formatting.
tests/features/data/well-inventory-invalid-utm.csv Updates fixture values; adds an extra valid row for mixed validity coverage.
tests/features/data/well-inventory-invalid-postal-code.csv Updates fixture values for canonical lexicon/status formatting.
tests/features/data/well-inventory-invalid-phone-number.csv Updates fixture values for canonical lexicon/status formatting.
tests/features/data/well-inventory-invalid-partial.csv Updates fixture values for canonical lexicon/status formatting.
tests/features/data/well-inventory-invalid-numeric.csv Updates elevation_method values to canonical lexicon terms.
tests/features/data/well-inventory-invalid-lexicon.csv Updates fixture values to better align with role/type lexicons.
tests/features/data/well-inventory-invalid-email.csv Updates fixture values for canonical lexicon/status formatting.
tests/features/data/well-inventory-invalid-date.csv Updates elevation_method values to canonical lexicon terms.
tests/features/data/well-inventory-invalid-date-format.csv Updates fixture values for canonical lexicon/status formatting.
tests/features/data/well-inventory-invalid-contact-type.csv Updates fixture values for canonical lexicon/status formatting.
tests/features/data/well-inventory-invalid-boolean-value-maybe.csv Updates fixture values for canonical lexicon/status formatting.
tests/features/data/well-inventory-duplicate.csv Updates duplicate IDs to avoid collisions with other fixtures.
tests/features/data/well-inventory-duplicate-columns.csv Updates fixture values for canonical lexicon/status formatting.
tests/conftest.py Adjusts env loading/host normalization; disposes engine around schema ops; improves fixture cleanup.
tests/init.py Changes dotenv override behavior; normalizes docker hostnames for host-run tests.
services/util.py Removes env helpers moved to services.env.
services/thing_helper.py Adds debug timing logs; adds water well contact eager loads; adds monitoring_status status history; changes rollback behavior.
services/sample_helper.py Adds comprehensive eager loading options; adds get_sample_by_id_with_relationships.
services/query_helper.py Moves to_bool import to services.env.
services/observation_helper.py Refactors transducer observation query logic; adds timing logs for observation pagination.
services/ngwmn_helper.py Tightens SQL column selection; avoids “None” XML text via helper.
services/gcs_helper.py Adds caching, timeouts, stage timing logs, and safer file hashing/rewind behavior.
services/env.py Introduces shared env parsing utilities (to_bool/get_bool_env).
services/contact_helper.py Changes refresh/rollback behavior for commit vs flush flows.
services/asset_helper.py Avoids runtime dependency on google storage types via TYPE_CHECKING forward ref.
schemas/well_export.py Adds response schema for well export payload.
schemas/well_details.py Adds response schema for consolidated well details payload.
schemas/water_level_csv.py Implements WaterLevelCsvRow normalization/validation and bulk upload schema tweaks.
schemas/thing.py Adds monitoring_status to CreateWell; adds WellContactSummaryResponse and WellResponse.contacts.
schemas/location.py Extends GeoJSON properties with county/state/quad_name.
schemas/contact.py Removes default contact_type; makes it required.
schedule Updates Cloud Scheduler warmup instructions for production/staging.
README.md Adds deployment/runtime notes; documents required env vars and Compose DB behavior.
pyproject.toml Updates dependency pins (FastAPI, SQLAlchemy, google libs, etc.).
main.py Switches to factory-based create_api_app entrypoint.
entrypoint.sh Adds strict mode + configurable host/port/module; optional migrations; reload toggle.
docker/db/init/01-create-test-db.sql Initializes ocotilloapi_test DB + PostGIS extension for dev/test DBs in Compose.
docker-compose.yml Switches DB to postgis image; adds init scripts; hardcodes dev DB name; adds pygeoapi env.
db/transducer.py Adds composite index for deployment/parameter/datetime query patterns.
db/engine.py Adds optional DB pool event logging; introduces DB_POOL_TIMEOUT; switches get_bool_env import.
core/pygeoapi-config.yml Adds new OGC collections (water elevation, chemistry, actively monitored wells, project areas).
core/permissions.py Caches JWKS retrieval; adds timeouts; avoids startup-time JWKS fetch.
core/lexicon.json Adds new organizations and note type; updates lexicon terms used across imports.
core/initializers.py Splits API route registration; decomposes middleware; adds lazy admin configuration.
core/factory.py Introduces runtime initialization + app factory; mounts pygeoapi and lazy admin.
core/enums.py Updates MonitoringStatus enum to build from status_value category.
cli/service_adapter.py Improves CSV decoding (BOM); adds progress callback; improves CLI stderr/exit-code rules.
cli/README.md Adds restore/scoped transfer guidance link updates.
cli/project_area_import.py Adds CLI helper to ingest/update project area polygon boundaries.
api/thing.py Adds well details/export endpoints returning consolidated payloads.
api/sample.py Uses relationship-loaded sample retrieval; refactors formatting/handlers.
api/asset.py Adds timing + threadpool for uploads; refactors bucket dependency handling.
alembic/versions/t6u7v8w9x0y1_add_project_areas_ogc_view.py Adds ogc_project_areas view migration.
alembic/versions/s4t5u6v7w8x9_drop_unused_well_type_ogc_views.py Drops unused OGC views; safe name checks; downgrade recreates.
alembic/versions/r2s3t4u5v6w7_add_actively_monitored_wells_pygeoapi_materialized_view.py Adds view for actively monitored wells based on status history + group membership.
alembic/versions/p9c0d1e2f3a4_add_transducer_observation_deployment_index.py Adds DB index migration for transducer observation lookups.
alembic/versions/o8b9c0d1e2f3_rebuild_water_elevation_materialized_view.py Rebuilds water elevation matview with explicit unit handling and indexes.
alembic/versions/n7a8b9c0d1e2_fix_water_elevation_units_to_feet.py Fixes water elevation units to feet; maintains downgrade path.
alembic/versions/m6f7a8b9c0d1_add_water_elevation_materialized_view.py Introduces initial water elevation matview.
alembic/versions/d5e6f7a8b9c0_create_pygeoapi_supporting_views.py Formatting/maintainability improvements to pygeoapi supporting view migration.
alembic/versions/c7f8a9b0c1d2_add_thing_id_to_nma_surface_water_data.py Formatting improvement in alter_column call.
alembic/env.py Switches get_bool_env import to services.env.
AGENTS.MD Adds post-change cleanup/code-analysis guidance.
.github/workflows/tests.yml Moves CI DB to Actions service container; adds explicit test DB creation step; updates actions versions.
.github/workflows/jira_codex_pr.yml Updates pinned setup-uv action ref.
.github/workflows/CD_staging.yml Adds refresh matviews step; renders more App Engine config via envsubst; improves version cleanup logic.
.github/workflows/CD_production.yml Mirrors staging deploy changes for production; fixes job naming/tag message.
.github/app.template.yaml Parameterizes entrypoint and scaling; enables warmup inbound service.
.env.example Updates default POSTGRES_DB to ocotilloapi_dev.
Comments suppressed due to low confidence (1)

tests/init.py:44

  • tests/init.py unconditionally overwrites POSTGRES_PORT to "5432", which defeats the earlier goal of letting explicit shell env vars override .env and can make it hard to run tests against a non-default mapped port. Consider using os.environ.setdefault("POSTGRES_PORT", "5432") (or leaving the value untouched) so callers can override it; also the comment about avoiding the production DB port is misleading if you always force 5432.

Comment thread entrypoint.sh
Comment thread services/gcs_helper.py
Comment thread services/gcs_helper.py
Comment thread api/asset.py
Comment thread services/thing_helper.py
Comment thread services/contact_helper.py
Copy link
Copy Markdown

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8cd1103cfd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread services/well_details_helper.py
Comment thread services/scoped_transfer.py
TylerAdamMartinez and others added 6 commits April 13, 2026 11:29
Bumps [actions/github-script](https://github.com/actions/github-script) from 8 to 9.
- [Release notes](https://github.com/actions/github-script/releases)
- [Commits](actions/github-script@v8...v9)

---
updated-dependencies:
- dependency-name: actions/github-script
  dependency-version: '9'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Bumps [dependabot/fetch-metadata](https://github.com/dependabot/fetch-metadata) from 2 to 3.
- [Release notes](https://github.com/dependabot/fetch-metadata/releases)
- [Commits](dependabot/fetch-metadata@v2...v3)

---
updated-dependencies:
- dependency-name: dependabot/fetch-metadata
  dependency-version: '3'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
…ctions/staging/actions/github-script-9

build(deps): bump actions/github-script from 8 to 9
…ctions/staging/dependabot/fetch-metadata-3

build(deps): bump dependabot/fetch-metadata from 2 to 3
Copilot AI review requested due to automatic review settings April 13, 2026 16:54
Bumps [pytest](https://github.com/pytest-dev/pytest) from 9.0.2 to 9.0.3.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@9.0.2...9.0.3)

---
updated-dependencies:
- dependency-name: pytest
  dependency-version: 9.0.3
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 121 out of 123 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

tests/init.py:44

  • The comment says "for safety don't test on the production database port", but this hard-sets POSTGRES_PORT to 5432 (the default Postgres prod port). Either update the comment to match the intent, or set a non-prod port here (and ensure CI/docker config aligns).

Comment thread api/asset.py
from services.gcs_helper import gcs_remove

asset = simple_get_by_id(session, Asset, asset_id)
gcs_remove(asset.uri, bucket)
Copy link

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This delete endpoint calls gcs_remove(asset.uri, bucket), but asset.uri appears to be a full GCS URL (as returned by the upload endpoint). gcs_remove/bucket.blob(...) expects a blob name; this likely won’t delete anything. Use asset.storage_path (or otherwise extract the blob name from the URI) when removing the object.

Suggested change
gcs_remove(asset.uri, bucket)
gcs_remove(asset.storage_path, bucket)

Copilot uses AI. Check for mistakes.
Comment thread core/enums.py
Comment on lines +51 to 52
MonitoringStatus: type[Enum] = build_enum_from_lexicon_category("status_value")
ParameterName: type[Enum] = build_enum_from_lexicon_category("parameter_name")
Copy link

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MonitoringStatus is now built from the status_value lexicon category, which (per lexicon.json) includes values for multiple status types (e.g., Open/Closed, Datalogger suitability, Well Status) in addition to monitoring statuses. This broadens validation so fields typed as MonitoringStatus can accept non-monitoring values. Consider splitting into separate enums per status_type (or introducing a dedicated monitoring_status category) and/or renaming this enum to reflect its broader scope.

Copilot uses AI. Check for mistakes.
Comment thread core/permissions.py
Comment on lines +40 to 55
@lru_cache(maxsize=1)
def get_jwks():
if not AUTHENTIK_ISSUER or auth_disabled:
return {}

resp = httpx.get(JWKS_URL, timeout=10.0)
resp.raise_for_status()
return resp.json()


def get_public_key(token):
unverified_header = jwt.get_unverified_header(token)
for key in jwks["keys"]:
for key in get_jwks().get("keys", []):
if key["kid"] == unverified_header["kid"]:
return RSAAlgorithm.from_jwk(key)
raise HTTPException(status_code=401, detail="Invalid signing key")
Copy link

Copilot AI Apr 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get_jwks() is cached indefinitely. If Authentik rotates signing keys, this process will keep using the stale JWKS and start rejecting valid tokens until restart. A common mitigation is: if kid is not found, clear the cache and refetch once before failing (or add a TTL-based cache).

Copilot uses AI. Check for mistakes.
[BDMS-690 & BDMS-689]: Update the columns on the Wells list page to include contact info
@jirhiker jirhiker merged commit 820ce0e into production Apr 13, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants