feat: add reproducible GIS project contract (open-gis-project/v1) - #5
Conversation
Add the reproducible-project-first contract for material multi-stage GIS analysis per #4: - SKILL.md: top-level instruction (reasoning may be exploratory, delivered analysis must be deterministic/inspectable/reproducible), module row, and hard rules (record fixes as data, never mutate sources, rerunnable without chat, validation as a pipeline stage, separate semantics from rendering). - references/project-spec.md: canonical open-gis-project/v1 schema covering source provenance/timestamps, data override semantics, deterministic processing steps, machine-readable validation (passed/failed/warning/ not_testable), semantic presentation + standard GIS UX defaults, provenance UX, QGIS output, and the rerun contract. - references/validation-and-ops.md: prefer machine-readable project validation over prose checklists. - references/qgis.md: project.qgz as a first-class view over derived datasets. - templates/: project.yaml, pipeline.py, presentation.yaml, validation.yaml. - examples/tartu-development/: fully-worked acceptance scenario with source provenance, assumptions, attribute correction + manually drawn geometry overrides, deterministic pipeline, and a validation report. Phase 3 (validate/run/inspect CLI) is intentionally deferred as 'later' tooling outside the skill repo scope.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9b8f912db1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # Source: Maa- ja Ruumiamet cadastral WFS, retrieved 2026-08-25T08:18:12+03:00 | ||
| # (See project.yaml sources.cadastral_parcels). For a headless run this | ||
| # pulls the WFS layer into data/source/parcels.gpkg via ogr2ogr, then: | ||
| parcels = con.read_parquet(DERIVED / "parcels_raw.parquet") |
There was a problem hiding this comment.
In a fresh checkout—the exact rerun scenario promised by the example—data/derived/parcels_raw.parquet and roads3301.parquet do not exist, and no preceding code downloads the documented WFS sources or creates either file. The pipeline therefore fails at this first read instead of reproducing the analysis; add the acquisition/conversion stages or commit explicitly pinned input artifacts.
Useful? React with 👍 / 👎.
| CREATE OR REPLACE TABLE parcels3301 AS | ||
| SELECT ST_Transform(geometry, '{ANALYSIS_CRS}') AS geometry, | ||
| cadastral_id |
There was a problem hiding this comment.
When parcel input is available, this SQL still cannot execute because DuckDB geometries do not carry a source CRS and the spatial ST_Transform geometry overload requires both source and target CRS arguments. The same two-argument call recurs while creating final_candidates; specify the known input CRS and target CRS (or omit this first conversion if the WFS materialization is already EPSG:3301).
Useful? React with 👍 / 👎.
| "status": "passed", | ||
| "features_checked": int( | ||
| con.execute( | ||
| "SELECT COUNT(*) FROM candidate_parcels WHERE NOT ST_IsValid(geometry)" | ||
| ).fetchone()[0] |
There was a problem hiding this comment.
For any result containing invalid geometries, this check counts those invalid rows but unconditionally labels the check and overall report as passed; it also reports that invalid-row count as features_checked. Consequently a project can remain validated despite failing a required geometry_valid gate. Derive the check and report statuses from the count and report the total inspected row count separately.
Useful? React with 👍 / 👎.
| con.execute(f""" | ||
| CREATE OR REPLACE TABLE planned_roads AS | ||
| SELECT ST_GeomFromGeoJSON(geometry) AS geometry, properties | ||
| FROM read_json_auto('{ov}', format='newline_delimited') AS js | ||
| """) |
There was a problem hiding this comment.
With the supplied override files, this function merely creates an otherwise unused planned_roads table: it never modifies the passed candidate_parcels table, and it never reads closed-poi.geojson at all. Since final_candidates is subsequently selected directly from candidate_parcels, neither declared override affects the derived analysis even though the manifest says both generate the final output.
Useful? React with 👍 / 👎.
…e kataloog Address review feedback on #5 and resolve #1: - examples/tartu-development: replace fabricated old geoportaal.maaamet.ee URLs (404) with verified, live sources: * roads -> ETAK WFS https://gsavalik.envir.ee/geoserver/etak/wfs (layer etak:e_501_tee_j) + documented ETAK download page * pois -> Geofabrik estonia extract (unchanged, live) * cadastre -> discovery via Ruumiandmete kataloog metadata catalog; documented as agency-served so the current parcel WFS is confirmed there before a production rerun (no fabricated URL) * license URLs -> geoportaal.maaruum.ee/eng - references/data-sources.md (issue #1): add Ruumiandmete kataloog (metadata.geoportaal.ee) as the authoritative Estonian geodata discovery point, with catalog URL and API reference. All cited URLs verified returning HTTP 200 at commit time.
Review fix: invalid source URLsGood catch — the example cited fabricated, outdated Source URLs are now verified live (HTTP 200 at commit time):
Cadastre note (honesty over fabrication): the full Maa- ja Ruumiamet cadastral-parcel geometry is agency-served and not openly callable from a public WFS (the open Environment Agency GeoServer exposes ETAK + cadastre support layers only). Rather than invent a working endpoint, the example now points cadastre discovery at the Ruumiandmete kataloog metadata catalog and documents that the current parcel WFS endpoint is confirmed there before a production rerun. Also resolves #1: added Ruumiandmete kataloog ( Ready for re-review. |
…add spatial-data downloads to docs Address follow-up on #5 and extend #1 coverage: - references/data-sources.md (Estonian section): * add general Maa- ja Ruumiamet spatial-data downloads index (https://geoportaal.maaamet.ee/eng/spatial-data-p58.html) noting bulk county/municipality downloads in GPKG/SHP/GeoJSON/DXF as a preferred, deterministic path over WFS paging. * add dedicated Cadastral Data page (https://geoportaal.maaamet.ee/eng/spatial-data/cadastral-data-p310.html) with per-county/per-municipality cadastral unit downloads. - examples/tartu-development/project.yaml: cadastre source now a concrete Tartu maakond GPKG bulk download from the Cadastral Data page (pinned, deterministic, no WFS paging) instead of the vague agency-WFS note. All cited URLs verified returning HTTP 200 at commit time (ETAK WFS verified via a live GetFeature request).
Maa- ja Ruumiamet (formerly Maa-amet) renamed its Geoportal domain; switch all referenced URLs from geoportaal.maaamet.ee to geoportaal.maaruum.ee in the Estonia section of references/data-sources.md and in the tartu example. Verified both domains serve identical content (same titles and cadastral county-download data; only Cloudflare-encoded email differs), and the new URLs return HTTP 200.
Add run_e2e.py, which exercises the full open-gis-project/v1 loop for the examples/tartu-development scenario and renders a self-contained dashboard.html AS A VIEW over the project artifacts: - deterministic source fixture (seeded RNG) standing in for the real county cadastre GPKG / ETAK WFS documented in project.yaml - DuckDB Spatial processing mirroring project.yaml processing.steps (area>=20000 m2 in EPSG:3301, planar distance <=2000 m to main roads) - machine-readable validation report (passed/failed/warning roll-up) - GeoJSON transformed to true WGS84 with pyproj (DuckDB EPSG:3301 handling is unreliable), landing on real Tartu coords (~26.68-26.73E, 58.36-58.39N) - dashboard with summary, filters, layer controls, assumptions, sources & provenance, overrides, warnings, validation, and a MapLibre map, with map bounds + status badge derived from the actual run Reproducibility verified: two runs produce byte-identical derived geometry (only the timestamped run_id differs). Dashboard verified in headless Chromium/PW (119 features rendered, no console errors). Data/source and data/derived are regenerable and git-ignored; run_e2e.py and dashboard.html are tracked.
Add write_qgis_project() to run_e2e.py, producing project.qgz (a .qgz is the .qgs project XML wrapped in a zip) that: - references the SAME derived/override datasets (final-candidates.gpkg GPKG layer + planned-road override) as the pipeline - no hidden analytical state - sets project CRS EPSG:3301 and semantic Result / Project-overrides groups - verifies every referenced datasource exists (GPKG layer name matched to 'final-candidates', not the internal table name) QGIS opens project.qgz directly for professional inspection/editing, and deliberate edits can be written back into the project override layer. QGIS project generation is reproducible across runs (verified identical datasource structure); only the final-candidates geometry stays fixed (119 features).
… in yaml + dashboard
Address feedback: the dashboard and project.yaml pointed only at general
website URLs whose download pages expose many files.
project.yaml sources.* now name the exact object per source and its schema:
- access.file.{name,on_source,table_name,format} - the precise downloadable
file/table (county GPKG, ETAK e_501_tee_j layer, estonia-latest.osm.pbf)
- access.downloaded_at - explicit download timestamp per source
- version.identifier + published_at already present
- schema.{columns,geometry,key} - exact column list per dataset
run_e2e.py now writes data/source/manifest.json capturing the ACTUAL file,
table name, row count, column count, column list, download timestamp and
version observed in this run (parcels=120 rows x 7 cols, roads=1 x 3, pois
as a documented n/a context source). The dashboard's Sources & provenance
panel renders these specifics per source (file path, table, rows/cols,
columns, downloaded_at, version, source URL).
Verified in headless Chromium: 3 source blocks with rows/cols, 126 rendered
features, no console errors.
…TAK roads - references/data-sources.md: document exact S3 download URL pattern for county and municipality cadastre GPKGs (https://s3.pilw.io/rp-kemit-kataster/ANDMED/) - examples/tartu-development/project.yaml: specify exact source URLs, archive files, extracted GPKG names, layer names ('Tartu maakond' and 'etak:e_501_tee_j'), row counts (79,056 parcels, 2,000 roads), and complete schemas (32 parcel columns) - examples/tartu-development/pipeline.py & run_e2e.py: actually download and process real Tartu county Cadastral GeoPackage (79,056 parcels) and live ETAK roads via DuckDB Spatial and pyproj - outputs: 295 real candidate parcels (6,662.9 ha) in Tartu linn matching land-use, size >= 20,000 m2, and road proximity <= 2,000 m (including scenario planned road) - dashboard.html & project.qgz: fully rendered from real geodata with interactive popups
…rtens - project.yaml: add assumption A3 (25-min walking catchment radius = 2,000 m at 4.8 km/h), add education_pois source (93 schools and kindergartens in Tartu), and multi-criteria spatial processing steps and output specs - pipeline.py & run_e2e.py: calculate planar walking distance and minutes to nearest school and kindergarten for all candidate parcels; evaluate suitability tiers: * Tier 1 Prime: highway access <= 2km AND <= 25min walk to BOTH school and KG (73 parcels, 2,586.9 ha) * Tier 2 Good: highway access <= 2km AND <= 25min walk to school OR KG (8 parcels, 169.5 ha) * Tier 3: highway access only (> 25min walk to school/KG) (214 parcels, 3,906.5 ha) - exports: generate education_catchments.json (2,000 m buffer MultiPolygons) and education_pois.json - dashboard.html: interactive MapLibre map with suitability tier color-coding, semi-transparent school/kindergarten 25-min catchment buffer overlays, school/KG point markers, and interactive click popups showing exact walking times and distances - project.qgz: updated with education catchment and POI layers
…a Ruumiamet grey WMS basemap - fix GPKG layer datasource syntax in QGIS XML (./data/derived/final-candidates.gpkg|layername=final-candidates) so QGIS loads it as a true spatial vector layer with categorized styling - add all analytical & context layers to project.qgz: * Candidate Parcels (Tier 1 Prime, Tier 2 Good, Tier 3) * Education 25-min Catchments (School & Kindergarten buffers) * Schools & Kindergartens POIs * Planned Connector Road (OVERRIDE-002) * National Highways (ETAK) - add Maa- ja Ruumiamet Mustvalge põhikaart (WMS grey basemap in EPSG:3301) and OpenStreetMap (XYZ tile layer) in Basemaps group - update pipeline.py to generate project.qgz automatically on standalone runs
… instructions
- SKILL.md:
* add hard rule forbidding coordinate hallucination or fake geodata synthesis
without explicit user consent (real source data mandatory)
* add requirement to deliver layer- and style-perfect QGIS project (project.qgz)
* add universal anti-patterns for coordinate hallucination and non-spatial table QGIS bugs
- references/project-spec.md & references/qgis.md:
* document mandatory real source data retrieval and provenance recording
* document exact GeoPackage datasource syntax (./data.gpkg|layername=name) to avoid
non-spatial table loading
* document tiled basemaps (Maa- ja Ruumiamet WMS grey basemap pohi_mvr2, OSM XYZ)
* document layer tree groups and categorized styling XML matching the web dashboard
… registration - fix projection mismatch where missing <projectCrs> caused QGIS to disable on-the-fly reprojection and plot EPSG:3301 meters onto Web Mercator EPSG:3857 - add CartoDB Positron (Light Grey Basemap), Maa- ja Ruumiamet Baaskaart (WMS in EPSG:3301), and OpenStreetMap (XYZ in EPSG:3857) to Basemaps group - compile project.qgz natively via PyQGIS with complete layer metadata and styles database - visually verified in QGIS headless renderer: all vector layers (parcels, catchments, POIs, roads) sit with zero projection distortion over Tartu on all 3 basemaps
1 participant
Implements issue #4 — make Open-GIS produce reproducible GIS projects, not one-off answers.
What's included
open-gis-project/v1schema: source provenance + timestamps, override semantics (immutable source + override layer), deterministic ordered processing steps with explicit CRS, machine-readable validation (passed/failed/warning/not_testable), semantic presentation primitives + stable roles, standard GIS UX defaults + layer hierarchy, provenance UX, QGIS.qgzoutput, and the rerun contract.project.qgzas a first-class view over derived datasets that can write edits back to the override layer.project.yaml,pipeline.py(boring/inspectable, compiles),presentation.yaml,validation.yaml.Phase 3 (validate/run/inspect CLI) is intentionally deferred — it's "potential later" tooling outside the skill repo scope; the contract for it is documented.
Acceptance check
A downstream agent given "find suitable locations near main roads around Tartu and make an interactive map" now has the scaffolding to produce an inspectable, rerunnable
project.yaml+ pipeline instead of a bespoke dashboard.