Skip to content

Week 12 Report

Jump to bottom
Vishmayraj Zala edited this page Aug 22, 2026 · 3 revisions

Weekly Report - Week 12 (August 15 - August 21, 2026)

I performed the following tasks during week 12:

Workshop page (istsos4-workshop#7)

  • Added a "Key concepts" page (concepts/stac_dcat.md) to the workshop site documenting istSOS4's STAC 1.0 and DCAT-AP 3.0 catalogs, as asked at last week's sync.
  • Covers the entity mapping between SensorThings, STAC, and DCAT-AP, the NETWORK scoping mode, and how to use the catalogs both as a consumer (endpoints, examples) and as a deployer (config, mandatory fields, access control).
  • Enabled Mermaid diagram rendering (pymdownx.superfences.custom_fences), which the site config had left unset, and wired the new page into zensical.toml's nav.

Auth gate fix -- authenticated callers weren't seeing closed networks

  • Found and fixed a bug in the auth gate: with AUTHORIZATION=1, ANONYMOUS_VIEWER=0, and a network listed in CATALOG_CLOSED_NETWORKS, the network stayed hidden even from an authenticated caller with a valid token, when it should only be hidden from anonymous ones.
  • On the STAC side, this was a straightforward fix since the catalog is built from plain Python dicts: /connector/stac and friends now build the network list based on auth state, showing every network to an authenticated caller and only the open ones otherwise.
  • On the DCAT side the same trick doesn't work, since the graph isn't a dict that can be filtered per request the same way. Instead of trying to filter an RDF graph per-request, built two separate root graphs under NETWORK=1:
    • root -- Catalog + DataService + Agents + hasPart links, open networks only.
    • root_all -- the same, but hasPart/catalog also lists the closed Networks. Served only to authenticated callers, via the auth check in api.py's dcat_root / dcat_root_ttl.
  • Same fix applied consistently across both formats so STAC and DCAT behave identically under every auth flag combination, matching what test_dcat_gate_parity.py already asserts.

Validation against PySTAC and PySHACL

  • Wrote validate_stac.py, validating the generated STAC catalog with pystac, and validate_dcat.py, validating the generated DCAT-AP graphs with pyshacl against the official DCAT-AP 3.0 SHACL shapes (turtle/dcat-ap-SHACL.ttl).
  • Ran both against turtle/root.ttl, orphan.ttl, subcatalog_1.ttl, and subcatalog_2.ttl, generated under N_THINGS=5600 and N_OBSERVED_PROPERTIES=2, close to the scale of the Fraunhofer FROST dataset (5,610 Things, 22,941 Datastreams) used for validation in earlier weeks.
  • Both catalogs validate clean with no errors (result_stac.txt, result_dcat.txt).

Connector cleanup and final validation

  • Continued the final cleanup of the connector, including resolving inconsistencies in the Docker environment variables and making the connector root more readable.

  • Refined the connector root so it exposes the refresh interval, configured network IDs, and the current STAC/DCAT availability and harvest timestamps in a much cleaner form. The harvesting-related fields are omitted when both transformers are disabled.

  • Fixed the STAC /collections and /collections/{collection-id}/items endpoints for NETWORK=1, so the network-scoped API exposes the expected collection and ItemCollection structures.

  • During the final mentor review, clarified that network-enabled istSOS deployments cannot contain orphan Datastreams. Removed the previously implemented orphan Datastream handling from the DCAT representation rather than retaining a case that cannot occur under that deployment mode.

  • Separated the connector's performance logging into individual harvest, transformation, serialization, and Redis-writing stages. This also clarified that the large DCAT cache-writing time was primarily coming from RDF graph serialization to JSON-LD, rather than the Redis write itself.

  • Reworked the validation into a reproducible test suite under tests/connector/validate, so the generated outputs can be checked directly with:

    pytest -v tests/connector/validate

    The suite checks STAC conformance with pystac and DCAT-AP 3.0 conformance with pyshacl against the official shapes. The suite was run successfully for both network configurations.

Details can be found in:

  • Workshop PR: istSOS/istsos4-workshop #7
  • Auth fix: api/app/v1/connector/auth_gate.py, api.py, dcat_transformer.py
  • Validation scripts: tests/connector/validate/
  • Connector configuration and root: api/app/v1/connector/
  • STAC network endpoints: api/app/v1/connector/api.py

Clone this wiki locally