Skip to content

GSoC '26 istSOS Metadata Connector

Jump to bottom
Vishmayraj Zala edited this page Aug 26, 2026 · 6 revisions

istSOS Metadata Connector for Data Spaces and STAC

GSoC 2026 Final Project Submission

Header information

Field Details
Project title istSOS Metadata Connector for Data Spaces and STAC
Contributor Vishmayraj Zala
Mentors Massimiliano Cannata, Daniele Strigaro, Claudio Primerano
Organization OSGeo / istSOS
GSoC project page summerofcode.withgoogle.com/programs/2026/projects/UmLEBaWM
Coding period May 25 - August 31, 2026
Project size Large (350 hours)

Abstract

The OGC SensorThings API exposes rich environmental and IoT sensor data, but the metadata describing that data has stayed invisible to the broader data-sharing ecosystem: researchers at other institutions, government data portals, and EU data space infrastructure had no standard way to discover what an istSOS4 deployment contains or what it measures. This project builds a Metadata Connector into istSOS4 that bridges the STA entity graph to two open catalog standards -- STAC 1.0 for geospatial discovery, and DCAT-AP 3.0 for European data space interoperability. The connector harvests directly from Postgres on a fixed schedule, transforms into both formats independently, and serves live, cacheable, access-controlled endpoints that any STAC browser or open data harvester can point at, with no changes required to an existing istSOS4 instance.

The challenge

Before this project, an istSOS4 deployment's metadata was only reachable by speaking SensorThings directly. That's fine for a client that already knows the STA data model, but it locks out the two ecosystems this project targets: STAC-native tools like STAC Browser, which expect Catalog/Collection/Item JSON, and EU open data infrastructure (data.europa.eu-style harvesters), which expect DCAT-AP RDF. There was also no precedent in the codebase for exposing a second, derived representation of the STA graph on a schedule rather than per-request, so the harvesting, caching, and multi-format-serving architecture had to be designed from scratch alongside the mapping itself.

The solution

The connector reads the full STA entity graph (Thing, Location, Datastream, ObservedProperty, Sensor, and optionally Network) directly from Postgres via a single asyncpg JOIN, on a schedule independent of any request. Two pure transformers consume that same harvested data and independently build a STAC 1.0 tree and a DCAT-AP 3.0 RDF graph, both cached in Redis and served through a read-only FastAPI layer that never touches Postgres.

A NETWORK scoping mode splits either catalog into per-Network sub-catalogs when a deployment groups sensors that way, and an authorization gate lets closed networks stay fully hidden from anonymous callers -- including from the DCAT root graph, which needed a different technique than STAC's since an RDF resource's URI is its identity and can't carry two different extents in one graph. The connector also exposes network-scoped STAC collection and ItemCollection endpoints, while the /connector root provides a compact status summary of the enabled catalogs and their latest harvest state.

Code & commits

Note on merge status: all four PRs below are open and unmerged as of this submission -- the finished work lives on the gsoc/final-cleanup-and-auth branch of my fork, which I'm leaving in place. This page and that branch are the canonical reference for the completed connector.

Pull requests (connector, all open / under review)

PR Title Depends on What it added
#195 STAC 1.0 connector -- harvesting, transformation, scheduling, caching, API layer -- Harvesting layer, stac_transformer.py, Redis caching, the initial /connector FastAPI router
#204 Network-as-subcatalog architecture, STAC compliance fixes, DCAT mapping migration #195 NETWORK=1 scoping for STAC, dummy-data/testing-infra fixes, DCAT-AP mapping doc migrated into docs/
#208 DCAT-AP 3.0 transformer -- dataset/series building, license & language resolution, dual-format caching #195, #204 dcat_transformer.py, master-switch env flags, Turtle + JSON-LD caching, SHACL-clean output
#210 Gated tier authorization, STAC auth extension, closed-network gating #204, #208 auth_gate.py, OPEN_CATALOG_METADATA / CATALOG_CLOSED_NETWORKS, stac-extensions/authentication wiring, 19 auth tests

Pre-application contributions to istSOS4

All merged, in istSOS/istSOS4, landed before the coding period as groundwork for the connector (safe helpers, self-link construction, test infrastructure it later relied on):

PR Description Area
#34 Simplified numeric type check and improved docs in get_result_type_and_column Refactor
#40 Added safe_parse_datetime() and extract_iot_id() safe helpers; integrated in datetime and association handling Utils
#69 Tests for get_result_type_and_column Testing
#79 Migrated unittest to pytest with parametrize for sta2rest Testing
#94 Unit tests for create/functions.py Testing
#95 Fixed geometry placeholder OR-to-AND condition in create_entity() Bug fix
#99 Replaced raw parser.parse() calls with safe_parse_datetime() across codebase Refactor
#100 Replaced manual @iot.id checks with extract_iot_id() across codebase Refactor
#101 Added build_self_link() utility and refactored self-link construction ahead of the connector Feature
#112 Replaced deprecated on_event startup with lifespan context manager Fix
#137 Comprehensive psycopg2 tests for result(), triggers, selfLink, and expand() Testing
#138 Fixed HistoricalLocation cascade trigger using join table and multi-row handling Bug fix
#140 Integration tests for schema versioning, history triggers, and traveltime views Testing
#141 Fixed Datastream versioning trigger to handle mixed updates correctly Bug fix
#148 Direct psycopg2 tests for istsos_auth.sql logic Testing
#149 Stabilized RLS policy generation, safe handling of custom settings, consistent policy naming Bug fix
#158 Consolidated shared database fixtures and removed duplicated setup logic Refactor
#159 Improved test onboarding docs, env setup, and contributor workflow Docs

Workshop documentation (separate repo)

PR Description
istSOS/istsos4-workshop#7 "Key concepts" page (concepts/stac_dcat.md) documenting the connector's STAC/DCAT-AP catalogs for workshop attendees, with entity-mapping and NETWORK-mode diagrams

Technical documentation

Full field-by-field references live alongside the code and are linked rather than reproduced in full here:

How to run it

The connector ships inside istSOS4's own FastAPI app and lifecycle, so there's no separate service to stand up -- it's part of the normal istSOS4 install:

git clone https://github.com/Vishmayraj/istSOS4.git
cd istSOS4
git checkout gsoc/final-cleanup-and-auth

Then follow istSOS4's own Docker Compose setup to bring up Postgres, Redis, and the API. Once running, enable the transformers in .env:

STAC_TRANSFORMER=1
DCAT_TRANSFORMER=1
 
# Mandatory once DCAT_TRANSFORMER=1 -- DCAT-AP 3.0 requires these identity
# fields and has no STA equivalent to derive them from. The scheduler skips
# writing a DCAT catalog and logs a warning every cycle until all three are set.
DCAT_CATALOG_TITLE="My istSOS4 Deployment"
DCAT_CATALOG_DESCRIPTION="Environmental sensor network for the Ticino basin"
DCAT_PUBLISHER_NAME="My Organization"

A disabled standard's routes return 404 rather than being unmounted, so the route table stays predictable, whichever flags are set. The scheduler starts harvesting on the interval set by HARVEST_INTERVAL_MINUTES (default 15); the full flag reference, including NETWORK, OPEN_CATALOG_METADATA, and CATALOG_CLOSED_NETWORKS, is in the README above.

How to test

The connector has both targeted tests and a dedicated standards-validation suite.

Full connector test suite

Run:

pytest -v tests/connector

The full connector suite currently contains 37 tests, covering:

  • authentication and authorization gates;
  • network access control and configuration;
  • STAC/DCAT gate parity;
  • DCAT root visibility;
  • error handling;
  • STAC conformance;
  • DCAT-AP 3.0 SHACL conformance.

The final cleanup state passes the complete suite with 37/37 tests passing.

Standards validation

The standards-validation tests live under:

tests/connector/validate/

and can be run independently with:

pytest -v tests/connector/validate

The suite automatically:

  • validates the generated STAC catalog with pystac;
  • validates the generated DCAT-AP 3.0 graphs with pyshacl;
  • checks the DCAT output against the official DCAT-AP 3.0 SHACL shapes.

The validation suite has been run successfully for both NETWORK=0 and NETWORK=1.

Usage examples

Point STAC Browser at the root catalog and it walks the whole tree with no further configuration:

https://<your-deployment>/v1.1/connector/stac

Fetch a Network's DCAT-AP catalog as Turtle:

curl http://localhost:8018/v4/v1.1/connector/dcat/1.ttl
@prefix dcat: <http://www.w3.org/ns/dcat#> .
@prefix dct:  <http://purl.org/dc/terms/> .
 
<.../connector/dcat/1> a dcat:Catalog ;
    dct:title "Network 1" ;
    dcat:dataset <.../connector/dcat/datasets/datastream-14> ,
                 <.../connector/dcat/series/thing-3> .
 
<.../connector/dcat/series/thing-3> a dcat:DatasetSeries ;
    dct:identifier "thing-3" ;
    dcat:seriesMember <.../connector/dcat/datasets/datastream-14> .

The same endpoint also serves JSON-LD by default (unsuffixed), for harvesters that prefer it over Turtle.

Visual proof

This project is a metadata/API service with no UI of its own, so there are no screenshots to embed. The clearest "before/after" is structural, so here's the entity mapping the connector implements, and the scheduling cycle every harvest runs through:

flowchart LR
    subgraph STA["SensorThings API (before)"]
        Root["Service root"] --> Thing["Thing"]
        Thing --> Datastream["Datastream"]
    end
    subgraph STAC["STAC 1.0 (after)"]
        Catalog["Catalog"] --> Collection["Collection"]
        Collection --> Item["Item"]
    end
    subgraph DCAT["DCAT-AP 3.0 (after)"]
        DCatalog["dcat:Catalog"] --> Series["dcat:DatasetSeries"]
        Series --> Dataset["dcat:Dataset"]
    end
    Root -.-> Catalog
    Root -.-> DCatalog
    Thing -.-> Collection
    Thing -.-> Series
    Datastream -.-> Item
    Datastream -.-> Dataset
Loading 
flowchart TD
    A["APScheduler fires\nevery HARVEST_INTERVAL_MINUTES"] --> B["Try Postgres advisory lock"]
    B -->|not acquired| C["Skip this cycle"]
    B -->|acquired| D["harvest(pool)\nsingle asyncpg JOIN"]
    D --> E["build_stac_catalog()"]
    D --> F["build_dcat_catalog()"]
    E --> G["write_stac_catalog()\nRedis: stac:catalog / stac:collection:* / stac:item:*"]
    F --> H["write_dcat_catalog()\nRedis: dcat:graph:root (+ :jsonld)"]
    G --> I["Release advisory lock"]
    H --> I
Loading

Before this project, only the left-hand STA side of the first diagram was reachable at all -- a STAC browser or DCAT harvester had nothing to point at. The full Redis key scheme, NETWORK-scoping diagram, and the closed-network auth-reveal flow are in the connector README and the Harvesting Layer Reference.

Project status & future work

What was completed

# Deliverable (from proposal) Status
1 Harvesting layer Done -- single asyncpg JOIN, both NETWORK modes, typed dataclasses, transformer contract documented
2 STAC 1.0 transformation layer Done -- full Thing/Datastream mapping, NETWORK-scoped sub-catalogs, STAC auth extension
3 DCAT-AP 3.0 transformation layer Done -- full mapping, license/language URI resolution, SHACL-clean output, root/root_all closed-network split
4 REST API Done -- /connector/stac/* and /connector/dcat/* families, .ttl + JSON-LD content variants, /connector health/status summary
5 Test suite Done
6 Deployment stack Done as part of istSOS4's existing Docker Compose -- no separate service required, just env flags
7 Mapping reference docs Done -- both mapping docs finalized against real harvested + validated data, plus the harvesting layer reference

What is left

The core implementation is complete. The following are potential future improvements rather than remaining project requirements:

  • PRs are unmerged. All four connector PRs (#195, #204, #208, #210) are open on istSOS/istSOS4 as of this submission. The gsoc/final-cleanup-and-auth branch on my fork is the complete, working reference until they're reviewed and merged.
  • STAC Browser asset-download auth limitation. STAC Browser doesn't forward the auth token on protected asset downloads (plain or CSV), confirmed via HAR capture across three instances. This is a STAC Browser-side issue, tracked upstream at radiantearth/stac-browser#356, not something fixable from the connector side. Unless we have a STAC Browser from istsos side.
  • Response size at scale. GET /connector/stac/collections on a Fraunhofer-scale dataset (5,610 Things, 22,941 Datastreams) returns roughly 11.3 MB of STAC-compliant JSON. Correct, but not workable for constrained clients; the fix is response-level pagination analogous to what STA itself already provides, and it doesn't require any change to the harvest or transform layers, but it has to be STAC-compliant and discussed as future work. Deeper, structural limitations (deliberate scope boundaries, not oversights):
  • No persistent per-cycle snapshot. The Redis cache holds only the most recent harvest cycle, so a Datastream cataloged today and later removed or changed at the source leaves no record here of what the catalog said about it yesterday. This is held deliberately rather than left unbuilt: catalog metadata like license, publisher, and access rights is a property of the cataloging, not of the underlying sensor data, and frequently wasn't tracked at the historical moment a snapshot would claim to represent -- backfilling "the catalog as of last year" with this year's configuration would misrepresent history rather than reconstruct it. istSOS4Things separately proposes a STA-level time-travel extension aimed at the same problem one layer deeper (versioned observation states), which is a substantially larger undertaking than this connector.
  • No globally unique, cross-deployment persistent identifier (FAIR sub-principle F1). Identifiers are stable for the lifetime of a given deployment's harvest cycle, but nothing mints a DOI-class, citation-grade identifier the way a data repository would.
  • Metadata doesn't outlive removed data (FAIR sub-principle A2). Because the cache only ever holds the latest cycle, once a Datastream is deleted from istSOS4 it also disappears from the catalog on the next harvest -- the same root cause as the snapshot limitation above.
  • Weak provenance (FAIR sub-principle R1.2). The catalog records what a Thing and Datastream currently are, not a documented history of how they changed.

Future Work

  • Add response-level pagination to /connector/stac/collections and /connector/stac/items, mirroring STA's own pagination model, to address the 11.3 MB response-size finding above.
  • Formalize closed-network handling (currently bespoke logic in both transformer layers) as a proper STAC Extension plus a corresponding DCAT-AP usage pattern, following the same precedent Rustad et al. used for confidentiality-marking in a military STAC deployment, rather than leaving it as connector-internal code.
  • Extend the transformer architecture to a third catalog standard. Each transformer is already a pure function from the harvested catalog to that standard's serialization, cached under its own key prefix and failing independently of the others, so adding one is additive rather than a redesign.
  • Once one or more of the connector PRs land, collections.json and any other artifacts intentionally left untracked during development should be revisited.
  • RDF named graphs (a quad store) were evaluated and rejected in favor of independent per-scope Graph objects for NETWORK scoping; if a future consumer needs true cross-scope RDF queries rather than per-scope Turtle/JSON-LD documents, that tradeoff would be worth revisiting.

Weekly reports

Links

Clone this wiki locally