Week 04 Report

Weekly Report - Week 4 (June 15 - June 21, 2026)

I performed the following tasks during week 4:

Architecture migration: standalone to integrated

Migrated the connector from the standalone microservice shape prototyped during bonding into the confirmed integrated architecture inside api/app/v1/connector/ within the main istSOS4 repo. Scaffolded all connector package files as per the structure agreed with mentors.
Added a temporary FROST migration utility to seed the local development environment with Fraunhofer's air quality dataset, allowing STAC development to proceed immediately without waiting for a populated local dataset. This commit will be reverted before the branch is merged.

Harvesting layer implementation (harvester.py)

Implemented the harvesting layer using a direct asyncpg JOIN query across Thing, Location, Datastream, ObservedProperty, and Sensor tables. The query runs in a single database round trip -- no application-level pagination or sequential HTTP fetches. On the Fraunhofer dataset (5,610 Things, 22,941 Datastreams), the query executes in 383.297 ms, confirming the architectural trade-off from the weeks 1-2 benchmarks was correct.
The LISTEN/NOTIFY cache invalidation approach, proposed in week 3, was reviewed with mentors and rejected. Triggers on the Postgres side introduce coupling between the SensorThings schema and the connector lifecycle; the connector is a pure reader and interval-based refresh is simpler, auditable, and sufficient. HARVEST_INTERVAL_MINUTES in .env is the single control point.
asyncpg does not register a JSON/JSONB codec on the shared istSOS4 pool, and the existing STA read paths rely on that behaviour -- they stream pre-serialised JSON straight through without an intermediate json.loads()/json.dumps() round trip. Registering a global codec would silently change behaviour for unrelated code. Instead, _coerce_json() decodes JSON columns locally, only for rows the harvester consumes, handling three cases: None, an already-decoded dict/list, and a raw JSON string. Malformed blobs log a warning and return None rather than failing the whole cycle.

Scheduling layer implementation (scheduler.py)

Implemented APScheduler integration -- as advised by Maxi in our last meeting -- wired into the istSOS4 application lifespan in main.py. The scheduler starts on application startup and fires _run_cycle at the interval configured in .env.
Implemented a Postgres advisory lock -- as advised by Maxi in our last meeting -- acquired before harvest begins and held for the entire cycle duration: harvest, both transforms, and both cache writes. This prevents duplicate execution across multiple Uvicorn workers without a distributed lock service. The advisory lock connection is kept separate from the harvest connection so the harvest query does not block on lock management.
STAC and DCAT writes inside _run_cycle are independent. A DCAT failure does not roll back the STAC write. The DCAT path is guarded behind a None check on build_dcat_catalog and logs a warning each cycle until dcat_transformer.py is implemented, leaving STAC serving unaffected in the interim.

Details can be found in:

What do I plan to do next week?

Implement stac_transformer.py and cache.py.
Implement the connector API router and wire it into the main FastAPI application.
Have a running end-to-end STAC pipeline serving responses by end of week.

Am I blocked on anything?

No blockages.