Milestone: #82
Depends on: #89
Context
No structured metadata backend exists. observability.py writes to .tycoon/metadata.duckdb via raw SQL strings. The new backend must live alongside it without conflict.
Goal
Create src/tycoon/metadata_backends/__init__.py (empty) and src/tycoon/metadata_backends/duckdb_file.py.
DuckDBFileBackend is a context manager class:
class DuckDBFileBackend:
def __init__(self, path: Path, read_only: bool = False) -> None: ...
def __enter__(self) -> "DuckDBFileBackend": ... # opens connection, ensures schema
def __exit__(self, *args: object) -> None: ... # closes connection
def append_event(self, event: BaseEvent) -> None:
# INSERT INTO events (event_id, event_type, source_id, runtime_id, timestamp, payload)
# payload is event.model_dump_json()
def query_events(self, filter: EventFilter | None = None) -> list[BaseEvent]:
# SELECT payload FROM events WHERE ... ORDER BY timestamp ASC
# Deserialize each row via TypeAdapter(Event).validate_json(row["payload"])
def upsert_snapshot(self, kind: str, key: str, blob: dict) -> None:
# INSERT OR REPLACE INTO snapshots (kind, key, blob, updated_at)
def read_snapshot(self, kind: str, key: str) -> dict | None:
# SELECT blob FROM snapshots WHERE kind=? AND key=?
Schema (created on __enter__ via CREATE TABLE IF NOT EXISTS):
CREATE TABLE IF NOT EXISTS events (
event_id TEXT PRIMARY KEY,
event_type TEXT NOT NULL,
source_id TEXT NOT NULL,
runtime_id TEXT NOT NULL,
timestamp TIMESTAMPTZ NOT NULL,
payload TEXT NOT NULL -- full JSON
);
CREATE TABLE IF NOT EXISTS snapshots (
kind TEXT NOT NULL,
key TEXT NOT NULL,
blob TEXT NOT NULL, -- JSON
updated_at TIMESTAMPTZ NOT NULL,
PRIMARY KEY (kind, key)
);
Important: Do NOT use duckdb.connect() on the same file path that dlt uses for the raw warehouse (config.raw_db). The metadata file is always .tycoon/metadata.duckdb — a separate file from the raw warehouse. This avoids DuckDB's intra-process double-open rejection.
Acceptance
Milestone: #82
Depends on: #89
Context
No structured metadata backend exists.
observability.pywrites to.tycoon/metadata.duckdbvia raw SQL strings. The new backend must live alongside it without conflict.Goal
Create
src/tycoon/metadata_backends/__init__.py(empty) andsrc/tycoon/metadata_backends/duckdb_file.py.DuckDBFileBackendis a context manager class:Schema (created on
__enter__viaCREATE TABLE IF NOT EXISTS):Important: Do NOT use
duckdb.connect()on the same file path that dlt uses for the raw warehouse (config.raw_db). The metadata file is always.tycoon/metadata.duckdb— a separate file from the raw warehouse. This avoids DuckDB's intra-process double-open rejection.Acceptance
DuckDBFileBackendpasses.tycoon/metadata.duckdbon first__enter__