Skip to content

T1-2: Implement DuckDBFileBackend #91

Description

@JesuFemi-O

Milestone: #82
Depends on: #89

Context

No structured metadata backend exists. observability.py writes to .tycoon/metadata.duckdb via raw SQL strings. The new backend must live alongside it without conflict.

Goal

Create src/tycoon/metadata_backends/__init__.py (empty) and src/tycoon/metadata_backends/duckdb_file.py.

DuckDBFileBackend is a context manager class:

class DuckDBFileBackend:
    def __init__(self, path: Path, read_only: bool = False) -> None: ...
    def __enter__(self) -> "DuckDBFileBackend": ...  # opens connection, ensures schema
    def __exit__(self, *args: object) -> None: ...   # closes connection

    def append_event(self, event: BaseEvent) -> None:
        # INSERT INTO events (event_id, event_type, source_id, runtime_id, timestamp, payload)
        # payload is event.model_dump_json()

    def query_events(self, filter: EventFilter | None = None) -> list[BaseEvent]:
        # SELECT payload FROM events WHERE ... ORDER BY timestamp ASC
        # Deserialize each row via TypeAdapter(Event).validate_json(row["payload"])

    def upsert_snapshot(self, kind: str, key: str, blob: dict) -> None:
        # INSERT OR REPLACE INTO snapshots (kind, key, blob, updated_at)

    def read_snapshot(self, kind: str, key: str) -> dict | None:
        # SELECT blob FROM snapshots WHERE kind=? AND key=?

Schema (created on __enter__ via CREATE TABLE IF NOT EXISTS):

CREATE TABLE IF NOT EXISTS events (
    event_id    TEXT PRIMARY KEY,
    event_type  TEXT NOT NULL,
    source_id   TEXT NOT NULL,
    runtime_id  TEXT NOT NULL,
    timestamp   TIMESTAMPTZ NOT NULL,
    payload     TEXT NOT NULL   -- full JSON
);

CREATE TABLE IF NOT EXISTS snapshots (
    kind        TEXT NOT NULL,
    key         TEXT NOT NULL,
    blob        TEXT NOT NULL,  -- JSON
    updated_at  TIMESTAMPTZ NOT NULL,
    PRIMARY KEY (kind, key)
);

Important: Do NOT use duckdb.connect() on the same file path that dlt uses for the raw warehouse (config.raw_db). The metadata file is always .tycoon/metadata.duckdb — a separate file from the raw warehouse. This avoids DuckDB's intra-process double-open rejection.

Acceptance

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-readyClear AC, no open design decisionsarea: metadataMetadata backend worktype: taskIndividual implementation tasks

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions