Adding a connector

Adding a connector

The first release ships an OpenCode connector in backfill.py. The next refactor should separate source adapters from the common distill/submit engine. New adapters should already follow the contract below.

Connector contract

class Connector:
    def stats(self) -> dict: ...
    def list_sessions(self, limit=None, offset=0) -> list[SessionRef]: ...
    def build_transcript(self, session_id: str) -> Transcript: ...

SessionRef needs a stable ID, title, project/workspace, and timestamp. Transcript needs normalized chronological text plus a substantive-content count.

Required invariants

1. Read-only source access. Enforce it in the storage client when possible.
2. Stable IDs. Resume depends on them. Namespace IDs (claude:<id>) if stores can collide.
3. Chronological ordering. Preserve causality and outcomes.
4. Bounded fields. A single tool call or attachment must not consume the session budget.
5. Explicit role mapping. Normalize user, assistant, reasoning, and tool events.
6. Source metadata. Submit connector name, source ID, title, workspace, and time to Cognee.
7. No silent errors. Malformed records are counted and logged; do not classify parse failures as junk.

Suggested layout

connectors/
  base.py
  opencode.py
  claude_code.py
  cursor.py
pipeline/
  distill.py
  scrub.py
  submit.py
  state.py

The stdlib-only goal is useful but not absolute. Prefer a small, auditable dependency over a brittle custom parser.

Connector test fixture

Every connector should ship an anonymized fixture containing:

• a short successful session;
• an aborted/no-value session;
• a long session with a useful result at the end;
• tool output containing a synthetic API key to verify redaction;
• malformed records that must surface as errors;
• duplicate/resume execution proving idempotence.

Quality acceptance

A connector is ready when:

1. it never writes to the source;
2. a three-session smoke test creates useful notes;
3. stop/restart does not lose or duplicate accepted work;
4. at least ten known-answer recall questions pass manual review;
5. logs and persisted notes contain no synthetic secrets from the fixture.

Planned sources

• Claude Code JSONL under ~/.claude/projects/
• Cursor / Windsurf transcript exports
• Cline / Continue storage
• generic normalized JSONL for external ETL