Conduit

The open-core data freshness engine for RAG. Point Conduit at your sources and your vector store. It ingests, parses, chunks, embeds, and keeps everything in sync, incrementally, with full pipeline observability.

Conduit is the extract-and-load layer for retrieval. It does not own your vector store. It does not own your retrieval logic. It owns the plumbing every team rebuilds from scratch: connectors, parsing, chunking, embedding, incremental sync, and freshness monitoring.

Why Conduit

• Incremental by design. Run sync twice and the second run skips everything. Edit one document and only it re-embeds. Delete one and its vectors disappear on the next run.
• Structure-aware. Markdown splits by heading, PDFs by page, code at top-level boundaries. Every chunk carries its heading path, page number, or language as queryable metadata.
• Laptop to fleet, zero code changes. One binary syncs a folder on your machine. The same binary scales to a worker fleet with per-source Postgres leases.
• Resumable, observable. A failed document is recorded and retried next run while the rest of the run completes. status shows every source, every run, every failure.

Quickstart

You need Docker and git. Two minutes to your first synced knowledge base:

git clone https://github.com/dynaum/conduit
cd conduit
docker compose up -d postgres
docker compose run --rm conduit init   --config examples/local-folder.json
docker compose run --rm conduit sync   --config examples/local-folder.json
docker compose run --rm conduit status --config examples/local-folder.json

sync prints a per-source summary:

{"runId":"1","status":"succeeded","docs_seen":2,"docs_processed":2,"docs_skipped":0,"docs_deleted":0,"docs_failed":0}

Run sync again and every document is skipped, nothing re-embeds. Edit a file in examples/knowledge/ and run once more: only the edited file is re-processed.

Port 5432 taken? Set POSTGRES_PORT in a .env file to a free port. Compose reads it automatically.

Query your vectors

Your vectors land in the target.embeddings table, ready for SQL. See what was indexed:

select document_external_id, left(text, 60) as preview, model
from target.embeddings
order by id
limit 10;

Run a similarity search by ordering on vector distance. Embed your query text with the same model, then:

-- $1 is your query embedding, same model and dimension as the table
select document_external_id, text
from target.embeddings
order by embedding <=> $1
limit 5;

Filter by document structure with the metadata column:

-- only chunks from the install guide, or only TypeScript code
where metadata->>'headingPath' like 'Guide%'
where metadata->>'language' = 'typescript'

The table is yours. Conduit writes to it and keeps it current. Your application reads from it however it likes.

Configuration

A config file declares the embedder, the chunking, the target table, and one or more sources. The ready-to-run examples:

File	What it indexes
examples/local-folder.json	A local folder of files
examples/website.json	A list of web URLs
examples/openai.json	A local folder, embedded with OpenAI
examples/multi-source.json	A folder and a website together
examples/postgres.json	A Postgres table, incremental by updated_at
examples/s3.json	An S3 bucket/prefix, incremental by ETag
examples/github.json	A GitHub repo, incremental by blob SHA

A config looks like this:

{
  "target": { "type": "pgvector", "schema": "target", "table": "embeddings" },
  "embedder": { "type": "local", "dimension": 256 },
  "chunking": { "size": 512, "overlap": 50 },
  "sources": [
    { "id": "handbook", "type": "file", "path": "./docs" },
    { "id": "marketing", "type": "url", "urls": ["https://example.com/"] }
  ]
}

• target — the pgvector schema and table to write into.
• embedder — local (deterministic, no API key, great for trying it out) or openai (set OPENAI_API_KEY in .env, model is text-embedding-3-small or text-embedding-3-large).
• chunking — size and overlap in tokens.
• sources — each has a unique id and a type. The built-in types:

Type	Indexes	Incremental by	Setup
file	a local file or directory	content hash	—
url	a list of urls	content hash	—
postgres	rows of a table	updated_at watermark	docs/connectors/postgres.md
s3	objects in a bucket/prefix	object ETag	docs/connectors/s3.md
github	files in a repo	blob SHA	docs/connectors/github.md

Content is parsed by type before chunking: markdown splits by heading, PDFs by page, code at top-level boundaries, everything else passes through whole. Chunks carry structural metadata (heading path, page number, language) into a metadata column you filter at query time. Details: docs/CONNECTORS.md.

To use OpenAI embeddings instead of the local embedder:

echo "OPENAI_API_KEY=sk-..." >> .env
docker compose run --rm conduit init --config examples/openai.json
docker compose run --rm conduit sync --config examples/openai.json

To point Conduit at your own content, mount your config and data over the image defaults:

# in docker-compose.yml, under the conduit service
volumes:
  - ./my-config.json:/app/conduit.config.json:ro
  - ./my-docs:/app/data:ro

To run the image against your own database:

docker build -t conduit .
docker run --rm \
  -e DATABASE_URL=postgres://user:pass@your-db:5432/conduit \
  -v "$PWD/conduit.config.json:/app/conduit.config.json:ro" \
  conduit sync

Keep it fresh

conduit sync is one-shot. To keep sources fresh on a schedule, give them a cadence and run the worker:

{ "id": "handbook", "type": "file", "path": "./docs", "syncIntervalSeconds": 300 }

docker compose up -d postgres
docker compose up -d conduit-worker                 # one worker
docker compose up -d --scale conduit-worker=4       # scale out

Each worker claims due sources with a Postgres per-source lease, so no source is processed by two workers at once, and a crashed worker's sources move to a healthy one. Force an immediate sync with conduit trigger --source <id>. See docs/USE-CASES.md for the local, team, and company scenarios.

Commands

Command	What it does
conduit init	Creates the schemas and sizes the embedding column to the embedder's dimension. Idempotent.
conduit sync	Runs the pipeline for every source. Add --source <id> to run one.
conduit status	Shows the last run per source, with counts and any failed documents.
conduit worker	Long-running loop: claims due sources, syncs them, repeats. Scale by running more.
conduit trigger	Marks a source due now, so a running worker picks it up immediately.
conduit search	Embeds a query and returns ranked hits from the target table. Add --json for agents, --limit n, --source id.

All take --config <path> (default conduit.config.json).

Use it with your AI tools

Conduit gives AI coding agents a grounded knowledge base. conduit search "<question>" --json returns ranked hits an agent can cite, and agents re-sync when content changes.

In Claude Code, install the plugin straight from this repo:

/plugin marketplace add dynaum/conduit
/plugin install conduit@conduit

Copy-paste setup for Codex and GitHub Copilot, and the no-plugin options, are in docs/AI-AGENTS.md.

Good to know

• One dimension per target table. init sizes the embedding column to the embedder's dimension. Every source writing to a table must use the same dimension; init refuses a mismatch with a clear error.
• Runs are resumable. A document failure (an unreachable URL, an embedding error) is recorded and retried on the next run. The rest of the run still completes.
• Deletes are detected by absence. Each run lists the full source; a document missing from the listing is treated as deleted and its vectors are removed.

Learn more

• docs/VISION.md — the full idea, positioning, and why now
• docs/USE-CASES.md — local, team, and company scenarios
• docs/CONNECTORS.md — the connector contract and roadmap

Contributing

Want to run from source, hack on the engine, or write a connector? Start with docs/DEVELOPMENT.md.