Operations

Operations

Preflight

cp .env.example .env
docker compose up -d
curl -f http://localhost:8010/health
python3 backfill.py --stats

Verify the configured chat and embedding model IDs against the inference server's /v1/models response.

Smoke test

Always use an isolated dataset:

python3 backfill.py --limit 3 --dataset kb_smoke \
  --recall "What durable decisions appear in these sessions?"

Review:

1. var/transcripts/*.md contains the scrubbed model input.
2. var/distilled/*.md contains concise notes, not raw logs.
3. Cognee processing status reaches completed.
4. Recall returns a grounded answer.
5. Container logs show no structured-output retries or rollback.

Delete the smoke dataset through /api/v1/forget, Swagger, or the UI before the real run.

Launch

On macOS, use the launchd user service rather than an interactive shell or nohup process:

chmod +x service/*.sh
./service/install-launchd.sh

The generated LaunchAgent:

• starts on login and immediately after installation;
• starts the Cognee container and waits for /health;
• resumes persisted distillation artifacts;
• retries nonzero exits after a five-minute throttle;
• does not restart after a successful full run.

For Linux or manual execution:

python3 backfill.py --dataset knowledge_base --improve

Monitor

launchctl print gui/$(id -u)/com.thomasmaerz.cognee-backfill
tail -f var/log/backfill.out.log
tail -f var/log/backfill.err.log

python3 - <<'PY'
import json
from collections import Counter
p = json.load(open("progress.json"))
print(Counter(v["status"] for v in p.values()), len(p))
PY

For phase 3, inspect dataset status:

curl -s http://localhost:8010/api/v1/datasets
curl -s http://localhost:8010/api/v1/datasets/<dataset-id>/processing-status

Pause and resume

launchctl kill SIGTERM gui/$(id -u)/com.thomasmaerz.cognee-backfill
# Resume immediately:
launchctl kickstart -k gui/$(id -u)/com.thomasmaerz.cognee-backfill

Resume is lossless because model inputs and accepted notes exist on disk. Cognee content hashes make Add idempotent, and Cognify skips data items whose pipeline status is already complete. Back up progress.json, var/transcripts/, and var/distilled/ together.

Review or reparse skipped sessions

Skipped sessions are never deleted from the source and their IDs remain in:

• progress.json — complete state machine;
• var/skipped.json — title, directory, reason, and prompt version for review.

Re-evaluate only skipped sessions:

python3 backfill.py --dataset knowledge_base --reparse-skipped

Re-distill all retained sessions after a major prompt-policy change:

DISTILL_PROMPT_VERSION=work-history-v3 \
  python3 backfill.py --dataset knowledge_base_v3 --reparse-all

--reparse-all is refused when the target dataset already contains data. Use a new dataset as shown, or explicitly delete the old dataset first.

Finish

When every dataset item is complete:

python3 backfill.py --dataset knowledge_base --improve \
  --recall "What decisions were made about authentication?" \
           "Which recurring failures have known fixes?"

Spot-check at least ten questions against the source history. Prefer questions with known exact answers, path names, or error strings.

Backup

Connector checkpoint:

tar -czf connector-checkpoint.tgz progress.json var/transcripts/ var/distilled/

Cognee knowledge base:

docker run --rm \
  -v cognee-backfill_cognee_storage:/data \
  -v "$PWD":/backup alpine \
  tar -czf /backup/cognee-storage.tgz -C /data .

Stop writers or accept a crash-consistent rather than transaction-consistent volume archive.

Cleanup

• Keep var/transcripts/ and var/distilled/ until ingestion and recall verification pass.
• After successful backup, they may be archived; keeping them makes future audits and rebuilds independent of the Cognee volume.
• Delete test datasets explicitly. They are intentionally not garbage-collected.
• docker compose down preserves the named volume. docker compose down -v destroys the knowledge base.

To remove the service without deleting its state:

./service/uninstall-launchd.sh