claw-reflect

claw-reflect is the autonomous memory distillation engine in the ClawDB ecosystem. It ingests raw memories generated by agents, runs multi-stage reflection pipelines, updates long-term profiles, and continuously decays stale knowledge so retrieval quality stays high over time.

Architecture

Raw memories
	-> [Scorer -> Duplicate Collapser -> Contradiction Detector -> Summariser -> Preference Extractor -> Promoter]
	-> AgentProfile + Scored Memories
	<- Decay Engine

Pipeline Execution Order

1. Composite scoring computes importance, recency, and confidence.
2. Duplicate collapse removes semantically redundant records.
3. Contradiction detector flags conflicts for resolution.
4. Summariser compresses session/message/context chunks.
5. Preference extractor writes durable preferences.
6. Promoter upgrades high-value memories.
7. Decay engine lowers stale memory scores outside reflection runs.

LLM Provider Configuration

Supported providers:

1. Anthropic
2. OpenAI-compatible APIs
3. Ollama (local)

Set via environment:

1. REFLECT_LLM_PROVIDER=anthropic|openai|ollama
2. REFLECT_LLM_MODEL=<model-name>
3. REFLECT_LLM_API_KEY=<key> (required except Ollama can use placeholder)
4. REFLECT_LLM_BASE_URL=<optional endpoint override>

Quick Start

1. Start stack:

docker-compose up --build

2. Ingest memories:

/Users/ginmax/claw-reflect/.venv/bin/python scripts/create_api_key.py \
	--workspace-id 00000000-0000-0000-0000-000000000001 \
	--label local-dev

curl -X POST http://localhost:8090/api/v1/reflect/memories \
	-H 'X-Claw-Api-Key: <printed-key>' \
	-H 'Content-Type: application/json' \
	-d '{
		"agent_id": "agent_1",
		"batch_id": "batch_1",
		"memories": [{
			"id": "MEM0000000000000000000001",
			"agent_id": "agent_1",
			"content": "User prefers concise bullet points.",
			"memory_type": "message",
			"metadata": {"session_id": "s1"},
			"tags": ["style"],
			"created_at": "2026-05-02T00:00:00Z",
			"updated_at": "2026-05-02T00:00:00Z"
		}]
	}'

3. Trigger reflection:

curl -X POST http://localhost:8090/api/v1/reflect/trigger \
	-H 'X-Claw-Api-Key: <printed-key>' \
	-H 'Content-Type: application/json' \
	-d '{"agent_id":"agent_1","job_type":"full","options":{}}'

All non-health API routes under /api/v1 require X-Claw-Api-Key. Only GET /api/v1/health, GET /api/v1/ready, and GET /api/v1/metrics are intentionally unauthenticated.

API Reference Summary

Health:

1. GET /api/v1/health
2. GET /api/v1/ready
3. GET /api/v1/metrics

Reflection:

1. POST /api/v1/reflect/trigger
2. POST /api/v1/reflect/trigger/dry-run
3. POST /api/v1/reflect/memories
4. POST /api/v1/reflect/score/{agent_id}
5. POST /api/v1/reflect/decay/{agent_id}
6. GET /api/v1/reflect/preview/{agent_id}

Jobs:

1. GET /api/v1/jobs/{job_id}
2. GET /api/v1/jobs
3. DELETE /api/v1/jobs/{job_id}
4. GET /api/v1/jobs/scheduled

Profiles:

1. GET /api/v1/profiles/{agent_id}
2. GET /api/v1/profiles/{agent_id}/preferences
3. PUT /api/v1/profiles/{agent_id}/preferences/{pref_id}
4. GET /api/v1/profiles/{agent_id}/contradictions
5. POST /api/v1/profiles/{agent_id}/contradictions/{contradiction_id}/resolve
6. GET /api/v1/profiles/{agent_id}/memories

Celery Worker Scaling Guide

1. Increase worker concurrency: set --concurrency on reflect-worker.
2. Horizontal scale: run multiple worker replicas behind same Redis broker.
3. Keep task_acks_late=true and worker_prefetch_multiplier=1 to reduce task loss and avoid hot-worker starvation.
4. Use queue routing by pipeline stage if throughput grows.

Decay Policy Guide

Exponential:

$$ score_{new} = score_{current} \cdot e^{-\lambda \cdot age_{days}},\ \lambda = \frac{\ln 2}{half_life_days} $$

Linear:

$$ score_{new} = \max(0, score_{current} - rate_{day} \cdot age_{days}) $$

Step:

Apply first matching threshold from configured steps:

1. (7, 0.9)
2. (30, 0.6)
3. (90, 0.3)
4. (365, 0.05)

Environment Variables

Variable	Description	Example
REFLECT_DATABASE_URL	SQLAlchemy async DB URL	postgresql+asyncpg://<user>:<password>@postgres:5432/<db>
REFLECT_REDIS_URL	Redis broker/backend URL	redis://redis:6379/0
REFLECT_CELERY_CONCURRENCY	Celery worker concurrency	4
REFLECT_LLM_PROVIDER	LLM provider name	anthropic
REFLECT_LLM_MODEL	Provider model identifier	claude-sonnet-4-20250514
REFLECT_LLM_API_KEY	LLM API key secret	sk-...
REFLECT_LLM_BASE_URL	Custom provider endpoint	http://localhost:11434
REFLECT_LLM_MAX_TOKENS	Max output tokens per call	2048
REFLECT_LLM_TIMEOUT_SECS	Request timeout	30
REFLECT_LLM_MAX_RETRIES	LLM retry attempts	3
REFLECT_REFLECTION_BATCH_SIZE	Pipeline batch size	50
REFLECT_REFLECTION_INTERVAL_MINUTES	Reflection schedule interval	30
REFLECT_DECAY_INTERVAL_HOURS	Decay schedule interval	6
REFLECT_SCORE_REFRESH_INTERVAL_HOURS	Score refresh interval	12
REFLECT_IMPORTANCE_WEIGHT	Composite importance weight	0.4
REFLECT_RECENCY_WEIGHT	Composite recency weight	0.35
REFLECT_CONFIDENCE_WEIGHT	Composite confidence weight	0.25
REFLECT_DEFAULT_DECAY_POLICY	Default decay policy	exponential
REFLECT_DECAY_HALF_LIFE_DAYS	Exponential half life	30
REFLECT_ARCHIVE_THRESHOLD_SCORE	Archive threshold	0.05
REFLECT_MIN_SCORE_TO_KEEP	Hard cleanup cutoff	0.02
REFLECT_DUPLICATE_SIMILARITY_THRESHOLD	Duplicate decision threshold	0.92
REFLECT_API_HOST	API bind host	0.0.0.0
REFLECT_API_PORT	API bind port	8090
REFLECT_DEBUG	Enables docs and debug settings	false
REFLECT_LOG_LEVEL	Log level	INFO
REFLECT_LOG_FORMAT	json or console	json