madcop v1.3.0-rc.3 — Skill crystallization (L4 cluster heads)

v1.3.0-rc.3 adds the L4 layer — skill crystallization. After enough plan-execute runs, the brain accumulates many small skill pages whose topic: values share a prefix (rate-limit-retry, rate-limit-burst, rate-limit-headers). L4 compresses them into a single named cluster head.

What changes

• New module madcop.agent.crystallize ships:
  cluster_topics() — pure: group pages by topic-prefix
  aggregate_outcome() — pure: majority-wins aggregation
  render_skill_body() — pure: frontmatter + body builder
  crystallize_skills() — pure driver: writes one head per cluster
  SkillCrystallizer — middleware: opt-in (enabled=False default), runs at HOOK_PLAN_END

• Cluster heads are type=skill so retrieval + outcome-aware ranking treat them like any other skill page.

• Cluster heads carry the aggregated outcome (success/failure/unknown) so L3 ranking benefits from cluster-level evidence.

• Cluster heads get a crystallized tag — call db.search(..., tags=["crystallized"]) to retrieve just the heads.

Why "delete + insert" on refresh

A cluster's member set may grow (a 4th reflection joins). The brain's pages_touch SQLite trigger fires on UPDATE pages when updated_at is unchanged, which on SQLite 3.40.x trips a "database disk image is malformed" error. The crystallizer works around this by db.delete(slug) then db.save(slug, ...) — a delete followed by an INSERT, which is the safe path. Idempotent for callers; one extra row in ingest_log.

Quick start

from madcop.agent.crystallize import crystallize_skills
from madcop.brain.store import PageDB

db = PageDB("~/.madcop/brain.db")
new_slugs = crystallize_skills(db, min_cluster_size=3)

Or run on the middleware chain (opt-in):

from madcop.agent import MiddlewareChain, SkillCrystallizer

chain = MiddlewareChain([
    SkillCrystallizer(db, enabled=True),   # opt-in
])

Defaults

• min_cluster_size = 3 — smallest pattern size that suggests a real recurring shape.
• prefix_split = "-" — topics are slug-shaped by convention.

Tests

27 new tests in tests/agent/test_crystallize.py. Total 928/928 passing.

Install

pip install --upgrade madcop==1.3.0rc3

Verification

• pytest: 928 passed in 4.92s
• python3 -c "from madcop import __version__" -> 1.3.0rc3
• Fresh-venv pip install --force-reinstall madcop==1.3.0rc3 -> imports L4 surface; E2E: skill-rate-limit written with crystallized tag; idempotent re-run returns [] PASS

madcop v1.3.0-rc.2 — Outcome-aware L3

v1.3.0-rc.2 closes the second half of the loop: every reflection is now stamped with outcome: success|failure at write time, and a new L3 layer re-ranks lessons accordingly before the planner sees them.

Backward-compat: rc.1 contract is bit-identical for any brain whose pages lack the outcome: field. Old skills, old tests, old chains keep working.

L1 side — outcome stamp

ReflectionMiddleware now stamps every skill page it writes with outcome: success|failure in the frontmatter and as a tag. The value is exposed on ctx.shared["reflection_outcome"] for chain observability.

L3 side — outcome-aware re-rank

New module madcop.agent.outcome ships:

• boost_outcome(lessons, outcome_weight=0.4, half_life_days=60) — pure additive rerank. score += outcome_weight * (-1|0|+1) * recency_factor. Stable for ties (preserves rc.1 FTS5+recency order).
• OutcomePrioritizer — middleware at HOOK_PLAN_START, runs after RetrievalMiddleware, rewrites ctx.shared["prior_lessons"] in place. strategy="boost" (default) or "filter" (drops outcome:failure lessons).
• format_lessons_with_outcome(lessons) — renders a ## Prior lessons (outcome-tagged) block with ✓/✗/? glyphs per line, so the planner can weight lessons visually without a second LLM call.

Quick start

from madcop.agent import (
    MiddlewareChain, QianControlMiddleware,
    RetrievalMiddleware, OutcomePrioritizer,
)

chain = MiddlewareChain([
    RetrievalMiddleware(db=db),                    # L2 — writes prior_lessons
    OutcomePrioritizer(outcome_weight=0.4),        # L3 — re-ranks them
    QianControlMiddleware(),
])

Defaults

• outcome_weight = 0.4 — small enough to preserve strong FTS5 hits, large enough to reorder ties.
• outcome_half_life_days = 60.0 — 60-day-old failure tag carries ~50% weight; 1-year-old tag carries ~3%.
• strategy = "boost" (default) — additive, never destructive.

Tests

23 new tests in tests/agent/test_outcome.py. Total 901/901 passing.

Install

pip install --upgrade madcop==1.3.0rc2

Verification

• pytest: 901 passed in 4.90s
• python3 -c "from madcop import __version__" -> 1.3.0rc2
• Fresh-venv pip install --force-reinstall madcop==1.3.0rc2 -> imports L3 surface; synthetic succ -> unk -> fail E2E order PASS

madcop v1.3.0-rc.1 — Loop engineering (L1 reflection + L2 retrieval)

v1.3.0-rc.1 closes the agent's closed-loop feedback circuit. After every plan-execute run, the agent writes 1-3 actionable lessons to the brain. Before the next run, the agent retrieves up to 3 lessons relevant to the new goal and injects them into the planner's context. Failure today becomes a retrievable, durable lesson tomorrow — no fine-tuning, no vector DB, no extra infra.

L1 — ReflectionMiddleware (HOOK_PLAN_END)

1 LLM call per plan -> 1-3 JSON reflections -> type=skill pages in the brain. Silent on LLM failure (agent never crashes on reflection). max_reflections=3 hard cap. Custom inject key.

L2 — RetrievalMiddleware (HOOK_PLAN_START)

Brain FTS5 query on the goal -> recency-weighted rerank -> top-k=3 -> ctx.shared["prior_lessons"]. Tunables: top_k, recency_weight, half_life_days, min_bm25, tags filter.

What's new

• madcop/agent/reflection.py — ReflectionMiddleware + parse_reflections + summarize_plan + DEFAULT_REFLECTION_PROMPT
• madcop/agent/retrieval.py — RetrievalMiddleware + PriorLesson + format_lessons + rerank + filter_hits
• 53 new tests (878/878 passing total)
• README "What's new" section + test badge 825 -> 878
• Reuses v1.0 MiddlewareChain + Directive + hook system — zero new concepts

Install

pip install --upgrade madcop==1.3.0rc1

Verification

pytest: 878/878 passed in 4.78s
python3 -c "from madcop import version; print(version)" -> 1.3.0rc1

v1.2.0-rc.2 — MCP client, tracing, knowledge brain + import fix

v1.2.0-rc.2

Release date: 2026-06-26
PyPI: https://pypi.org/project/madcop/1.2.0rc2/
Full diff: v1.1.0rc.1...v1.2.0rc.2

What is this?

v1.2.0 closes the local-first loop. madcop can now talk to external tool
servers (MCP), trace what it did (JSONL), and remember what it
learned (knowledge brain). Together they make a multi-session agent that
can grow without re-explaining itself.

This is rc.2, a pure fix release over rc.1 (same code + a 1-line
import fix; PyPI does not allow re-uploading the same version).

New in v1.2.0

1. MCP client (stdio transport)

Talk to external tool servers via JSON-RPC 2.0 over the server's
stdin/stdout. Surface mcp://server/... as a tool namespace, mix with
local tools.

import asyncio
from madcop.tools import MCPClient

async def main():
    client = MCPClient(command=["python3", "my_mcp_server.py"])
    await client.start()
    tools = await client.list_tools()
    result = await client.call_tool("search_docs", {"query": "madcop"})
    await client.stop()

asyncio.run(main())

Stdio only in v1.2.0; HTTP/SSE on the v1.3.0 roadmap.

2. Tracing (JSONL dump + viewer)

Every plan-execute run can write a JSONL trace. Each event
(plan_start, step_start, step_end, llm_call, tool_call,
directive, halt, error, plan_end) is one line. Tracer is
thread-safe (lock + per-line flush).

from pathlib import Path
from madcop.agent import Tracer, TraceMiddleware, MiddlewareChain, QianControlMiddleware

chain = MiddlewareChain([
    QianControlMiddleware(),
    TraceMiddleware(Tracer(Path("~/.madcop/trace.jsonl").expanduser())),
])

For a single-run summary, print_summary(read_traces(path)) gives event
counts, duration, and step order.

3. Knowledge brain (PageDB + Dream consolidation)

Long-term memory in a single SQLite file: 8 tables, FTS5 indexed, versioned,
audit-logged, full-text searchable.

from madcop.brain import PageDB, parse, scan

db = PageDB("~/.madcop/brain.db")
parsed = parse("---\ntitle: My lesson\ntype: skill\n---\n\n## Body\nX")
db.save(slug=parsed.slug, title=parsed.title, page_type=parsed.type,
        compiled_truth=parsed.compiled_truth, timeline=parsed.timeline,
        frontmatter=parsed.frontmatter)

# Search later
hits = db.search("lesson")

# Pre-screen before saving (catches API keys, JWTs, PEM blocks, etc.)
if scan(body):
    print("sensitive content — needs human review")

Prescreen (sensitive-content guard)

18 regex patterns catch the secrets we never want in the brain: AWS
keys, OpenAI / Anthropic / GitHub / PyPI / HuggingFace tokens, JWTs,
PEM private keys, database connection strings, internal IPs, Chinese
mobile numbers, and .env-style KEY=VALUE pairs. Hits route to
review_queue instead of pages.

Dream consolidation

A Dream pass periodically:

• dedups pages by content hash (older is the survivor; tags and
  timeline entries are merged in; links are repointed)
• prunes orphan links (defensive; CASCADE handles most)
• marks pages stale by last_accessed_at

It's a report, not a silent mutator. dry_run=True shows what
would have happened; real runs write one ingest_log row with
operation='consolidate' and a JSON detail blob for audit.

BrainMiddleware

BrainMiddleware plugs into the v1.0 middleware chain. The agent
opts in to memory by writing learn:-prefixed notes in a step
outcome — the middleware parses them, slugifies, and writes to the
brain.

Stats

• Tests: 825/825 pass (was 674 in v1.1.0, +151)
• New modules: 7 in madcop/brain/, 1 in madcop/agent/ (tracing),
  1 in madcop/tools/ (MCP client)
• Public surface added: madcop.brain package, Tracer /
  TraceMiddleware in madcop.agent, MCPClient in madcop.tools

What's changed since v1.2.0-rc.1

• madcop/agent/tracing.py: changed from .middleware import ... to
  absolute from madcop.agent.middleware import ... so that
  pip install --no-deps works without langgraph being installed
  (real users were unaffected). Same fix already used in
  madcop/brain/middleware.py.

Install

pip install madcop[all]==1.2.0rc2

What's not in this release

• HTTP/SSE MCP transport (v1.3.0)
• Vector search in brain (FTS5 is enough at single-user scale)
• Dream auto-merge of non-identical content (we deliberately don't
  auto-merge; dedup is safe, merge is dangerous)