docs+bench: ship 5-issue user-facing surface bundle (#181, #197, #198, #199, #200)

[dgenio]

Summary

Ship the 5-issue "external-facing surface polish" bundle plus tokenizer
improvements and expanded evaluation infrastructure. The bundle addresses
discoverability, documentation, benchmarking, and reference-architecture gaps
identified during triage.

Fixes #181, #197, #198, #199, #200
Related: #208, #209, #213, #214, #215

Motivation

contextweaver's core pipeline was maturing but its external surface lagged:
no benchmark scorecard, no decision-tree for new users, no reference
architecture beyond the quickstart, no structured citation, and scattered
README metadata. Meanwhile the evaluation harness needed a stress scenario
that exercises budget-driven drops and dedup, a naïve-baseline comparison,
a gold-set expansion from 50→200 items, and a weight-sweep tool for
ScoringConfig tuning. The namespace-aware tokenizer rewrite (#213) also
landed in this bundle because it directly improved routing recall measured
by the expanded benchmark.

Changes

User-facing documentation & discoverability (#199, #200)

• docs/which_pattern.md — decision-tree landing page ("Which pattern
  should I use?"), linked from README and architecture/interop docs
• README.md — CI/PyPI/license/docs/Discussions badges; "context
  engineering" phrasing; CITATION.cff; pyproject classifiers and URLs

Reference architecture (#198)

• examples/architectures/slack_ops_bot/ — 48-tool Slack ops bot
  demonstrating bounded-choice routing, firewall on large log dumps,
  and persistent facts across 6 turns. Includes catalog, scripted main,
  README, and OUTPUT.md
• docs/architectures/ — index + Slack ops bot write-up
• make architectures umbrella target wired into make example

Benchmark scorecard & evaluation expansion (#181, #197, #208, #209, #213, #214, #215)

• benchmarks/scenarios/stress_conversation.jsonl — 30-turn SEV2
  incident transcript forcing drops, dedup, and firewall ([eval] Expand long_conversation scenario to stress-test budget-driven selection #181)
• scripts/render_scorecard.py — deterministic stdlib-only renderer;
  make scorecard / make scorecard-check targets; --check mode now
  reports line-diff count on mismatch
• scripts/benchmark_delta.py — head-vs-base delta renderer for sticky
  PR comments; robust JSON error handling for missing/corrupt files
• scripts/sweep_scoring.py — weight-sweep grid search (243 configs)
  for ScoringConfig tuning; default detection uses ScoringConfig() instance
  rather than hardcoded floats
• scripts/baseline_naive.py — naïve-concat baseline comparison
• benchmarks/routing_gold.json — expanded from 50→200 gold-set items
• .github/workflows/ci.yml — scorecard-check gating step;
  benchmark-delta sticky comment
• .github/workflows/scorecard-weekly.yml — weekly scorecard regen cron
• .github/prompts/add-eval.prompt.md — eval-expansion prompt for agents

Tokenizer rewrite & routing cleanup (#213)

• src/contextweaver/_utils.py — namespace-aware compound tokenizer:
  splits slack:list_channels into meaningful tokens via configurable
  separators, improving routing recall on namespaced tool catalogs
• src/contextweaver/routing/router.py — removed legacy
  replace(":"/"_"/"/" ) workaround now that the tokenizer handles
  compound names natively
• tests/test_utils.py — 15 new tests covering the tokenizer rewrite

Agent & CI docs

• AGENTS.md — updated command table for new make targets
• docs/agent-context/workflows.md — added eval-expansion workflow
• docs/benchmarks.md — methodology page for the scorecard
• llms-full.txt — regenerated

Scope

In scope: documentation, benchmarks, evaluation tooling, one reference
architecture, tokenizer improvement that directly feeds benchmark results.

Out of scope (deferred):

• Social-preview PNG (requires GitHub UI upload)
• Code-review bot and voice-agent reference architectures
• Per-backend benchmark matrix in CI (tracked as follow-up)
• ANN/graph retriever integration

Testing & verification

• ruff format --check + ruff check — clean
• mypy src/ — 54 files, no issues
• pytest — 768 passed, 8 skipped (+20 new: 12 scorecard, 8 Slack arch)
• make example — all scripts including Slack ops bot
• make demo — clean
• make scorecard-check — deterministic (no drift)
• make llms-check — up to date

Risks & rollback

Low risk. Changes are additive: new docs, new scripts, new benchmark
scenarios, new tests. The only src/ changes are the tokenizer
(_utils.py) and router cleanup (router.py), both covered by existing

• new tests. Rollback: revert the merge commit.

Checklist

• Tests added or updated for every new/changed public function
• make ci passes locally (fmt + lint + type + test + example + demo)
• CHANGELOG.md updated under ## [Unreleased]
• Docstrings added for all new public APIs (Google-style)
• Every modified module stays ≤ 300 lines (or a decomposition issue is linked above)
• Related issue linked in the summary above

[Copilot]

Pull request overview

Lands a five-issue "external-facing surface polish" bundle: discoverability metadata, a decision-tree landing page, a Slack-ops-bot reference architecture, a reproducible benchmark scorecard, and a stress-test scenario that exercises budget-driven drops and dedup. No changes to src/contextweaver/ pipeline code beyond a tokeniser refactor referenced by the gold-set expansion.

Changes:

• Discoverability: README badges + "context engineering" phrasing, CITATION.cff, expanded pyproject.toml URLs/classifiers/keywords.
• Docs: new docs/which_pattern.md decision tree, docs/benchmarks.md, docs/architectures/ pages; cross-linked from architecture/interop/cookbook and mkdocs nav.
• Tooling/benchmarks: scripts/render_scorecard.py, scripts/benchmark_delta.py, scripts/baseline_naive.py, scripts/sweep_scoring.py; new make targets (scorecard, scorecard-check, benchmark-matrix, sweep-scoring, architectures); CI gating scorecard-check step and sticky benchmark-delta comment job; weekly scorecard cron; new stress_conversation.jsonl scenario; namespace-aware tokenizer (#213) with router cleanup; expanded 200-query gold set; Slack ops bot architecture under examples/architectures/.

Reviewed changes

Copilot reviewed 45 out of 45 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
README.md, llms.txt, llms-full.txt	Badges, "context engineering" phrasing, scorecard link, decision-tree link, examples table addition.
pyproject.toml	New URLs, classifiers, keywords for PyPI discoverability.
CITATION.cff	New CFF v1.2 file for "Cite this repository" button.
mkdocs.yml	Adds Which-pattern, Architectures, Benchmarks nav entries.
Makefile, AGENTS.md, docs/agent-context/workflows.md	New make targets (scorecard*, benchmark-matrix, sweep-scoring, architectures), scripts/ added to lint/fmt scope.
docs/which_pattern.md, docs/benchmarks.md, docs/architectures/{index,slack_ops_bot}.md	New documentation pages.
docs/architecture.md, docs/interop.md, docs/cookbook.md	Cross-links to the new decision-tree and architectures pages.
examples/architectures/slack_ops_bot/{main.py,catalog.yaml,README.md,OUTPUT.md,init.py}, examples/architectures/init.py	New Slack ops bot reference architecture.
scripts/render_scorecard.py, scripts/benchmark_delta.py, scripts/baseline_naive.py, scripts/sweep_scoring.py	New stdlib-only renderer, PR-delta generator, naive baseline, and weight-sweep tool.
scripts/gen_llms.py, scripts/weaver_spec_conformance.py	Formatting/cleanup.
benchmarks/benchmark.py	--matrix, --backends, --sizes, --no-naive-delta flags; per-backend/per-namespace cells; v1.1 schema bump.
benchmarks/scorecard.md, benchmarks/sweep_scoring.md, benchmarks/results/latest.json, benchmarks/README.md, benchmarks/routing_gold.json	Generated/committed scorecard outputs and expanded 200-query gold set.
src/contextweaver/_utils.py, src/contextweaver/routing/router.py, tests/test_utils.py	Namespace-aware tokenizer with retained underscore behavior and tests.
tests/test_render_scorecard.py, tests/test_architectures_slack.py	New test files for the renderer and the Slack ops bot.
.github/workflows/{ci.yml,scorecard-weekly.yml}, .github/pull_request_template.md, .github/prompts/add-eval.prompt.md	CI gating scorecard-check, sticky benchmark-delta comment, weekly regen cron, eval prompt.
CHANGELOG.md	Documents the bundled changes.

[github-actions]

Benchmark delta (vs main)

Soft regression feedback only — this comment never blocks the PR.
Latency budget: ⚠️ when head > base × 1.3. Accuracy budget: ⚠️ when head < base - 1pp.

Routing summary (single backend × catalog sizes)

size	recall@k (head Δ vs base)	MRR (head Δ vs base)	p99 (ms)
50	✅ 0.5649 (+0.0000)	✅ 0.4978 (+0.0000)	✅ 0.498 (base 0.463)
83	✅ 0.3825 (+0.0000)	✅ 0.3242 (+0.0000)	✅ 0.741 (base 0.876)
1000	✅ 0.1475 (+0.0000)	✅ 0.1456 (+0.0000)	✅ 40.009 (base 31.897)

Per-backend × per-size matrix

backend	size	recall@k (Δ)	MRR (Δ)	p99 (ms)
bm25	100	✅ 0.3825 (+0.0000)	✅ 0.3399 (+0.0000)	✅ 6.290 (base 5.642)
bm25	500	✅ 0.2250 (+0.0000)	✅ 0.2165 (+0.0000)	✅ 29.525 (base 27.538)
bm25	1000	✅ 0.1575 (+0.0000)	✅ 0.1525 (+0.0000)	✅ 88.189 (base 78.368)
fuzzy	100	✅ 0.0000 (+0.0000)	✅ 0.0000 (+0.0000)	✅ 0.000 (base 0.000)
fuzzy	500	✅ 0.0000 (+0.0000)	✅ 0.0000 (+0.0000)	✅ 0.000 (base 0.000)
fuzzy	1000	✅ 0.0000 (+0.0000)	✅ 0.0000 (+0.0000)	✅ 0.000 (base 0.000)
tfidf	100	✅ 0.3825 (+0.0000)	✅ 0.3220 (+0.0000)	✅ 1.035 (base 0.872)
tfidf	500	✅ 0.2325 (+0.0000)	✅ 0.2314 (+0.0000)	✅ 9.078 (base 8.660)
tfidf	1000	✅ 0.1475 (+0.0000)	✅ 0.1456 (+0.0000)	✅ 36.170 (base 30.071)

Context pipeline (per scenario)

scenario	tokens	dropped	dedup
large_catalog	1514 (base 1514, Δ+0)	0 (base 0, Δ+0)	0 (base 0, Δ+0)
long_conversation	2548 (base 2548, Δ+0)	0 (base 0, Δ+0)	0 (base 0, Δ+0)
short_conversation	496 (base 496, Δ+0)	0 (base 0, Δ+0)	0 (base 0, Δ+0)
stress_conversation	6651 (base 6651, Δ+0)	7 (base 7, Δ+0)	4 (base 4, Δ+0)

---

Numbers come from make benchmark / make benchmark-matrix.
Latency is hardware-dependent — treat the markers as a rough guide.
See benchmarks/scorecard.md for the full picture.