docs(benchmark): publish candid WWDC26 comparison [issue:#10]

[tcballard]

Summary

Implements Issue #10.

Adds:

• A sourced feature comparison between Sorting Hat and Apple’s WWDC26 file-sorting demonstration.
• A reproducible system/PCC prompt matrix, recorded environment, aggregate measurements, privacy boundary, limitations, and negative results.
• A candid Hacker News launch draft and a toolkit-reviewed README front door.
• Evaluator fixes for installed resource discovery and accurate Apple infrastructure diagnostics.

Scope

Included

• Apple demo versus Sorting Hat capability matrix.
• Six-class anonymized corpus methodology and exact scoring contract.
• On-device prompt/use-case measurements and explicitly approved PCC attempt.
• OCR, batching, privacy, provider, tool-evaluation, and architecture evidence.
• PCC unavailable result classified as infrastructure rather than model quality.
• Deterministic evaluator regression coverage.

Excluded

• Private corpus documents and raw model outputs.
• A claim that Sorting Hat currently beats Apple on classification quality.
• Shipping any tool-calling candidate rejected by Issue feat(analyze): evaluate bounded tool calling for filing context #9.
• Notarization or production distribution work.

Product / Architecture Decisions

• Kept filesystem mutation in validated Swift; language models only propose decisions.
• Published failed and inconclusive results instead of converting them into marketing claims.
• Added a committed system-only matrix so users can reproduce evaluation without cloud disclosure.
• Treated “PCC inference is not available in this context” as infrastructure failure.
• Preserved the full raw evaluation artifacts outside the repository.

User-Facing Contract

CLI

The evaluator now resolves checked-in matrices and prompts when run from either the repository root or the evaluation directory.

Fully local evaluation:

cd evaluation
uv run sortinghat-evaluate \
  --corpus ~/SortingHat-Evaluation/corpus/corpus.json \
  --matrix system-matrix.json \
  --output ~/SortingHat-Evaluation/results/run-001

Full matrix requires explicit --allow-pcc.

Human Output

The comparison documents exact aggregate results, environment, methodology, negative findings, and limitations. The README links directly to that evidence.

JSON Output

No schema shape changed. Infrastructure diagnostics are classified more accurately in existing error_kind fields.

Exit Codes

• 0: evaluation artifacts were produced, including rows that record model or infrastructure failures.
• 1: the evaluator itself could not complete.
• 2: reserved by the Swift live evaluator for threshold or baseline regression.

Verification

Ran

• swift test
• env PYTHONPATH=src uv run --no-sync pytest
• Full approved four-variant Apple model matrix against the private six-case corpus.
• PCC-only diagnostic rerun after exposing fm stderr.
• python3 .../readme-front-door/scripts/audit_front_door.py README.md
• git diff --check

Covered

• 32 Swift tests: batching, OCR, path/extension safety, provider policy, live evaluation, and filesystem behavior.
• 13 Python tests: scoring, artifacts, PCC consent, tool gates, resource discovery, and infrastructure classification.
• 18 successful on-device structured responses from the full run, with one content-tagging generation failure.
• Six PCC attempts rejected before inference because PCC was unavailable in the current context.
• README title, placeholders, and local asset paths.

Review Path

1. docs/wwdc26-comparison.md
2. README.md and docs/hacker-news-draft.md
3. Evaluator CLI and infrastructure classification changes
4. Evaluation matrix and tests

Notes For Reviewer

The most important review question is whether the comparison remains appropriately candid. The current evidence supports “a more complete product than the WWDC demo,” but not superior classification quality. Raw corpus inputs and outputs are intentionally excluded.