feat(v0.4-alpha): deterministic-tokenfold compressor baseline

[snchimata]

Summary

Adds the deterministic-tokenfold baseline to the v0.4-alpha harness (F-057) — the primary "baseline to beat" from model-research.md. It runs the real tokenfold Rust CLI as a whole-pipeline compressor so the learned selector (v0.4-beta) can be measured against the deterministic pipeline it must improve on.

What's in it

• New compressor-kind baseline (distinct from the unit-ranking selectors): deterministic-tokenfold shells out to tokenfold compress --target-tokens <budget>. Because it's a whole-pipeline best-effort compressor, the harness does not force critical atoms through it or enforce the exact ceiling — its critical-atom survival and achieved ratio are measured, not asserted (honest: a selector gets guarantees the CLI can't).
• Binary discovery + graceful skip: TOKENFOLD_BIN → local target/{release,debug} → PATH; when absent the baseline is cleanly skipped (n/a) and the report/gate say so, so the harness still runs in a build-less CI. Verified both paths.
• Whitespace-insensitive task containment so a lossless reformat (e.g. JSON minify collapsing "max_results": 25 → "max_results":25) still counts as surviving. Selector scores are unchanged.
• .gitignore now covers Python __pycache__.

The finding

baseline                 ratio   task   crit  achieved  over
keep_all                  0.25    1.0    1.0       1.0     0
bm25                      0.25 0.3333    1.0     0.246     1
deterministic-tokenfold   0.25    1.0    1.0    0.8727     3

deterministic-tokenfold is lossless/evidence-safe (task + critical survival ≈ 1.0) but only reaches ~0.87 and cannot hit the 0.5/0.25 budgets on these low-repetition inputs (over_budget), while the aggressive selectors hit the budget but drop task success. That gap — safe-but-limited-ratio vs. aggressive-but-lossy — is precisely what a learned selector must close.

Testing

python eval/run_baselines.py --gate → pass (30 selector invariant checks; tokenfold_available reported). Verified the graceful-skip path (binary absent → n/a, gate still passes).

Still deferred

RTK / RTK+tokenfold (external tool), LLMLingua-style selector, Headroom Kompress-v2 achieved-token sweep; Tier-B public-repo corpora + project-disjoint splits; structural segmentation; real paired build/test execution.