fix: enforce lookup max_tokens budgets

[laynepenney]

Summary

• Enforce the approximate max_tokens budget against the formatter's actual selected_blocks accumulator.
• Include max_tokens in the query-cache key. This is an incomplete-key defect: a cached payload produced for one budget is not valid for another budget.
• Add separate regression witnesses for the token ladder, cache miss/hit identity, and the unchanged max_chunks control.

Evidence

On a fixed ten-chunk fixture:

Parameter	Values	Payload bytes
max_tokens	10 / 500 / 2400 / 100000	500 / 1937 / 4811 / 4811
max_chunks control	1 / 3 / 5 / 10	979 / 2895 / 4811 / 4811

The token ladder is monotone non-decreasing, the 10-token row is strictly smaller than the 2400-token row, and each result is bounded by the requested estimate plus the documented one-first-block and header tolerance. Distinct max_tokens calls create distinct cache entries; repeating the same value keeps the cache size unchanged.

Three isolated mutations each killed only its matching witness:

• Replacing selected_blocks with the stale len(lines) > 1 guard killed only the token ladder.
• Removing max_tokens from the cache key killed only the cache witness.
• Removing max_chunks from the cache key killed only the max_chunks control.

Validation:

• Core plus hybrid integration: 165 passed, 1 xfailed.
• Full suite at this head: 23 failed, 3107 passed, 25 skipped, 5 xfailed.
• The same three failing files on exact base af83265e reproduced the same 23 failures: two missing-build-tool failures, twenty code-index parser/plugin environment failures, and one model-router plugin-state failure.
• Ruff was unavailable and is not claimed.

Premium boundary: core OSS retrieval budget and cache correctness only. No identity, organization, entitlement, or premium behavior.

Ref #993 - closes at promotion.