OPT v1.3.0 — Working-Set-Aware Cache Capacity

OPT v1.3.0 expands the QSOL Optimization Catalog with OPT-CACHE-001 — Working-set-aware cache capacity, a reusable pattern for preventing bounded reuse caches from turning intended reuse into repeated reconstruction, recapture, recompilation, or reload work.

The pattern was motivated by independently reported evidence from Unsloth issue #12468, where a multi-GPU CUDA graph cache regression was isolated to a fixed cache-capacity bound. OPT generalizes the lesson without promoting the donor's cache constants as universal settings.

Highlights

• Adds OPT-CACHE-001 — Working-set-aware cache capacity.
• Adds a provenance-linked source note for the Unsloth CUDA graph cache regression.
• Expands the OPT catalog to 21 optimization records, while preserving the five frozen v1 records.
• Adds cache-capacity guidance to the human and machine-oriented decision maps.
• Strengthens safeguards against cargo-culting cache sizes, worker counts, thresholds, or other source-environment constants.
• Updates catalog regression tests so future record additions do not require hard-coded total-record-count changes.

OPT-CACHE-001

The new pattern addresses a common performance failure:

A bounded reuse cache can become slower when useful states are evicted before their next reuse, forcing expensive work to be reconstructed repeatedly.

Rather than treating cache size as a universal constant, OPT-CACHE-001 requires targets to reason about:

• overlapping live reusable-state cardinality;
• reuse and stack-distance distributions;
• reconstruction or recapture cost;
• eviction and premature-eviction frequency;
• retained-state memory and resource cost;
• cache admission, retention and partitioning policy;
• end-to-end performance rather than raw cache hit rate.

The objective is minimum total workload cost inside an explicit resource envelope, not maximum cache size or maximum hit rate.

Stronger correctness boundaries

The final record incorporates several safeguards identified during review.

Cache-key equivalence

Reducing key cardinality now requires a named equivalence invariant.

Targets must directly test representative distinct source states that intentionally collide under the merged cache key rather than relying only on generic cached-versus-reconstructed parity.

Atomic cache publication

New cache entries must remain private until computation and required validation complete successfully.

Failure or cancellation must not expose partial reusable state or replace an existing authoritative entry.

Correct working-set measurement

The record no longer defines required capacity as every distinct state appearing within a time horizon.

Instead, targets should measure overlapping live retention intervals and reuse/stack-distance distributions.

This avoids oversizing cases such as:

A, A, B, B, C, C

where three distinct states appear overall but a cache capacity of one can preserve every immediate reuse.

Finite tuning budgets

Cache-policy tuning now requires a finite, predeclared evaluation budget with an explicit accounting unit such as:

• candidate evaluations;
• benchmark samples;
• wall-clock time;
• GPU-seconds; or
• another bounded resource.

Targets must also define an operational stopping rule before tuning begins.

Noise-aware performance margins

Performance promotion and rollback margins must be derived from characterized benchmark and environment variability.

A single noisy measurement is not sufficient evidence to promote or reject a cache policy.

External evidence

The new source record documents:

unslothai/unsloth issue #12468

The report isolated a tensor-split decode regression across Windows, WSL2 and bare-metal Linux.

In the reporter's exact-source WSL2 reproduction:

• the released bounded-cache configuration operated at roughly 32–33 tokens/s;
• changing only the cache-capacity bound to a deliberately oversized diagnostic value restored roughly 94–101 tokens/s.

The oversized value was explicitly used as an isolation experiment, not as a proposed production setting.

OPT therefore promotes the mechanism:

measure the reuse working set and manage cache capacity accordingly

—not:

replace one magic number with another.

Catalog integration

OPT-CACHE-001 is now integrated into:

• README.md
• README4AI.md
• CATALOG.md

The pattern composes with existing OPT records including:

• OPT-INV-001 — invariant-driven reuse;
• OPT-BUDGET-001 — performance regression budgets;
• OPT-AUTO-001 — calibrated host-aware path promotion;
• OPT-CONT-001 — partitioned coordination domains;
• OPT-REDUCE-001 — early working-set reduction.

Test maintenance

Catalog regression tests no longer assume a fixed total record count.

Successful integrity output is validated structurally while the immutable v1 boundary remains explicitly checked:

frozen_v1=5

This allows the post-v1 catalog to continue expanding without turning each new optimization record into an unrelated test-maintenance failure.

Provenance

Implemented in:

PR #4 — Add OPT-CACHE-001 working-set-aware cache capacity

#4

External evidence:

Unsloth issue #12468 — Tensor split decode regression

unslothai/unsloth#12468

Upgrade

No migration is required.

Existing OPT records and the frozen v1 formalization remain unchanged. OPT-CACHE-001 is a new post-v1 catalog record and does not extend or alter the immutable v1 Lean proof boundary.