Path F5 — LRU eviction + size cap + stale-tmp cleanup
Production hardening for the persistent KV cache that shipped in alpha.9. The cache directory now self-caps at 1 GB by default and evicts oldest *.bin files by mtime when over budget. Stale *.tmp files left behind by crashed prior saves are cleaned at the next save. Same throughput / TTFT / hydrate behavior as alpha.9 — purely operational safety.
Defaults work out of the box
| Var |
Default |
Effect |
JLLM_KV_CACHE_MAX_MB |
1024 |
Total *.bin budget in MB. 0 disables eviction. |
JLLM_KV_CACHE_STALE_TMP_S |
60 |
Age (s) past which *.tmp files get unlinked. 0 keeps everything. |
Verified on Jetson Orin Nano Super 8 GB
| Test |
Result |
| Defaults, 5 MB save |
Silent — no eviction triggered (1 GB ≫ 5 MB). |
MAX_MB=8, 3 × 3.7 MB saves |
conv-1 + conv-2 saved silent; conv-3 triggers evicted conv-1.bin (oldest LRU). Final dir = conv-2 + conv-3, total 7.3 MB ≤ 8 MB ✓ |
old.tmp aged 5 min, default 60 s threshold |
cleaned stale tmp .../old.tmp (age 302s) then file gone from disk ✓ |
Save latency overhead ~5–10 ms (scandir + stat + optional unlink). Within the day-to-day eMMC jitter we already see on the save path.
Throughput / TTFT vs alpha.9
|
alpha.9 |
alpha.10 |
Δ |
| Cold prefill |
38.7 tok/s |
38.7 tok/s |
unchanged |
| Decode |
10.0 tok/s |
10.0 tok/s |
unchanged |
| Warm-turn TTFT (24 / 36 hydrate) |
444 ms |
444 ms |
unchanged |
| Cache dir growth |
unbounded |
capped at 1 GB (default) |
bounded ✓ |
| Stale .tmp from crashed saves |
accumulate |
cleaned (default 60 s) |
recovered ✓ |
Path F series complete
| PR |
Phase |
What |
| #46 |
F1 |
Serialization round-trip test |
| #47 |
F2 |
--conv-id surface (CLI + HTTP) |
| #48 |
F3 |
Save on turn end (full pool, replaced by F3b) |
| #49 |
F3b |
Pack only used_tokens (34× save speedup) |
| #50 |
F4a |
Format v2: persist token IDs + off-by-one fix |
| #51 |
F4b |
Hydrate on turn start + longest-common-prefix |
| #53 |
F5 |
LRU eviction + size cap + stale-tmp cleanup |
Path F plan: #45.
Cumulative since alpha.2
|
alpha.2 |
alpha.10 |
Δ |
| Cold prefill |
8.2 tok/s |
38.7 tok/s |
+372 % |
| Decode |
7.5 tok/s |
10.0 tok/s |
+33 % |
| Cold TTFT |
2200 ms |
~860 ms |
−61 % |
| Warm-turn TTFT |
n/a |
~444 ms |
new (Path F) |
| Cache safety |
n/a |
1 GB cap + LRU + .tmp cleanup |
new (Path F5) |
Not in this release
- Multi-process flock around scandir/eviction. Single-process today.
- Cache index file (scandir-on-every-save is < 10 ms at our sizes).
- Cosmetic prefill-log fix (post-hydrate
Prefill: N tokens still counts hydrated tokens in N). Will pair with the next focused fix.
