ssd_streaming_backend: scan-resistant eviction and look-ahead prefetch - #40
Merged
Merged
Conversation
Two optimizations on the CXL data-supply path (the ranked bottleneck for the ternary-weight inference workload this backend feeds). Scan-resistant eviction: set_streaming()-hinted pages are use-once weight-scan pages that lingered in the CLOCK cache and evicted hot/reused pages (KV cache, shared weights), inflating backing faults. Drop a clean, unreferenced streaming page right after it is served. Measured on a synthetic scan + KV-reuse bench: page faults -6%, evictions -5%, pread64 -6% (strace-confirmed). Look-ahead prefetch: an optional modelled per-fault backing latency (backing_latency_ns, default 0) lets the supply path be studied deterministically; bench/ssd_streaming_pipeline_bench.cpp streams use-once tiles with a prefetcher staying D ahead over P channels, overlapping the backing latency with compute. The consumer's demand faults become hits, moving latency off the critical path. L=50us C=30us 4096 tiles: P=2 1.95x, P=4 4.1x supply throughput; depth saturates once D>=L/C. backing_latency_ns is a model knob, not a measured device latency. Signed-off-by: Leo Cheng <chengkelfan@qq.com>
Lfan-ke
force-pushed
the
feat/ssd-streaming-scan-resistant
branch
from
August 8, 2026 06:26
e035333 to
46caf7f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two optimizations on the SSD streaming backend's data-supply path, which is the ranked bottleneck for the ternary-weight inference workload this backend feeds.
Scan-resistant eviction: pages hinted via set_streaming() are use-once weight-scan pages. They lingered in the CLOCK page cache and evicted hot, reused pages (KV cache, shared weights), inflating backing-store faults. A clean, unreferenced streaming page is now dropped right after it is served, so the weight scan no longer displaces hot data. On a synthetic scan + KV-reuse microbenchmark: page faults -6%, evictions -5%, pread64 calls -6% (strace-confirmed), hit rate +1.6pp.
Look-ahead prefetch: an optional modelled per-fault backing latency (SsdStreamingConfig::backing_latency_ns, default 0) lets the supply path be studied deterministically, independent of the noisy host disk. bench/ssd_streaming_pipeline_bench.cpp streams use-once tiles with a look-ahead prefetcher staying D tiles ahead over P parallel channels, so the backing latency overlaps the compute of earlier tiles. The consumer's demand faults become cache hits, moving the latency off the critical path. Measured on the real backend concurrency + streaming eviction (L=50us, C=30us, 4096 tiles): P=2 gives 1.95x, P=4 gives 4.1x effective supply throughput; depth saturates once D >= L/C, as a fill-the-pipeline model predicts.
backing_latency_ns is a model parameter, not a measured device latency, so the sleep-based absolute times are not device numbers; the relative speedup, the monotonic P/D sensitivity, and the fault-to-hit mechanism are the reproducible findings. The backend still compiles standalone (stdlib + linux/io_uring.h).