Handling KV Cache Corruption During L2-to-L1 Promotion in HiCache

[montagetao]

Background

HiCache supports hierarchical KV cache management, allowing cached data to be transferred from L2 (host memory) to L1 (GPU memory) when a cache hit occurs.

However, what happens if the KV cache data stored in L2 is already corrupted when it is promoted back to L1?

As far as I understand the current implementation, there does not appear to be an explicit data integrity validation mechanism in this path to detect such corruption before the data is reused. If corrupted KV cache data is loaded into L1 and used for inference, it could potentially lead to incorrect model outputs or other unexpected behavior.

Discussion Points

I'd like to discuss how HiCache should handle this scenario:

Data integrity validation: Should we introduce checksums or hashes to detect corruption when transferring KV cache data from L2 to L1? What would be the performance overhead?

Corruption handling: If corruption is detected, should we discard the corrupted cache entry and recompute the KV cache from the original tokens, rather than reusing invalid data?

Error propagation and recovery: How should the system handle corrupted entries during cache promotion? Should it fall back to a cache miss and continue inference, or report an error in certain cases?

Validation granularity: Should integrity checks be performed per token, per page, or per cache entry? Given HiCache's page-based management, would page-level checks provide a reasonable balance between reliability and overhead?

Performance considerations: How can we introduce integrity protection without significantly affecting cache-hit latency and overall inference throughput?

[IlyaKhalafi]

The distinction I'd make is between transfer completion and payload integrity. In the current load path, start_loading() submits the host-to-device transfer and registers completion events. The transfer engine reports completion per layer. Those events establish ordering, not that the bytes are correct. The storage hashes derived from token keys are also different from checksums of the KV payload.

For the corruption scenario you describe, I'd start with an optional page-level integrity mode and a narrowly defined failure model.

• Store a reference checksum when a host page is first populated from a trusted producer, after the write completes. Recomputing a checksum only immediately before promotion and comparing it with the destination would miss corruption that already happened in L2.
• Bind the checksum to the cache identity, allocation generation, layer and payload layout. Cover the actual K/V bytes and any required scale or auxiliary state. A checksum of an unchanged but incorrectly associated page is not sufficient.
• Keep the page pinned against mutation or reuse while it is verified and copied. If the goal includes detecting transfer corruption, verify the destination against that reference too. Host-only verification covers a narrower failure window.

The important integration point is before a consumer can use the corresponding layer. Because loading is pipelined, adding one final check after all layers finish would be too late if an earlier layer has already run. Verification would need to gate the layer-ready event, or the whole promotion would need to remain unavailable until verified. I'd measure both designs rather than assume their latency costs are equivalent.

A detected mismatch can become a cache miss if it is caught before consumption and the original request inputs are available for recomputation. Invalidate the bad page, keep only a verified contiguous prefix, and coordinate the retry across participating ranks. Reclaim transfer buffers only after their in-flight work is finished. If inference has already consumed the page, restart from a known-good state or fail the request. A device failure or an unavailable original input should not be hidden as an ordinary miss.

I haven't benchmarked this on GPUs, so I wouldn't attach an overhead estimate. I'd compare integrity off/on across page sizes, dtypes, prefix lengths and concurrency, measuring hit TTFT, p99 latency, throughput and extra memory traffic. Fault injection should cover a host bit flip after checksum creation, wrong-page association, slot reuse and corruption on only one rank. That would test both detection and recovery, rather than just whether a checksum kernel runs quickly.

[montagetao]

抱歉使用中文，
感谢您这么详细的分析——把「传输完成」和「载荷完整性」区分开来，这个框架非常准确。我也同意，只在提升时才计算校验和，会漏掉那些已经在 L2 中发生的损坏。

关于校验和的绑定与生命周期。 我赞同在 page 首次填充时就存一个参考校验和，并把它绑定到缓存身份、分配层和载荷布局上。有一个问题：当一个 page 被驱逐、之后又被重新填充（slot 重用）时，我们如何保证旧校验和与新写入是原子性地一起失效的？

关于门控 layer-ready 事件。 你提到了两种设计：用验证来门控 layer-ready 事件，或者让整个提升过程在验证完成前都不可用。第一种保留了流水线，但要求验证器按层跑在关键路径上；第二种更简单，但可能把提升过程串行化。

关于故障模型。 我强烈同意：设备故障或原始输入不可用，不应该被悄悄降级成一次缓存未命中。可能值得定义一小组明确的结果：(a) 验证通过的命中；(b) 检测到损坏 → 如果原始 token 可用，就重新计算；(c) 检测到损坏但原始输入不可用或已被消费 → 以明确的错误让请求失败。这样能让恢复路径可审计。

还有一个问题，如果kv cache 所在的L2 的page 发了UCE ，在H2D的过程中发现了这个UCE ，要如何处理？

[IlyaKhalafi]

For slot reuse, I would use a (slot, generation) handle and a state transition protected by the allocator lock. Invalidate the old descriptor and advance the generation before refilling. Publish the new checksum and VALID state only after the write completes. Readers must validate the generation while acquiring a pin, and asynchronous completion callbacks must reject a different generation. The lock makes metadata publication atomic; it does not make a whole-page write atomic. Reuse must still wait for outstanding readers and transfers.

A host-memory UCE during H2D needs a separate hardware-failure path. Do not publish layer readiness or consume the destination. Quarantine the affected host allocation and treat the destination as untrusted. A checksum cannot make poisoned memory safe to read. Linux can deliver SIGBUS for poisoned pages, so this is not guaranteed to arrive as a catchable Python transfer exception.

Recomputation is appropriate only when the process and CUDA context remain usable, other work is contained, and original inputs are available. For a fatal CUDA/context error, restart the affected worker group under supervision rather than retrying CUDA in that context. NVIDIA documents process restart for errors such as cudaErrorContained and cudaErrorIllegalAddress. The exact host/GPU error and platform determine which recovery path applies. I have not tested real UCE recovery.