[Bug]: KvEvent Metrics, usedNumBlocks, can have negative block sizes in disagg/prefill mode

[michaelfeil]

System Info

To my surprise, using tensorrt-llm 1.3.0rc4, I have seen these metrics:

prefillworker-1       | 2026-03-04T00:39:03.034016Z [warning  ] Received negative values for kv blocks: kv_active_block: -2, kv_total_blocks: 7934. Setting them to 0 in published metrics. [common.publisher] file=/workspace/trtllm/common/publisher.py line=385

stats = await self._llm_engine.get_kv_cache_events_async(timeout=5)

async for stat in stats:
            request_active_slots = stat["numActiveRequests"]
            request_total_slots = stat["maxNumActiveRequests"]
            kv_active_block = stat["kvCacheStats"]["usedNumBlocks"]
            kv_total_blocks = stat["kvCacheStats"]["maxNumBlocks"]

Who can help?

Just using a minimal example for disagg prefill, with prefill first

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Expected behavior

actual behavior

additional notes

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[pengbowang-nv]

Hi @michaelfeil , do you have a detailed setup that would reproduce this issue (including model, GPU configuration, trtllm config and command)? Thanks!

[wojciech-wais]

I've introduced the additional in the linked PR to not have negative block size.

[eopXD]

@pcastonguay have you observed such behavior when running dis-agg?

[wojciech-wais]

@eopXD I agree that this is just the actual fix.

IMO the actual fix should be in the claimBlock/releaseBlock - either:

• pass the intended cache level explicitly instead of inferring from getCacheLevel()
• or do the claim/release before the swap so getCacheLevel() returns the correct value.

I have introduced TLLM_LOG_WARNING when primaryFreeCount > totalBlocks which might help us to narrow down the path that triggers this condition.