[https://nvbugs/6701493][fix] Limit Gemma3 FP8 accuracy test KV cache memory fraction

[erictsai-nv]

Dev Engineer Review

• Updated the Gemma3 27B FP8 accuracy test to set free_gpu_memory_fraction=0.5 in KvCacheConfig.
• Added comments that document the memory cap and activation headroom purpose.
• The setting applies only to this test.
• Product defaults and allocator behavior remain unchanged.
• Removed the duplicate dtype assignment in kv_cache_config.
• The configuration is consistent with the stated OOM mitigation.
• No public API changes or test-list changes were found.

QA Engineer Review

• Modified the Gemma3 27B FP8 accuracy test in tests/integration/defs/accuracy/test_llm_api_pytorch.py.
• No test function was added or removed.
• No tests/integration/test_lists/ coverage change was reported.
• Five reproduced runs passed with consistent CNN/DailyMail, GSM8K, and MMLU accuracy results.
• Verdict: sufficient.

Description

The Gemma3 27B FP8 accuracy test can run out of memory during MMLU on an H100 even with max_seq_len=4096 because the default KV cache allocation reserves 90% of free GPU memory and leaves insufficient dynamic PyTorch headroom.

This test-only change sets free_gpu_memory_fraction=0.5 in the FP8 test's KvCacheConfig, matching the limit already used by the Gemma3 BF16 test. In the reproduced failure environment, this reduced the final KV cache quota from approximately 42.38 GiB to 23.55 GiB and prevented the runtime OOM without changing product defaults or allocator behavior.

Test Coverage

• python3 -m pytest -vs --timeout-method=thread --timeout=3600 accuracy/test_llm_api_pytorch.py::TestGemma3_27BInstruct::test_fp8_prequantized
• Replayed with build 2941's exact CI-built TensorRT-LLM rc26 wheel, source bundle, and runtime container on H100 80GB GPUs: 5/5 runs passed across two nodes with no OOM recurrence.
• Accuracy results were consistent across all five runs: CNN/DailyMail 28.730, GSM8K 91.016, and MMLU 77.364.

PR Checklist

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e4fd6333-ec9d-4588-977b-4f5538ec105f

📥 Commits

Reviewing files that changed from the base of the PR and between 87f5d10 and 2b47250.

📒 Files selected for processing (1)

• tests/integration/defs/accuracy/test_llm_api_pytorch.py

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/integration/defs/accuracy/test_llm_api_pytorch.py

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The Gemma 3 27B FP8 accuracy test moves KV-cache memory-headroom comments above the KvCacheConfig call. Runtime configuration remains unchanged.

Changes

Gemma FP8 accuracy test

Layer / File(s)	Summary
KV-cache allocation documentation tests/integration/defs/accuracy/test_llm_api_pytorch.py	Comments document the 50% free GPU memory cap and PyTorch activation headroom before the configuration call.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Merge Risk: ⚪ Minimal · up to 2b472

This test-only change reduces Gemma3 FP8 KV cache allocation to preserve runtime memory headroom; no actionable merge-blocking risk remains after normal checks and review.

Suggested reviewers: niukuo, qijune, larryxfly, dpitman-nvda

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 1 functions across 1 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title identifies the NVBugs issue, uses the valid "fix" type, and clearly describes the main change: limiting KV-cache memory for the Gemma3 FP8 accuracy test.
Description check	✅ Passed	The description explains the issue and solution, provides specific test coverage and results, and includes the required checklist with the review confirmation checked.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[ruodil]

Review summary - Approve

Reviewed the full diff; no blocking or major issues found.

Left 1 non-blocking note(s) inline on the diff:

• [MINOR] tests/integration/defs/accuracy/test_llm_api_pytorch.py:1406 - free_gpu_memory_fraction=0.5 is an unexplained magic value in the test

Minor, non-blocking notes:

• tests/integration/defs/accuracy/test_llm_api_pytorch.py: Halving the KV cache pool can reduce concurrency and lengthen this test
  • Dropping the KV cache pool from ~0.9 to 0.5 of free memory (per the PR description, ~42.4 GiB -> ~23.6 GiB) halves the number of resident KV blocks. Since enable_block_reuse=False and enable_partial_reuse=False are also set on lines 1403-1404, every request pays full prefill and holds its own blocks, so the achievable in-flight batch size for MMLU is bounded directly by this pool. The concrete effect: MMLU/GSM8K on the 27B FP8 model will schedule fewer requests per iteration and the wall-clock of test_fp8_prequantized grows, which matters because this test already runs under a 3600s timeout in the reproducer command. This is not a correctness problem and should not block, but if post-merge L0 starts timing out on this test rather than OOM-ing, this line is the cause — the fix would then be a targeted max_batch_size/max_num_tokens cap plus a modest fraction rather than a single blunt 0.5.

Automated review by NVCortex Lite, run by @ruodil.

[erictsai-nv]

/bot run --stage-list "DGX_H100-PyTorch-Post-Merge-2"

[tensorrt-cicd]

PR_Github #70871 [ run ] triggered by Bot. Commit: 2b47250 Link to invocation

[tensorrt-cicd]

PR_Github #70871 [ run ] completed with state FAILURE. Commit: 2b47250
/LLM/main/L0_MergeRequest_PR pipeline #58046 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[erictsai-nv]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #70917 [ run ] triggered by Bot. Commit: 2b47250 Link to invocation

[tensorrt-cicd]

PR_Github #70917 [ run ] completed with state SUCCESS. Commit: 2b47250
/LLM/main/L0_MergeRequest_PR pipeline #58083 completed with status: 'SUCCESS'

CI Report

Link to invocation

[erictsai-nv]

Closing in favor of #18557, which fixes the underlying FlashInfer plan-cache growth.