[TRTLLM-12090][infra] add static tests validation hook

[xinhe-nv]

@coderabbitai summary

Description

local test result.

% git commit -sam "add tests validate script"                     
[WARNING] Unexpected key(s) present at root: common-files, legacy-files, static-analysis-files
isort............................................................................Passed
CRLF end-lines remover...........................................................Passed
yapf.............................................................................Passed
check for added large files......................................................Passed
check for merge conflicts........................................................Passed
check for broken symlinks....................................(no files to check)Skipped
detect private key...............................................................Passed
fix end of files.................................................................Passed
check yaml.......................................................................Passed
trim trailing whitespace.........................................................Passed
check toml...................................................(no files to check)Skipped
mixed line ending............................................(no files to check)Skipped
debug statements (python)....................................(no files to check)Skipped
check json...................................................(no files to check)Skipped
autoflake........................................................................Passed
clang-format.................................................(no files to check)Skipped
cmake-format.................................................(no files to check)Skipped
codespell........................................................................Passed
ruff.............................................................................Passed
ruff-format......................................................................Passed
mdformat.....................................................(no files to check)Skipped
Validate AutoDeploy model registry...........................(no files to check)Skipped
ruff-legacy (baseline-gated lint for legacy files)...............................Passed
verify legacy lint configs are up to date........................................Passed
Check for tabs and multiple spaces in test_lists txt files...(no files to check)Skipped
Checks for duplicated test items in waives.txt...................................Passed
Validate test list entries exist in source files (AST)...........................Failed
- hook id: validate-test-lists
- exit code: 1

-----------Skipping L0 test list verification.-----------
-----------Skipping QA test list verification.-----------
-----------Skipping waive list verification.-----------
-----------Starting AST test list validation...-----------
Found 4 validation error(s):

METHOD NOT FOUND: accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_chunked_prefill
  -> tests/integration/test_lists/qa/llm_function_core.txt:531
  -> tests/integration/test_lists/qa/llm_function_rtx6k.txt:170

METHOD NOT FOUND: accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_chunked_prefill
  -> tests/integration/test_lists/qa/llm_function_core.txt:532

METHOD NOT FOUND: accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_chunked_prefill
  -> tests/integration/test_lists/qa/llm_function_core.txt:533

METHOD NOT FOUND: accuracy/test_llm_api_pytorch.py::TestGPTOSS::test_w4_chunked_prefill
  -> tests/integration/test_lists/qa/llm_function_core.txt:534

Disallow raw pinned-memory APIs in runtime code..............(no files to check)Skipped
Run static analysis of type correctness using mypy...........(no files to check)Skipped
- hook id: type-check

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[xinhe-nv]

/bot run --skip-test

[coderabbitai]

📝 Walkthrough

Walkthrough

This pull request introduces a test validation and cleanup infrastructure system. A new pre-commit hook invokes test list validation, which uses AST analysis to verify test list entries exist in source files. A new cleanup script automates removal of unused test and helper functions from test directories.

Changes

Cohort / File(s)	Summary
Pre-commit Configuration .pre-commit-config.yaml	Adds a new pre-commit hook that runs test list validation against integration test list files and Python source files.
Test List Validation scripts/check_test_list.py	Extends script with AST-based validation mode. Adds functions to parse test list entries, build AST indexes of source files, validate test entries against source, and detect malformed entries, missing files/classes/methods, and invalid waive parameter IDs. Introduces new CLI flags --validate, --test-lists-dir, --test-base-dir.
Cleanup Automation scripts/clean_tests.py	New standalone script that scans test lists, removes unlisted test functions, runs vulture to detect and remove unused helpers, and performs cross-file symbol reference checks to identify unreferenced non-test symbols.

Sequence Diagram(s)

sequenceDiagram
    participant PC as Pre-commit Hook
    participant VS as Validation Script
    participant TLP as Test List Parser
    participant AST as AST Indexer
    participant VL as Validation Logic

    PC->>VS: invoke check_test_list.py --validate
    VS->>TLP: collect_entries(test_lists_dir)
    TLP->>TLP: parse .txt/.yml/waives.txt files
    TLP-->>VS: test entry list
    
    VS->>AST: build_ast_index(source_files)
    AST->>AST: extract functions/methods/parameters
    AST-->>VS: indexed symbols
    
    VS->>VL: validate_test_lists()
    VL->>VL: check malformed entries
    VL->>VL: verify files/classes/methods exist
    VL->>VL: validate waive parameter IDs
    VL-->>VS: validation errors (if any)
    
    VS-->>PC: pass/fail result

Loading

Estimated Code Review Effort

🎯 4 (Complex) | ⏱️ ~60 minutes

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR description is largely blank—it contains only the template structure with empty sections and no actual content explaining what was changed or why.	Fill in the Description section explaining the purpose of the new validation scripts, and the Test Coverage section listing relevant tests that verify the new functionality.
Docstring Coverage	⚠️ Warning	Docstring coverage is 63.64% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main change: adding test validation scripts as infrastructure. It references the JIRA ticket and type appropriately.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 7

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@scripts/check_test_list.py`:
- Around line 733-744: The --test-lists-dir and --test-base-dir parser args (and
their defaults _DEFAULT_TEST_LISTS_DIR/_DEFAULT_TEST_BASE_DIR) must be resolved
as absolute paths relative to the repository root (llm_src) before invoking the
validator; update the main/validate invocation to call
os.path.abspath(os.path.join(llm_src, value)) (or equivalent) for those args so
passing "tests/..." works regardless of CWD, and then pass the normalized paths
into the validator call.
- Around line 402-404: The comprehension's hanging indent around the sorted()
call trips flake8 E125/E131; reformat the for loop so the key= lambda expression
is not split with a misaligned hanging indent: move the entire key=lambda x:
(...) expression onto the same line as sorted( or place the tuple (x[0], x[1] or
"", x[2], x[3] or "") on the next indented line aligned under the opening
parenthesis, keeping the variable names waive_only_keys and the lambda x
reference unchanged (i.e., preserve the sorted(waive_only_keys, key=lambda x:
(x[0], x[1] or "", x[2], x[3] or "")) structure).
- Around line 381-399: collecting malformed entries from both active_malformed
and waive_malformed currently double-reports the same bad lines because
collect_entries(include_waives=True) rescans non-waive files; fix by
deduplicating before adding to errors—e.g., build an ordered unique list (or
append only those in waive_malformed that are not in active_malformed) and then
extend errors with that deduplicated set; reference the variables
active_malformed, waive_malformed and errors (and collect_entries) when making
the change.

In `@scripts/clean_tests.py`:
- Around line 158-185: The function _collect_top_level_defs is using ast.walk
which also visits nested functions and class methods; restrict collection to
actual module-level definitions by iterating only over tree.body and selecting
instances of ast.FunctionDef or ast.AsyncFunctionDef there (check decorator_list
and determine is_fixture as before), so update _collect_top_level_defs to
inspect top-level nodes from tree.body (keeping the existing decorated/fixture
logic and keys in defs) instead of using ast.walk to avoid capturing nested defs
and methods.
- Around line 69-106: The current _parse_test_name function only returns the
bare function token, causing collisions across modules; update it (or replace
it) to return a canonical test identifier including relative file path, optional
class name, and function name (e.g., "path.py::Class::test_name" or
"path.py::test_name") rather than just the final token; adapt get_used_tests to
add those full identifiers to the used set (use the same parsing semantics as
scripts/check_test_list.py's parser for .txt and .yml entries to preserve class
and param stripping) and ensure YAML "- " prefixes and inline TIMEOUT(...)
tokens are ignored the same way so duplicate test names in different
files/classes are treated distinctly.
- Around line 1-21: This new source file is missing the required NVIDIA
copyright header; add the repository-mandated NVIDIA copyright header (with the
year of latest meaningful modification) to the top of the file before any
imports or code. Update the header in scripts/clean_tests.py so it appears above
the first symbol (e.g., before DEFAULT_TEST_DIR and CONFTEST) and ensure the
wording and year match other files in the repo's header template.
- Around line 35-37: The run_cmd function uses subprocess.run(..., shell=True)
and returns only stdout, allowing shell injection when test_dir is interpolated
and hiding tool failures; change run_cmd to call subprocess.run with a list of
arguments (no shell=True) and include test_dir as a separate arg to avoid shell
interpretation, capture stderr as well, check result.returncode and on non-zero
either raise subprocess.CalledProcessError or log/exit with both stdout and
stderr so missing/broken tools (e.g., vulture) aren't silently treated as
success; update all call sites that pass a single combined string to run_cmd to
pass argument lists (reference: run_cmd and the call that builds the vulture
command).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b7c91681-a031-4ceb-9aa1-01e42b13dcd9

📥 Commits

Reviewing files that changed from the base of the PR and between 5277e62 and 790af50.

📒 Files selected for processing (3)

• .pre-commit-config.yaml
• scripts/check_test_list.py
• scripts/clean_tests.py

[tensorrt-cicd]

PR_Github #45378 [ run ] triggered by Bot. Commit: 790af50 Link to invocation

[tensorrt-cicd]

PR_Github #45378 [ run ] completed with state FAILURE. Commit: 790af50
/LLM/main/L0_MergeRequest_PR pipeline #35619 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[xinhe-nv]

/bot run --skip-test

[tensorrt-cicd]

PR_Github #45631 [ run ] triggered by Bot. Commit: db0ff55 Link to invocation

[tensorrt-cicd]

PR_Github #45631 [ run ] completed with state SUCCESS. Commit: db0ff55
/LLM/main/L0_MergeRequest_PR pipeline #35845 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[xinhe-nv]

/bot run --skip-test

[xinhe-nv]

/bot run

[tensorrt-cicd]

PR_Github #45658 [ run ] triggered by Bot. Commit: af1d9c0 Link to invocation

[tensorrt-cicd]

PR_Github #45660 [ run ] triggered by Bot. Commit: af1d9c0 Link to invocation

[tensorrt-cicd]

PR_Github #45658 [ run ] completed with state ABORTED. Commit: af1d9c0

Link to invocation

[tensorrt-cicd]

PR_Github #45660 [ run ] completed with state FAILURE. Commit: af1d9c0
/LLM/main/L0_MergeRequest_PR pipeline #35871 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[xinhe-nv]

/bot run

[tensorrt-cicd]

PR_Github #45672 [ run ] triggered by Bot. Commit: 997c037 Link to invocation

[xinhe-nv]

/bot run --skip-test

[tensorrt-cicd]

PR_Github #45687 [ run ] triggered by Bot. Commit: 997c037 Link to invocation

[tensorrt-cicd]

PR_Github #45687 [ run ] completed with state SUCCESS. Commit: 997c037
/LLM/main/L0_MergeRequest_PR pipeline #35893 (Partly Tested) completed with status: 'SUCCESS'

CI Report

Link to invocation

[xinhe-nv]

/bot reuse-pipeline

[tensorrt-cicd]

PR_Github #45897 [ reuse-pipeline ] triggered by Bot. Commit: 86df1ee Link to invocation

[tensorrt-cicd]

PR_Github #45897 [ reuse-pipeline ] completed with state SUCCESS. Commit: 86df1ee
Reusing PR_Github #45687 (Partly Tested) for commit 86df1ee

Link to invocation