[None][infra] Generate json with cmake fetched contents in build stage

[yuanjingx87]

Summary by CodeRabbit

Release Notes

• New Features

  • Build artifacts now include comprehensive third-party dependency source information for improved transparency.

• Chores

  • Enhanced build pipeline with automated third-party dependency tracking and mirror management.
  • Updated dependency configuration for consistency.

Description

When build the wheel, generate a json with all cmake fetched packages, copy that json file to release containers

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[yuanjingx87]

/bot run

[coderabbitai]

📝 Walkthrough

Walkthrough

The changes introduce a new Python script to generate a third-party-sources.json artifact documenting third-party dependencies used in the CMake build. The script maps dependencies from fetch metadata, creates GitLab OSS component mirrors when needed, and updates the build pipeline to include this JSON in the tarball output.

Changes

Cohort / File(s)	Summary
Dependency Declaration 3rdparty/fetch_content.json	Updated deep_ep_download dependency's git_repository field from a templated ${github_base_url} URL to hardcoded https://github.com/deepseek-ai/DeepEP.
Build Pipeline Integration jenkins/Build.groovy	Added invocation of scripts/generate_cpp_dependency_json.py after wheel build to generate C++ dependency JSON artifact. Configured tarball packaging to include third-party-sources.json in build output.
New Dependency Generator scripts/generate_cpp_dependency_json.py	New CLI script that enumerates CMake dependencies from a --deps-dir, maps them to metadata in fetch_content.json, queries/creates GitLab OSS component mirrors under nvidia/tensorrt-llm/oss-components, verifies refs exist in mirrors, and outputs updated dependency records as third-party-sources.json.

Sequence Diagram(s)

sequenceDiagram
    actor Jenkins
    participant Script as generate_cpp_dependency_json.py
    participant FileSystem
    participant Metadata as fetch_content.json
    participant GitLab API
    participant Output as third-party-sources.json

    Jenkins->>Script: Invoke with --deps-dir and --output-dir
    Script->>FileSystem: Enumerate *-src directories
    Script->>Metadata: Load dependency metadata index
    
    loop For each dependency
        Script->>GitLab API: Check if mirror exists<br/>(nvidia/tensorrt-llm/oss-components)
        alt Mirror not found
            Script->>GitLab API: Create new mirror project<br/>(upstream URL, public)
            GitLab API-->>Script: Mirror creation initiated
            Script->>GitLab API: Poll mirror status<br/>(until import completes or timeout)
        end
        Script->>GitLab API: Verify tag/ref exists in mirror
        Script->>Script: Build updated dependency record<br/>(git_repository → mirror URL)
    end
    
    Script->>Output: Write collected records as indented JSON
    Script-->>Jenkins: Script completes

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Description check	❓ Inconclusive	The PR description provides a brief explanation of what changes ('generate a json with all cmake fetched packages, copy that json file to release containers'), but lacks details on why this change is needed, test coverage information, and leaves the Test Coverage section incomplete.	Expand the description to explain the motivation behind generating and distributing this JSON file, and clearly document which tests validate the new script functionality and build pipeline changes.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and specifically describes the main change: generating a JSON file with CMake fetched contents during the build stage, which aligns with the primary modifications across all files.
Docstring Coverage	✅ Passed	Docstring coverage is 87.50% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Review rate limit: 9/10 reviews remaining, refill in 6 minutes.}

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 4

🧹 Nitpick comments (2)

scripts/generate_cpp_dependency_json.py (2)

93-93: Remove debug print statement.

This print(upstream_url) appears to be debug output. Use logger.info() for consistency with the rest of the script, or remove it entirely.

Proposed fix

 def create_oss_component(
     package_name: str, namespace_id: int, upstream_url: str
 ) -> tuple[str, int]:
     """Create a new project under oss-components with a pull mirror and return (web_url, project_id)."""
-    print(upstream_url)
+    logger.debug("Creating OSS component from upstream: %s", upstream_url)
     payload = json.dumps(

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@scripts/generate_cpp_dependency_json.py` at line 93, Remove the stray debug
print by deleting the print(upstream_url) call or replacing it with a structured
logger call (e.g., logger.info("upstream_url=%s", upstream_url)) to match the
script's existing logging; locate the print statement in
generate_cpp_dependency_json.py (the print(upstream_url) line) and either remove
it or convert it to use logger.info so output is consistent with other log
messages.

---

134-134: Add return type annotation to main().

As per coding guidelines, always annotate Python function return types.

Proposed fix

-def main():
+def main() -> None:

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@scripts/generate_cpp_dependency_json.py` at line 134, The function signature
for main() lacks a return type annotation; update the declaration of main (def
main()) in generate_cpp_dependency_json.py to include an explicit return type
(e.g., def main() -> None:) and ensure any return statements inside main conform
to that type (or remove returns that return values) so the function signature
and implementation remain consistent with the codebase typing guidelines.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@jenkins/Build.groovy`:
- Line 402: The shell invocation running generate_cpp_dependency_json.py is
executed from the workspace root so relative paths fail; update the command that
currently calls sh "python3 scripts/generate_cpp_dependency_json.py --deps-dir
cpp/build/_deps --output-dir ./" to first change to ${LLM_ROOT} (e.g., use cd
${LLM_ROOT} && ...) so both the script path
scripts/generate_cpp_dependency_json.py and the deps-dir cpp/build/_deps resolve
correctly and the expected output third-party-sources.json is created under
${LLM_ROOT}; ensure the same pattern is used wherever the script is invoked so
later references (line 417) find the output.

In `@scripts/generate_cpp_dependency_json.py`:
- Around line 1-8: This file is missing the required NVIDIA copyright header;
add the SPDX/Apache-2.0 copyright block at the very top of the Python module
(above the existing module docstring) using the provided text and the latest
modification year (2026) so the header precedes the triple-quoted docstring that
currently begins the file.
- Around line 183-187: The code assumes dep_source_index[package_name] exists
and can raise KeyError when a *-src dir has no entry in fetch_content.json;
modify the block handling tag confirmation (the code that calls
commit_exists_in_project and appends to third_party_source_list) to first check
for the package key (e.g., using package_name in dep_source_index or
dep_source_index.get(package_name)), and if missing, log a clear warning
mentioning package_name/project_id and skip appending; otherwise merge the dict
as before ({**dep_source_index[package_name], "git_repository": oss_url}) so you
avoid the KeyError.
- Line 20: Replace the hardcoded GitLab PAT assigned to the TOKEN variable in
generate_cpp_dependency_json.py by reading it from an environment variable
(e.g., GITLAB_OSS_TOKEN) and fail fast with a clear error if the env var is
missing; update any CI/Jenkins usage to pass the secret via credentials (e.g.,
withCredentials setting GITLAB_OSS_TOKEN) rather than embedding the token in the
script, and remove the literal token string so it is not stored in the repo or
history.

---

Nitpick comments:
In `@scripts/generate_cpp_dependency_json.py`:
- Line 93: Remove the stray debug print by deleting the print(upstream_url) call
or replacing it with a structured logger call (e.g.,
logger.info("upstream_url=%s", upstream_url)) to match the script's existing
logging; locate the print statement in generate_cpp_dependency_json.py (the
print(upstream_url) line) and either remove it or convert it to use logger.info
so output is consistent with other log messages.
- Line 134: The function signature for main() lacks a return type annotation;
update the declaration of main (def main()) in generate_cpp_dependency_json.py
to include an explicit return type (e.g., def main() -> None:) and ensure any
return statements inside main conform to that type (or remove returns that
return values) so the function signature and implementation remain consistent
with the codebase typing guidelines.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f0f47d07-8343-4773-9d55-012b6f60f433

📥 Commits

Reviewing files that changed from the base of the PR and between e903428 and 230ef2b.

📒 Files selected for processing (3)

• 3rdparty/fetch_content.json
• jenkins/Build.groovy
• scripts/generate_cpp_dependency_json.py

[tensorrt-cicd]

PR_Github #46107 [ run ] triggered by Bot. Commit: 230ef2b Link to invocation

[tensorrt-cicd]

PR_Github #46107 [ run ] completed with state FAILURE. Commit: 230ef2b
/LLM/main/L0_MergeRequest_PR pipeline #36248 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[yuanjingx87]

/bot run --stage-list "A10-PackageSanityCheck-PY310-UB2204"

[tensorrt-cicd]

PR_Github #46184 [ run ] triggered by Bot. Commit: 42c32f1 Link to invocation

[yuanjingx87]

/bot run --stage-list "A10-PackageSanityCheck-PY310-UB2204"

[tensorrt-cicd]

PR_Github #46203 [ run ] triggered by Bot. Commit: f836d75 Link to invocation

[tensorrt-cicd]

PR_Github #46184 [ run ] completed with state ABORTED. Commit: 42c32f1

Link to invocation

[tensorrt-cicd]

PR_Github #46203 [ run ] completed with state FAILURE. Commit: f836d75
/LLM/main/L0_MergeRequest_PR pipeline #36316 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[yuanjingx87]

/bot run --stage-list "Build-Docker-Images"

[tensorrt-cicd]

PR_Github #46220 [ run ] triggered by Bot. Commit: 53adf7f Link to invocation

[tensorrt-cicd]

PR_Github #46220 [ run ] completed with state FAILURE. Commit: 53adf7f
/LLM/main/L0_MergeRequest_PR pipeline #36331 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[yuanjingx87]

/bot run

[tensorrt-cicd]

PR_Github #47120 [ run ] triggered by Bot. Commit: fa84f0c Link to invocation

[tensorrt-cicd]

PR_Github #47120 [ run ] completed with state FAILURE. Commit: fa84f0c
/LLM/main/L0_MergeRequest_PR pipeline #37087 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[yuanjingx87]

/bot skip --comment "No need to run CI"

[tensorrt-cicd]

PR_Github #50909 [ skip ] triggered by Bot. Commit: d342041 Link to invocation

[tensorrt-cicd]

PR_Github #50909 [ skip ] completed with state SUCCESS. Commit: d342041
Skipping testing for commit d342041

Link to invocation