Skip to content

[IsaacLab CI] upload omni-github test-result artifacts from CI jobs - #6234

Merged
mataylor-nvidia merged 26 commits into
isaac-sim:developfrom
mataylor-nvidia:mataylor/omni-github-test-results
Jun 27, 2026
Merged

[IsaacLab CI] upload omni-github test-result artifacts from CI jobs#6234
mataylor-nvidia merged 26 commits into
isaac-sim:developfrom
mataylor-nvidia:mataylor/omni-github-test-results

Conversation

@mataylor-nvidia

@mataylor-nvidia mataylor-nvidia commented Jun 22, 2026

Copy link
Copy Markdown

Description

  • Add upload-omni-github-test-results composite action to convert JUnit XML into the omni-github test-result artifact contract.

  • Wire the upload step into install-ci, package test, and Docker test composite actions.

  • Use IsaacLab upstream repository ID 567038244 in artifact identities for omni-github registration.

  • Installation Tests workflow uploads pytest-results-*--v1-567038244-* artifacts

  • omni-github registration PR can reference real sample artifact names from this run

link to slack discussion:

https://nvidia.slack.com/archives/C0B16FFDDBP/p1782216333267729

link to artifact schema:

https://github.com/NVIDIA-Omniverse/omni-github/blob/main/docs/test-results/clients/artifact-upload/artifact-schema.md

reference to junit format: https://github.com/testmoapp/junitxml#structure

link to test results dashboard:

https://omni-github-stg.horde-aks.nvidia.com/test-results/repository-test-summary?repository=isaac-sim%2FIsaacLab&days=30&prMode=include&groupBy=job_name&groupValue=isaaclab+%28core%29+%5B3%2F3%5D

example artifact from most recent pipeline run:

corresponding golden images are uploaded with

Fixes # (OMPE-98494)

Type of change

  • New feature (non-breaking change which adds functionality)

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the changelog and the corresponding version in the extension's config/extension.toml file
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

@greptile-apps

greptile-apps Bot commented Jun 22, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a new upload-omni-github-test-results composite action that converts JUnit XML reports into the omni-github test-result artifact format and wires it into install-ci-run, run-package-tests, and run-tests. A bundled Python converter handles skip/fail/crash/timeout detection, and the JSON output is validated against a local copy of the omni-github schema before upload.

  • New composite action (upload-omni-github-test-results) derives the runner platform from RUNNER_OS/RUNNER_ARCH, creates a per-run venv to validate the output against a local JSON schema, and uploads the artifact using the omni-github naming contract (--v1-<repo_id>-<run_id>-<run_attempt>-<check_run_id>).
  • Caller updates in install-ci-run, run-package-tests, and run-tests add id labels to the existing JUnit/comparison-image upload steps so their artifact-url outputs can be forwarded as log_paths into the converted result rows.
  • build.yaml sets explicit omni-github-test-type values (training-e2e, rendering-correctness) on the relevant jobs.

Confidence Score: 5/5

Safe to merge; all new logic is additive CI instrumentation with graceful fallbacks that cannot block existing test reporting.

The converter correctly handles passed, skipped, failed, and error test states. The composite action gracefully skips upload when the JUnit file is missing, the schema validation venv is self-contained, and callers all set if: always() so upload failures cannot block actual test results. No existing production code paths are modified.

.github/actions/upload-omni-github-test-results/action.yml — two URL inputs bypass the env-var pattern used by all other inputs in the same run block.

Important Files Changed

Filename Overview
.github/actions/upload-omni-github-test-results/action.yml New composite action; derives platform from RUNNER_OS/RUNNER_ARCH, creates a per-run venv for schema validation, and uploads the artifact. Two inputs (junit-log-url, comparison-images-url) are interpolated directly into the shell script rather than via env vars, inconsistent with the other four inputs.
.github/actions/upload-omni-github-test-results/junit_to_omni_github_results.py JUnit-to-omni-github converter; correctly handles passed/skipped/failed/error cases, crash and timeout flag detection, and message truncation. Logic is sound and well-tested.
.github/actions/upload-omni-github-test-results/result-json.schema.json Local copy of the omni-github result JSON schema; used for pre-upload validation. Schema definition is consistent with the converter's output format.
.github/actions/upload-omni-github-test-results/test_junit_to_omni_github_results.py Unit tests for the converter; covers failure details, skip, crash/timeout flags, and log_path propagation. All three test cases use realistic JUnit XML fixtures.
.github/actions/run-tests/action.yml Adds id labels to the comparison-images and junit upload steps, new omni-github-test-type input, and the omni-github upload step. Minor style inconsistency: id : upload-comparison-images has an extra space before the colon.
.github/actions/run-package-tests/action.yml Adds omni-github-test-type input and threads it through to run-tests; straightforward pass-through with no issues.
.github/actions/install-ci-run/action.yml Adds id to existing JUnit upload step and wires the new omni-github upload composite action with test-type: installation-e2e. Change is correct and minimal.
.github/workflows/build.yaml Adds omni-github-test-type values to three existing job calls (training-e2e, rendering-correctness, rendering-correctness-kitless); all values are consistent with the new input's description.
.github/workflows/install-ci.yml Only adds a blank line before the jobs section after the permissions block; no functional change.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Job as CI Job (install-ci-run / run-tests)
    participant JUnit as Upload JUnit XML actions/upload-artifact@v7
    participant Comp as Upload Comparison Images actions/upload-artifact@v7
    participant Convert as Convert JUnit XML (bash + python3)
    participant Schema as Schema Validation (jsonschema venv)
    participant Upload as Upload omni-github artifact actions/upload-artifact@v7
    participant OmniGH as omni-github dashboard

    Job->>JUnit: results.xml
    JUnit-->>Convert: artifact-url (junit-log-url)
    Job->>Comp: comparison-images/ (run-tests only)
    Comp-->>Convert: artifact-url (comparison-images-url)
    Convert->>Convert: derive app_platform from RUNNER_OS/RUNNER_ARCH
    Convert->>Convert: python3 junit_to_omni_github_results.py
    Convert->>Schema: pip install jsonschema in venv
    Schema->>Schema: validate test_results.json vs result-json.schema.json
    alt "validation passes and tests > 0"
        Schema-->>Convert: "upload=true"
        Convert->>Upload: artifact_dir
        Upload-->>OmniGH: "artifact name = prefix--v1-{repo_id}-{run_id}-{attempt}-{check_run_id}"
    else no JUnit file or empty suite or schema fail
        Schema-->>Convert: "upload=false warning emitted"
    end
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Job as CI Job (install-ci-run / run-tests)
    participant JUnit as Upload JUnit XML actions/upload-artifact@v7
    participant Comp as Upload Comparison Images actions/upload-artifact@v7
    participant Convert as Convert JUnit XML (bash + python3)
    participant Schema as Schema Validation (jsonschema venv)
    participant Upload as Upload omni-github artifact actions/upload-artifact@v7
    participant OmniGH as omni-github dashboard

    Job->>JUnit: results.xml
    JUnit-->>Convert: artifact-url (junit-log-url)
    Job->>Comp: comparison-images/ (run-tests only)
    Comp-->>Convert: artifact-url (comparison-images-url)
    Convert->>Convert: derive app_platform from RUNNER_OS/RUNNER_ARCH
    Convert->>Convert: python3 junit_to_omni_github_results.py
    Convert->>Schema: pip install jsonschema in venv
    Schema->>Schema: validate test_results.json vs result-json.schema.json
    alt "validation passes and tests > 0"
        Schema-->>Convert: "upload=true"
        Convert->>Upload: artifact_dir
        Upload-->>OmniGH: "artifact name = prefix--v1-{repo_id}-{run_id}-{attempt}-{check_run_id}"
    else no JUnit file or empty suite or schema fail
        Schema-->>Convert: "upload=false warning emitted"
    end
Loading

Reviews (16): Last reviewed commit: "skip upload on empty tests" | Re-trigger Greptile

Comment thread .github/actions/upload-omni-github-test-results/junit_to_omni_github_results.py Outdated
Comment thread .github/actions/upload-omni-github-test-results/action.yml Outdated
@mataylor-nvidia

Copy link
Copy Markdown
Author

@greptileai

…or-nvidia/IsaacLab into mataylor/omni-github-test-results
@mataylor-nvidia mataylor-nvidia changed the title ci: upload omni-github test-result artifacts from CI jobs Draft: ci: upload omni-github test-result artifacts from CI jobs Jun 22, 2026
@mataylor-nvidia mataylor-nvidia changed the title Draft: ci: upload omni-github test-result artifacts from CI jobs ci: upload omni-github test-result artifacts from CI jobs Jun 22, 2026
@mataylor-nvidia
mataylor-nvidia marked this pull request as draft June 22, 2026 20:03
Comment thread .github/actions/install-ci-run/action.yml Outdated
@mataylor-nvidia mataylor-nvidia self-assigned this Jun 24, 2026
@mataylor-nvidia
mataylor-nvidia force-pushed the mataylor/omni-github-test-results branch from ba6ba0e to 245ebd1 Compare June 24, 2026 15:19
@mataylor-nvidia
mataylor-nvidia marked this pull request as ready for review June 24, 2026 15:22
Comment thread .github/actions/upload-omni-github-test-results/junit_to_omni_github_results.py Outdated
Comment thread .github/actions/upload-omni-github-test-results/action.yml Outdated
Comment thread .github/actions/upload-omni-github-test-results/action.yml Outdated
@mataylor-nvidia mataylor-nvidia changed the title ci: upload omni-github test-result artifacts from CI jobs [IsaacLab CI] upload omni-github test-result artifacts from CI jobs Jun 26, 2026
@mataylor-nvidia
mataylor-nvidia requested a review from nvsekkin June 26, 2026 15:02
# Validating with a local pinned copy of the schemas is recommended by the omni-github team.
schema_venv="${RUNNER_TEMP}/omni-github-jsonschema-venv"
python3 -m venv "$schema_venv"
"$schema_venv/bin/python" -m pip install -q jsonschema

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

polluting isaac_sim venv with jsonschema dep causes errors

@mataylor-nvidia
mataylor-nvidia merged commit 0316aea into isaac-sim:develop Jun 27, 2026
60 of 61 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants