Split CI tests for Numba-CUDA and MLIR

[isVoid]

Summary:

• Add changed-path detection so PR CI can run Numba-CUDA tests, MLIR tests, or both based on touched paths.
• Split conda and wheel test paths for Numba-CUDA and MLIR, with MLIR jobs installing numba-cuda-mlir and explicitly removing numba-cuda from that environment.
• Run ast_canopy in the MLIR conda path, skip CCCL there, and treat shared routing/config files as touching both test paths.

Testing:

• python -m py_compile ci/detect_test_paths.py ci/run_tests.py
• bash -n ci/test_conda_python.sh
• Parsed updated workflow YAML files with PyYAML
• Parsed numbast/pyproject.toml with tomllib
• python ci/detect_test_paths.py --base public/main --head HEAD
• env PYTHONPATH=numbast/src:ast_canopy python -m pytest numbast/src/numbast/tools/tests/test_mlir_backend_routing.py numbast/src/numbast/tools/tests/test_config_schema_docs.py

Summary by CodeRabbit

• New Features

  • Introduced MLIR backend support as an alternative to Numba-CUDA for testing and deployment
  • Implemented lazy module attribute loading for improved package import performance

• Chores

  • Restructured CI pipelines to support backend-specific test execution
  • Enhanced test infrastructure with automatic change detection for targeted test runs

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR introduces multi-backend test infrastructure for MLIR and Numba-CUDA through path-based change detection, conditional CI job execution, backend-parameterized workflows, and runtime test branching. It also refactors module initialization to use lazy attribute loading, avoiding eager imports of backend-specific bindings.

Changes

Backend test orchestration and execution

Layer / File(s)	Summary
Path-based test backend detection ci/detect_test_paths.py	New script classifies file paths into boolean flags controlling whether Numba-CUDA and/or MLIR test suites run, with fallback to enable both when no path rules match.
PR workflow orchestration with change detection .github/workflows/pr.yaml	Introduces detect-changes job that diffs against origin/main and runs path classification to set outputs gating test-conda-mlir and test-wheels-mlir execution; numba-cuda variants are disabled; docs depend on all test results.
Workflow parameterization for backend selection .github/workflows/conda-python-test.yaml, .github/workflows/wheels-test.yaml	Both workflows accept test_backend input parameter (default "numba-cuda") and expose it as TEST_BACKEND environment variable.
CI workflow backend-specific job split .github/workflows/ci.yaml	Splits generic test jobs into numba-cuda and mlir variants wired via backend parameters; updates build-docs dependencies to include all new test jobs.
Conda test execution with backend-specific setup ci/test_conda_python.sh	Validates TEST_BACKEND, constructs conda packages dynamically (numba-cuda or mlir backend), performs backend-specific environment setup and verification, and branches test runner command.
Test runner backend flag support ci/run_tests.py	Adds --mlir flag to select MLIR test suite execution; validates mutual exclusivity with other subpackage selectors.
Wheels test execution with backend branching .github/workflows/wheels-test.yaml	Validates TEST_BACKEND, conditionally installs backend-specific wheels and packages, swaps numba-cuda for numba-cuda-mlir on mlir backend, and branches pytest: numba tests for numba-cuda, mlir experimental suite for mlir, CCCL tests only for numba-cuda.
Module lazy attribute loading numbast/src/numbast/__init__.py	Replaces eager imports with __getattr__ and __dir__ for lazy module attribute loading; guards __version__ with fallback to "0+unknown" when metadata unavailable; removes __all__ list.
Lazy loading test validation numbast/src/numbast/tests/test_package_init.py	Verifies that importing numbast.experimental avoids eager loading backend modules into sys.modules.
Package dependencies and test infrastructure numbast/pyproject.toml, numbast/src/numbast/experimental/mlir/tests/conftest.py, numbast/src/numbast/experimental/mlir/tools/tests/test_symbol_exposure.py	Pyproject adds mlir and test-cu12-mlir/test-cu13-mlir extras; test fixture uses parse_declarations_from_source with explicit SM target; symbol exposure test uses stable cuda.core.Device path.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 Two backends dance in conditional flow,
Path detection whispers which tests to go,
Lazy loading waits 'til you call its name,
MLIR and CUDA now share the game! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 11.11% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title directly and clearly describes the main change: splitting CI tests to handle both Numba-CUDA and MLIR backends separately, which aligns with the comprehensive refactoring across workflow files, test scripts, and configuration changes.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/split-ci-numba-cuda-mlir

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

Doc Preview CI
🚀 View pre-built docs at
https://NVIDIA.github.io/numbast/pr-preview/pr-363/
Preview will be ready when GitHub Pages deployment finishes.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/ci.yaml:
- Around line 15-54: The workflow jobs (e.g., test-conda-numba-cuda,
test-conda-mlir, build-wheels, test-wheels-numba-cuda, test-wheels-mlir)
currently use secrets: inherit and implicit token permissions; tighten them by
adding an explicit permissions: block at the top-level or per-job with only the
minimum required scopes (e.g., contents: read, packages: read/write if
publishing, id-token: write only if needed), and replace secrets: inherit with
explicit secret mappings (e.g., MY_PYPI_TOKEN: ${{ secrets.MY_PYPI_TOKEN }})
only for the secrets each reusable workflow actually requires; ensure each uses:
call includes the same minimal permissions/secrets mapping so the called
reusable workflows run with least privilege.

In @.github/workflows/pr.yaml:
- Around line 41-43: The checkout step currently uses actions/checkout@v6 with
fetch-depth: 0 but leaves credentials persisted; for security hardening add
persist-credentials: false to the checkout action configuration (the
actions/checkout@v6 step) so the job does not keep repo credentials after
checkout—update the checkout step options to include persist-credentials: false
alongside fetch-depth: 0.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 37b04301-e4e3-47b7-aa79-9647eb851c40

📥 Commits

Reviewing files that changed from the base of the PR and between fb74340 and 98b8ac2.

📒 Files selected for processing (12)

• .github/workflows/ci.yaml
• .github/workflows/conda-python-test.yaml
• .github/workflows/pr.yaml
• .github/workflows/wheels-test.yaml
• ci/detect_test_paths.py
• ci/run_tests.py
• ci/test_conda_python.sh
• numbast/pyproject.toml
• numbast/src/numbast/__init__.py
• numbast/src/numbast/experimental/mlir/tests/conftest.py
• numbast/src/numbast/experimental/mlir/tools/tests/test_symbol_exposure.py
• numbast/src/numbast/tests/test_package_init.py