Skip to content

CI: Fix pathfinder CI infrastructure and remove redundant component testing #904

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Aug 26, 2025
Merged

CI: Fix pathfinder CI infrastructure and remove redundant component testing #904

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
61f352c
Initial plan
Copilot Aug 25, 2025
ad5736b
Update test workflows to fetch cuda-pathfinder from main branch
Copilot Aug 25, 2025
c110dcf
Consolidate GitHub CLI installation steps as requested in PR review
Copilot Aug 25, 2025
7f46e6c
Remove pathfinder test steps from test workflows
Copilot Aug 25, 2025
402759e
Remove cuda.core test steps from both test workflows
Copilot Aug 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 30 additions & 61 deletions .github/workflows/test-wheel-linux.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,11 +149,35 @@ jobs:
SHA: ${{ github.sha }}
run: ./ci/tools/env-vars test

- name: Download cuda-pathfinder build artifacts
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
name: cuda-pathfinder-wheel
path: ./cuda_pathfinder
- name: Install GitHub CLI
# gh is needed for artifact fetching in subsequent steps
run: |
# See https://github.com/cli/cli/blob/trunk/docs/install_linux.md#debian-ubuntu-linux-raspberry-pi-os-apt.
mkdir -p -m 755 /etc/apt/keyrings \
&& out=$(mktemp) && wget -nv -O$out https://cli.github.com/packages/githubcli-archive-keyring.gpg \
&& cat $out | tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null \
&& chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
&& apt update \
&& apt install gh -y
Comment on lines +152 to +162
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you look into using an existing action?

This is from ChatGPT, I didn't try it out:

      - name: Setup GitHub CLI
        uses: actions4gh/setup-gh@v1
        with:
          gh-version: latest         # or specify "2.65.0", etc.
          token: ${{ secrets.GITHUB_TOKEN }}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems setup-gh is no longer maintained... 😢 https://github.com/actions4gh/setup-gh

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW we already have a install_unix_deps action. Perhaps we could just move this bits there to allow using it to install gh?


- name: Download cuda-pathfinder build artifacts from main branch
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
MAIN_BRANCH="main"
PATHFINDER_BASENAME="cuda-pathfinder-wheel"
LATEST_MAIN_RUN_ID=$(gh run list -b ${MAIN_BRANCH} -L 1 -w "CI" -s completed -R NVIDIA/cuda-python --json databaseId | jq '.[]| .databaseId')
if [[ "$LATEST_MAIN_RUN_ID" == "" ]]; then
echo "LATEST_MAIN_RUN_ID not found!"
exit 1
fi

gh run download $LATEST_MAIN_RUN_ID -p ${PATHFINDER_BASENAME} -R NVIDIA/cuda-python
ls -al $PATHFINDER_BASENAME
mkdir -p ./cuda_pathfinder
mv $PATHFINDER_BASENAME/*.whl ./cuda_pathfinder/
rmdir $PATHFINDER_BASENAME

- name: Download cuda-python build artifacts
if: ${{ env.SKIP_CUDA_BINDINGS_TEST == '0'}}
Expand All @@ -174,16 +198,6 @@ jobs:
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
# See https://github.com/cli/cli/blob/trunk/docs/install_linux.md#debian-ubuntu-linux-raspberry-pi-os-apt.
# gh is needed for artifact fetching.
mkdir -p -m 755 /etc/apt/keyrings \
&& out=$(mktemp) && wget -nv -O$out https://cli.github.com/packages/githubcli-archive-keyring.gpg \
&& cat $out | tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null \
&& chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | tee /etc/apt/sources.list.d/github-cli.list > /dev/null \
&& apt update \
&& apt install gh -y

OLD_BRANCH=$(cat .github/BACKPORT_BRANCH)
OLD_BASENAME="cuda-bindings-python${PYTHON_VERSION_FORMATTED}-cuda*-${{ inputs.host-platform }}*"
LATEST_PRIOR_RUN_ID=$(gh run list -b ${OLD_BRANCH} -L 1 -w "build-and-test.yml" -s completed -R NVIDIA/cuda-python --json databaseId | jq '.[]| .databaseId')
Expand Down Expand Up @@ -226,29 +240,7 @@ jobs:
pwd
ls -lahR $CUDA_BINDINGS_CYTHON_TESTS_DIR

- name: Download cuda.core build artifacts
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
name: ${{ env.CUDA_CORE_ARTIFACT_NAME }}
path: ${{ env.CUDA_CORE_ARTIFACTS_DIR }}

- name: Display structure of downloaded cuda.core build artifacts
run: |
pwd
ls -lahR $CUDA_CORE_ARTIFACTS_DIR

- name: Download cuda.core Cython tests
if: ${{ env.SKIP_CYTHON_TEST == '0' }}
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
name: ${{ env.CUDA_CORE_ARTIFACT_NAME }}-tests
path: ${{ env.CUDA_CORE_CYTHON_TESTS_DIR }}

- name: Display structure of downloaded cuda.core Cython tests
if: ${{ env.SKIP_CYTHON_TEST == '0' }}
run: |
pwd
ls -lahR $CUDA_CORE_CYTHON_TESTS_DIR

- name: Set up Python ${{ matrix.PY_VER }}
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
Expand Down Expand Up @@ -278,23 +270,14 @@ jobs:
- name: Set up compute-sanitizer
run: setup-sanitizer

- name: Run cuda.pathfinder tests with see_what_works
env:
CUDA_PATHFINDER_TEST_LOAD_NVIDIA_DYNAMIC_LIB_STRICTNESS: see_what_works
run: run-tests pathfinder

- name: Run cuda.bindings tests
if: ${{ env.SKIP_CUDA_BINDINGS_TEST == '0' }}
env:
CUDA_VER: ${{ matrix.CUDA_VER }}
LOCAL_CTK: ${{ matrix.LOCAL_CTK }}
run: run-tests bindings

- name: Run cuda.core tests
env:
CUDA_VER: ${{ matrix.CUDA_VER }}
LOCAL_CTK: ${{ matrix.LOCAL_CTK }}
run: run-tests core


- name: Ensure cuda-python installable
run: |
Expand All @@ -303,17 +286,3 @@ jobs:
else
pip install $(ls cuda_python*.whl)[all]
fi

- name: Install cuda.pathfinder nvidia_wheels_cu12
if: startsWith(matrix.CUDA_VER, '12.')
run: |
pushd cuda_pathfinder
pip install -v .[nvidia_wheels_cu12]
pip freeze
popd

- name: Run cuda.pathfinder tests with all_must_work
if: startsWith(matrix.CUDA_VER, '12.')
env:
CUDA_PATHFINDER_TEST_LOAD_NVIDIA_DYNAMIC_LIB_STRICTNESS: all_must_work
run: run-tests pathfinder
75 changes: 20 additions & 55 deletions .github/workflows/test-wheel-windows.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -117,12 +117,6 @@ jobs:
shell: bash --noprofile --norc -xeuo pipefail {0}
run: ./ci/tools/env-vars test

- name: Download cuda-pathfinder build artifacts
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
name: cuda-pathfinder-wheel
path: ./cuda_pathfinder

- name: Download cuda-python build artifacts
if: ${{ env.SKIP_CUDA_BINDINGS_TEST == '0'}}
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
Expand Down Expand Up @@ -152,6 +146,25 @@ jobs:
}
gh --version

- name: Download cuda-pathfinder build artifacts from main branch
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
$MAIN_BRANCH = "main"
$PATHFINDER_BASENAME = "cuda-pathfinder-wheel"
$runData = gh run list -b $MAIN_BRANCH -L 1 -w "CI" -s completed -R NVIDIA/cuda-python --json databaseId | ConvertFrom-Json
if (-not $runData -or $runData.Length -eq 0 -or -not $runData[0].databaseId -or [string]::IsNullOrEmpty($runData[0].databaseId)) {
Write-Host "LATEST_MAIN_RUN_ID not found!"
exit 1
}
$LATEST_MAIN_RUN_ID = $runData[0].databaseId

gh run download $LATEST_MAIN_RUN_ID -p $PATHFINDER_BASENAME -R NVIDIA/cuda-python
Get-ChildItem -Path $PATHFINDER_BASENAME
New-Item -Path "./cuda_pathfinder" -ItemType Directory -Force
Move-Item -Path "$PATHFINDER_BASENAME/*.whl" -Destination "./cuda_pathfinder/"
Remove-Item -Path $PATHFINDER_BASENAME -Force

- name: Install zstd
# the GPU runner image does not have zstd pre-installed... and it's needed by actions/cache
if: ${{ matrix.LOCAL_CTK == '1' }}
Expand Down Expand Up @@ -215,29 +228,7 @@ jobs:
Get-Location
Get-ChildItem -Recurse -Force $env:CUDA_BINDINGS_CYTHON_TESTS_DIR | Select-Object Mode, LastWriteTime, Length, FullName

- name: Download cuda.core build artifacts
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
name: ${{ env.CUDA_CORE_ARTIFACT_NAME }}
path: ${{ env.CUDA_CORE_ARTIFACTS_DIR }}

- name: Display structure of downloaded cuda.core build artifacts
run: |
Get-Location
Get-ChildItem -Recurse -Force $env:CUDA_CORE_ARTIFACTS_DIR | Select-Object Mode, LastWriteTime, Length, FullName

- name: Download cuda.core Cython tests
if: ${{ env.SKIP_CYTHON_TEST == '0' }}
uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
with:
name: ${{ env.CUDA_CORE_ARTIFACT_NAME }}-tests
path: ${{ env.CUDA_CORE_CYTHON_TESTS_DIR }}

- name: Display structure of downloaded cuda.core Cython tests
if: ${{ env.SKIP_CYTHON_TEST == '0' }}
run: |
Get-Location
Get-ChildItem -Recurse -Force $env:CUDA_CORE_CYTHON_TESTS_DIR | Select-Object Mode, LastWriteTime, Length, FullName

- name: Set up Python ${{ matrix.PY_VER }}
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
Expand All @@ -252,12 +243,6 @@ jobs:
host-platform: ${{ inputs.host-platform }}
cuda-version: ${{ matrix.CUDA_VER }}

- name: Run cuda.pathfinder tests with see_what_works
env:
CUDA_PATHFINDER_TEST_LOAD_NVIDIA_DYNAMIC_LIB_STRICTNESS: see_what_works
shell: bash --noprofile --norc -xeuo pipefail {0}
run: run-tests pathfinder

- name: Run cuda.bindings tests
if: ${{ env.SKIP_CUDA_BINDINGS_TEST == '0' }}
env:
Expand All @@ -266,12 +251,7 @@ jobs:
shell: bash --noprofile --norc -xeuo pipefail {0}
run: run-tests bindings

- name: Run cuda.core tests
env:
CUDA_VER: ${{ matrix.CUDA_VER }}
LOCAL_CTK: ${{ matrix.LOCAL_CTK }}
shell: bash --noprofile --norc -xeuo pipefail {0}
run: run-tests core


- name: Ensure cuda-python installable
run: |
Expand All @@ -281,18 +261,3 @@ jobs:
pip install "$((Get-ChildItem -Filter cuda_python*.whl).FullName)[all]"
}

- name: Install cuda.pathfinder nvidia_wheels_cu12
if: startsWith(matrix.CUDA_VER, '12.')
shell: bash --noprofile --norc -xeuo pipefail {0}
run: |
pushd cuda_pathfinder
pip install -v .[nvidia_wheels_cu12]
pip freeze
popd

- name: Run cuda.pathfinder tests with all_must_work
if: startsWith(matrix.CUDA_VER, '12.')
env:
CUDA_PATHFINDER_TEST_LOAD_NVIDIA_DYNAMIC_LIB_STRICTNESS: all_must_work
shell: bash --noprofile --norc -xeuo pipefail {0}
run: run-tests pathfinder
Loading