Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ci] build aarch64 wheel builds on a real aarch64 machine #6843

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Prev Previous commit
Next Next commit
restore all CI
  • Loading branch information
jameslamb committed Feb 28, 2025
commit 5e9ef3bba6171b57790bcb7147e2792ae91b686f
42 changes: 42 additions & 0 deletions .appveyor.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
version: 4.6.0.99.{build}

image: Visual Studio 2015
platform: x64
configuration:
- '3.8'

# only build on 'master' and pull requests targeting it
branches:
only:
- master

environment:
matrix:
- COMPILER: MSVC
TASK: python
- COMPILER: MINGW
TASK: python

clone_depth: 5

install:
- git submodule update --init --recursive # get `external_libs` folder
- set PATH=C:\mingw-w64\x86_64-8.1.0-posix-seh-rt_v6-rev0\mingw64\bin;%PATH%
- set PYTHON_VERSION=%CONFIGURATION%
- ps: |
$env:ALLOW_SKIP_ARROW_TESTS = "1"
$env:APPVEYOR = "true"
$env:CMAKE_BUILD_PARALLEL_LEVEL = 4
$env:MINICONDA = "C:\Miniconda3-x64"
$env:PATH = "$env:MINICONDA;$env:MINICONDA\Scripts;$env:PATH"
$env:BUILD_SOURCESDIRECTORY = "$env:APPVEYOR_BUILD_FOLDER"

build: false

test_script:
- conda config --remove channels defaults
- conda config --add channels nodefaults
- conda config --add channels conda-forge
- conda config --set channel_priority strict
- conda init powershell
- powershell.exe -ExecutionPolicy Bypass -File %APPVEYOR_BUILD_FOLDER%\.ci\test-windows.ps1
140 changes: 140 additions & 0 deletions .github/workflows/cuda.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,140 @@
name: CUDA Version

on:
push:
branches:
- master
pull_request:
branches:
- master
# Run manually by clicking a button in the UI
workflow_dispatch:
inputs:
restart_docker:
description: 'Restart nvidia-docker on the runner before building?'
required: true
type: boolean
default: false

# automatically cancel in-progress builds if another commit is pushed
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
# Optionally reinstall + restart docker on the runner before building.
# This is safe as long as only 1 of these jobs runs at a time.
restart-docker:
name: set up docker
runs-on: [self-hosted, linux]
timeout-minutes: 30
steps:
- name: Setup or update software on host machine
if: ${{ inputs.restart_docker }}
run: |
# install core packages
sudo apt-get update
sudo apt-get install --no-install-recommends -y \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
lsb-release \
software-properties-common
# set up nvidia-docker
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository -y \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
curl -sL https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -sL \
https://nvidia.github.io/nvidia-docker/$(. /etc/os-release;echo $ID$VERSION_ID)/nvidia-docker.list \
| sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update
sudo apt-get install --no-install-recommends -y \
containerd.io \
docker-ce \
docker-ce-cli \
nvidia-docker2
sudo chmod a+rw /var/run/docker.sock
sudo systemctl restart docker
- name: mark job successful
run: |
exit 0
test:
# yamllint disable-line rule:line-length
name: ${{ matrix.task }} ${{ matrix.cuda_version }} ${{ matrix.method }} (${{ matrix.linux_version }}, ${{ matrix.compiler }}, Python ${{ matrix.python_version }})
runs-on: [self-hosted, linux]
needs: [restart-docker]
container:
image: nvcr.io/nvidia/cuda:${{ matrix.cuda_version }}-devel-${{ matrix.linux_version }}
env:
CMAKE_BUILD_PARALLEL_LEVEL: 4
COMPILER: ${{ matrix.compiler }}
CONDA: /tmp/miniforge
DEBIAN_FRONTEND: noninteractive
METHOD: ${{ matrix.method }}
OS_NAME: linux
PYTHON_VERSION: ${{ matrix.python_version }}
TASK: ${{ matrix.task }}
SKBUILD_STRICT_CONFIG: true
options: --gpus all
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
include:
- method: wheel
compiler: gcc
python_version: "3.11"
cuda_version: "12.8.0"
linux_version: "ubuntu22.04"
task: cuda
- method: source
compiler: gcc
python_version: "3.13"
cuda_version: "12.2.2"
linux_version: "ubuntu22.04"
task: cuda
- method: pip
compiler: clang
python_version: "3.12"
cuda_version: "11.8.0"
linux_version: "ubuntu20.04"
task: cuda
steps:
- name: Install latest git and sudo
run: |
apt-get update
apt-get install --no-install-recommends -y \
ca-certificates \
software-properties-common
add-apt-repository ppa:git-core/ppa -y
apt-get update
apt-get install --no-install-recommends -y \
git \
sudo
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 5
submodules: true
- name: Setup and run tests
run: |
export BUILD_DIRECTORY="$GITHUB_WORKSPACE"
export PATH=$CONDA/bin:$PATH

# check GPU usage
nvidia-smi

# build and test
$GITHUB_WORKSPACE/.ci/setup.sh
$GITHUB_WORKSPACE/.ci/test.sh
all-cuda-jobs-successful:
if: always()
runs-on: ubuntu-latest
needs: [test]
steps:
- name: Note that all tests succeeded
uses: re-actors/alls-green@v1.2.2
with:
jobs: ${{ toJSON(needs) }}
Loading
Oops, something went wrong.
Loading
Oops, something went wrong.