Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build wheels for Linux ARM64 #5511

Merged
merged 28 commits into from
Mar 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
1ef8c51
Build wheels for Linux-aarch64
abey79 Mar 14, 2024
750681c
(TEMPORARY) Add linux arm64 build/test on pr
abey79 Mar 14, 2024
de2644b
lint
abey79 Mar 14, 2024
fdcb625
[TEMPORARY] try another way to test the #&%@ build
abey79 Mar 14, 2024
6aebcae
Made pixi.toml possibly compatible with linux-aarch64 + bumped to 0.1…
abey79 Mar 15, 2024
bb9393e
Remove pixi workaround
abey79 Mar 15, 2024
f36275d
Merge branch 'main' into antoine/wheels-linux-aarch64
abey79 Mar 15, 2024
58b98cc
Merge branch 'main' into antoine/wheels-linux-aarch64
abey79 Mar 15, 2024
011b59e
Try with manylinux container?
abey79 Mar 15, 2024
86a0c4b
Revert manylinux container and set compat to 2.35 instead
abey79 Mar 15, 2024
26463d1
fix
abey79 Mar 15, 2024
d3918ba
Merge branch 'main' into antoine/wheels-linux-aarch64
abey79 Mar 15, 2024
6010ac0
spelling
abey79 Mar 15, 2024
9e42bac
pr tests back to linux-x64
abey79 Mar 15, 2024
d6b1083
sort some stuff
abey79 Mar 15, 2024
82b4fbe
Merge branch 'main' into antoine/wheels-linux-aarch64
abey79 Mar 15, 2024
ca13fdc
add rationale for the min test wheel
abey79 Mar 15, 2024
3833e42
Include multi-platform dockerbuild with arm64 support (#5543)
jleibs Mar 15, 2024
69e81df
use our container for all linux-arm64 build
abey79 Mar 16, 2024
2a4092e
[TEMP] switch to arm for PR wheel build&test
abey79 Mar 16, 2024
26c27ca
Revert "[TEMP] switch to arm for PR wheel build&test"
abey79 Mar 16, 2024
f8fcb02
always use pixi to build the web viewer (needed for wasm-opt)
abey79 Mar 16, 2024
e718959
Merge branch 'main' into antoine/wheels-linux-aarch64
abey79 Mar 16, 2024
c1b3397
Merge branch 'main' into antoine/wheels-linux-aarch64
abey79 Mar 18, 2024
838990f
Misc minor change to make the reusable_build_* files more similar
abey79 Mar 18, 2024
cff5b0c
Try remove double google auth in CLI build
abey79 Mar 18, 2024
0724ba2
Remove double GCS auth and clarify that setup-rust also auth to gcs
abey79 Mar 18, 2024
c029b41
Merge branch 'main' into antoine/wheels-linux-aarch64
abey79 Mar 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/auto_release_crates.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ jobs:

- uses: prefix-dev/setup-pixi@v0.4.1
with:
pixi-version: v0.13.0
pixi-version: v0.16.1

- name: Install dependencies
shell: bash
Expand Down
16 changes: 8 additions & 8 deletions .github/workflows/contrib_checks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ jobs:
name: Check if running codegen would produce any changes
runs-on: ubuntu-latest-16-cores
container:
image: rerunio/ci_docker:0.11.0
image: rerunio/ci_docker:0.12.0
steps:
# Note: We explicitly don't override `ref` here. We need to see if changes would be made
# in a context where we have merged with main. Otherwise we might miss changes such as one
Expand All @@ -101,7 +101,7 @@ jobs:

- uses: prefix-dev/setup-pixi@v0.4.1
with:
pixi-version: v0.13.0
pixi-version: v0.16.1

- name: Codegen check
shell: bash
Expand All @@ -112,7 +112,7 @@ jobs:
name: Rust lints (fmt, check, cranky, tests, doc)
runs-on: ubuntu-latest-16-cores
container:
image: rerunio/ci_docker:0.11.0
image: rerunio/ci_docker:0.12.0
steps:
- uses: actions/checkout@v4

Expand Down Expand Up @@ -183,7 +183,7 @@ jobs:
name: Check Rust web build (wasm32 + wasm-bindgen)
runs-on: ubuntu-latest-16-cores
container:
image: rerunio/ci_docker:0.11.0
image: rerunio/ci_docker:0.12.0
steps:
- uses: actions/checkout@v4

Expand Down Expand Up @@ -212,7 +212,7 @@ jobs:

- uses: prefix-dev/setup-pixi@v0.4.1
with:
pixi-version: v0.13.0
pixi-version: v0.16.1

- run: pixi run lint-taplo
shell: bash
Expand Down Expand Up @@ -268,7 +268,7 @@ jobs:
name: Cargo Deny
runs-on: ubuntu-latest
container:
image: rerunio/ci_docker:0.11.0
image: rerunio/ci_docker:0.12.0
steps:
- uses: actions/checkout@v4

Expand All @@ -294,13 +294,13 @@ jobs:
name: C++ tests
runs-on: ubuntu-latest
container:
image: rerunio/ci_docker:0.11.0
image: rerunio/ci_docker:0.12.0
steps:
- uses: actions/checkout@v4

- uses: prefix-dev/setup-pixi@v0.4.1
with:
pixi-version: v0.13.0
pixi-version: v0.16.1

# TODO(emilk): make this work somehow. Right now this just results in
# > Compiler: GNU 12.3.0 (/__w/rerun/rerun/.pixi/env/bin/x86_64-conda-linux-gnu-c++)
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/contrib_rerun_py.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,13 +36,13 @@ jobs:
name: Build Wheels
runs-on: ubuntu-latest-16-cores
container:
image: rerunio/ci_docker:0.11.0
image: rerunio/ci_docker:0.12.0
steps:
- uses: actions/checkout@v4

- uses: prefix-dev/setup-pixi@v0.4.1
with:
pixi-version: v0.13.0
pixi-version: v0.16.1

# These should already be in the docker container, but run for good measure. A no-op install
# should be fast, and this way things don't break if we add new packages without rebuilding
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,8 @@ jobs:
uses: ./.github/workflows/reusable_build_and_upload_wheels.yml
with:
CONCURRENCY: nightly-linux
PLATFORM: linux
WHEEL_ARTIFACT_NAME: linux-wheel
PLATFORM: linux-x64
WHEEL_ARTIFACT_NAME: linux-x64-wheel
MODE: "pr"
secrets: inherit

Expand All @@ -65,7 +65,7 @@ jobs:
with:
CONCURRENCY: nightly
CHANNEL: nightly
WHEEL_ARTIFACT_NAME: linux-wheel
WHEEL_ARTIFACT_NAME: linux-x64-wheel
secrets: inherit

upload-examples:
Expand Down
14 changes: 8 additions & 6 deletions .github/workflows/on_pull_request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -76,15 +76,17 @@ jobs:
FULL: "true"
secrets: inherit

# Build and test a single wheel to limit CI cost. We use linux-x64 because it's fast. linux-arm64 would also be a good
# choice, but reusable_test_wheels.yml is broken for that target (https://github.com/rerun-io/rerun/issues/5525)
min-wheel-build:
name: "Minimum Wheel Build"
if: github.event.pull_request.head.repo.owner.login == 'rerun-io'
uses: ./.github/workflows/reusable_build_and_upload_wheels.yml
with:
CONCURRENCY: pr-${{ github.event.pull_request.number }}
MODE: "pr"
PLATFORM: linux
WHEEL_ARTIFACT_NAME: "linux-wheel-fast"
PLATFORM: linux-x64
WHEEL_ARTIFACT_NAME: "linux-x64-wheel-fast"
secrets: inherit

min-wheel-test:
Expand All @@ -94,8 +96,8 @@ jobs:
uses: ./.github/workflows/reusable_test_wheels.yml
with:
CONCURRENCY: pr-${{ github.event.pull_request.number }}
PLATFORM: linux
WHEEL_ARTIFACT_NAME: "linux-wheel-fast"
abey79 marked this conversation as resolved.
Show resolved Hide resolved
PLATFORM: linux-x64
WHEEL_ARTIFACT_NAME: "linux-x64-wheel-fast"
secrets: inherit

build-js:
Expand Down Expand Up @@ -133,7 +135,7 @@ jobs:
with:
CONCURRENCY: pr-${{ github.event.pull_request.number }}
CHANNEL: main
WHEEL_ARTIFACT_NAME: linux-wheel-fast
WHEEL_ARTIFACT_NAME: linux-x64-wheel-fast
secrets: inherit

track-sizes:
Expand Down Expand Up @@ -163,7 +165,7 @@ jobs:
uses: ./.github/workflows/reusable_run_notebook.yml
with:
CONCURRENCY: pr-${{ github.event.pull_request.number }}
WHEEL_ARTIFACT_NAME: linux-wheel-fast
WHEEL_ARTIFACT_NAME: linux-x64-wheel-fast
secrets: inherit

save-pr-summary:
Expand Down
106 changes: 59 additions & 47 deletions .github/workflows/on_push_main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -209,78 +209,90 @@ jobs:

# -----------------------------------------------------------------------------------

build-wheel-linux:
build-wheel-linux-arm64:
needs: [checks]
name: "Linux: Build & Upload Wheels"
name: "Linux-arm64: Build & Upload Wheels"
uses: ./.github/workflows/reusable_build_and_upload_wheels.yml
with:
CONCURRENCY: push-linux-${{ github.ref_name }}
PLATFORM: linux
WHEEL_ARTIFACT_NAME: linux-wheel
CONCURRENCY: push-linux-arm64-${{ github.ref_name }}
PLATFORM: linux-arm64
WHEEL_ARTIFACT_NAME: linux-arm64-wheel
MODE: "pypi"
secrets: inherit

build-wheel-linux-x64:
needs: [checks]
name: "Linux-x64: Build & Upload Wheels"
uses: ./.github/workflows/reusable_build_and_upload_wheels.yml
with:
CONCURRENCY: push-linux-x64-${{ github.ref_name }}
PLATFORM: linux-x64
WHEEL_ARTIFACT_NAME: linux-x64-wheel
MODE: "pypi"
secrets: inherit

build-wheel-windows:
build-wheel-windows-x64:
needs: [checks]
name: "Windows: Build & Upload Wheels"
name: "Windows-x64: Build & Upload Wheels"
uses: ./.github/workflows/reusable_build_and_upload_wheels.yml
with:
CONCURRENCY: push-windows-${{ github.ref_name }}
PLATFORM: windows
WHEEL_ARTIFACT_NAME: windows-wheel
CONCURRENCY: push-windows-x64-${{ github.ref_name }}
PLATFORM: windows-x64
WHEEL_ARTIFACT_NAME: windows-x64-wheel
MODE: "pypi"
secrets: inherit

build-wheel-macos-arm:
build-wheel-macos-arm64:
needs: [checks]
name: "Macos-Arm: Build & Upload Wheels"
name: "Macos-arm64: Build & Upload Wheels"
uses: ./.github/workflows/reusable_build_and_upload_wheels.yml
with:
CONCURRENCY: push-macos-arm-${{ github.ref_name }}
PLATFORM: macos-arm
WHEEL_ARTIFACT_NAME: macos-arm-wheel
CONCURRENCY: push-macos-arm64-${{ github.ref_name }}
PLATFORM: macos-arm64
WHEEL_ARTIFACT_NAME: macos-arm64-wheel
MODE: "pypi"
secrets: inherit

build-wheel-macos-intel:
build-wheel-macos-x64:
needs: [checks]
name: "Macos-Intel: Build & Upload Wheels"
name: "Macos-x64: Build & Upload Wheels"
uses: ./.github/workflows/reusable_build_and_upload_wheels.yml
with:
CONCURRENCY: push-macos-intel-${{ github.ref_name }}
PLATFORM: macos-intel
WHEEL_ARTIFACT_NAME: "macos-intel-wheel"
CONCURRENCY: push-macos-x64-${{ github.ref_name }}
PLATFORM: macos-x64
WHEEL_ARTIFACT_NAME: "macos-x64-wheel"
MODE: "pypi"
secrets: inherit

test-wheel-linux:
needs: [checks, build-wheel-linux]
name: "Linux: Test Wheels"
test-wheel-linux-x64:
needs: [checks, build-wheel-linux-x64]
name: "Linux-x64: Test Wheels"
uses: ./.github/workflows/reusable_test_wheels.yml
with:
CONCURRENCY: push-linux-${{ github.ref_name }}
PLATFORM: linux
WHEEL_ARTIFACT_NAME: linux-wheel
CONCURRENCY: push-linux-x64-${{ github.ref_name }}
PLATFORM: linux-x64
WHEEL_ARTIFACT_NAME: linux-x64-wheel
secrets: inherit

test-wheel-windows:
needs: [checks, build-wheel-windows]
name: "Windows: Test Wheels"
test-wheel-windows-x64:
needs: [checks, build-wheel-windows-x64]
name: "Windows-x64: Test Wheels"
uses: ./.github/workflows/reusable_test_wheels.yml
with:
CONCURRENCY: push-windows-${{ github.ref_name }}
PLATFORM: windows
WHEEL_ARTIFACT_NAME: windows-wheel
CONCURRENCY: push-windows-x64-${{ github.ref_name }}
PLATFORM: windows-x64
WHEEL_ARTIFACT_NAME: windows-x64-wheel
secrets: inherit

generate-pip-index:
name: "Generate Pip Index"
needs:
[
build-wheel-linux,
build-wheel-windows,
build-wheel-macos-arm,
build-wheel-macos-intel,
build-wheel-linux-arm64,
build-wheel-linux-x64,
build-wheel-macos-arm64,
build-wheel-macos-x64,
build-wheel-windows-x64,
]
uses: ./.github/workflows/reusable_pip_index.yml
with:
Expand All @@ -291,10 +303,10 @@ jobs:
name: "Bundle and upload rerun_cpp_sdk.zip"
needs:
[
build-rerun_c-and-upload-linux-x64,
build-rerun_c-and-upload-linux-arm64,
build-rerun_c-and-upload-macos-x64,
build-rerun_c-and-upload-linux-x64,
build-rerun_c-and-upload-macos-arm64,
build-rerun_c-and-upload-macos-x64,
build-rerun_c-and-upload-windows-x64,
]
uses: ./.github/workflows/reusable_bundle_and_upload_rerun_cpp.yml
Expand All @@ -307,19 +319,19 @@ jobs:
cancel-in-progress: true
needs:
[
upload-web,
generate-pip-index,
build-rerun_c-and-upload-linux-x64,
build-rerun_c-and-upload-linux-arm64,
build-rerun_c-and-upload-macos-x64,
build-rerun_c-and-upload-macos-arm64,
build-rerun_c-and-upload-windows-x64,
build-rerun-cli-and-upload-linux-x64,
build-rerun-cli-and-upload-linux-arm64,
build-rerun-cli-and-upload-macos-x64,
build-rerun-cli-and-upload-linux-x64,
build-rerun-cli-and-upload-macos-arm64,
build-rerun-cli-and-upload-macos-x64,
build-rerun-cli-and-upload-windows-x64,
build-rerun_c-and-upload-linux-arm64,
build-rerun_c-and-upload-linux-x64,
build-rerun_c-and-upload-macos-arm64,
build-rerun_c-and-upload-macos-x64,
build-rerun_c-and-upload-windows-x64,
bundle-and-upload-rerun_cpp,
generate-pip-index,
upload-web,
]
runs-on: "ubuntu-latest"
steps:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ jobs:

- uses: prefix-dev/setup-pixi@v0.4.1
with:
pixi-version: v0.13.0
pixi-version: v0.16.1

- name: Update crate versions
id: versioning
Expand Down Expand Up @@ -391,7 +391,7 @@ jobs:

- uses: prefix-dev/setup-pixi@v0.4.1
with:
pixi-version: v0.13.0
pixi-version: v0.16.1

- name: Install dependencies
shell: bash
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/reusable_bench.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ jobs:
runs-on: ubuntu-latest-16-cores

container:
image: rerunio/ci_docker:0.11.0
image: rerunio/ci_docker:0.12.0

steps:
- uses: actions/checkout@v4
Expand Down