[ci] build aarch64 wheel builds on a real aarch64 machine #6843
Conversation
.github/workflows/python_package.yml
Outdated
| steps: | ||
| - name: Checkout repository | ||
| uses: actions/checkout@v4 | ||
| uses: actions/checkout@v3 |
There was a problem hiding this comment.
🙃 first challenge... actions/checkout@v4 requires GLIBC 2.28, which is too new for manylinux2014.
/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libc.so.6: version `GLIBC_2.25' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /__e/node20/bin/node)
/__e/node20/bin/node: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.[21](https://github.com/microsoft/LightGBM/actions/runs/13579891865/job/37963851509#step:3:22)' not found (required by /__e/node20/bin/node)
Trying the older version to preserve manylinux_2014 support a bit longer.
There was a problem hiding this comment.
Got around this by using docker run in a run: step, instead of passing the image URI to container:.
That way, actions/checkout runs on the runner which is Ubuntu 24.04, with new-enough GLIBC.
| export PATH=\$CONDA/bin:/opt/rh/llvm-toolset-7.0/root/usr/bin:\$PATH | ||
| export LD_LIBRARY_PATH=/opt/rh/llvm-toolset-7.0/root/usr/lib64:\$LD_LIBRARY_PATH |
There was a problem hiding this comment.
I intentionally did not preserve this in the workflow. It shouldn't be necessary.
docker run \
--rm \
-it lightgbm/vsts-agent:manylinux2014_aarch64 \
bash
echo $PATH
# /opt/rh/llvm-toolset-7.0/root/usr/bin:/opt/rh/devtoolset-10/root/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bi
echo $LD_LIBRARY_PATH
# /opt/rh/llvm-toolset-7.0/root/usr/lib64:/opt/rh/devtoolset-10/root/usr/lib64:/opt/rh/devtoolset-10/root/usr/lib:/opt/rh/devtoolset-10/root/usr/lib64/dyninst:/opt/rh/devtoolset-10/root/usr/lib/dyninst:/usr/local/lib64There was a problem hiding this comment.
I can see that the correct compiler (gcc) is being used:
-- The C compiler identification is GNU 10.2.1
-- The CXX compiler identification is GNU 10.2.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/rh/devtoolset-10/root/usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/rh/devtoolset-10/root/usr/bin/g++ - skipped
jameslamb
changed the title
jameslamb
requested review from
guolinke,
shiyu1994,
jmoralez,
borchero and
StrikerRUS
as code owners
| pytest.mark.skipif( | ||
| getenv("LGBM_SKIP_DASK_TESTS", "") == "true", | ||
| reason="Skipping lightgbm.dask tests (found env variable LGBM_SKIP_DASK_TESTS=true)", | ||
| ), |
There was a problem hiding this comment.
This isn't necessary any more: https://github.com/microsoft/LightGBM/pull/6677/files#r1974827615
|
@StrikerRUS the next time you have a bit of time to work on LightGBM, could you please review this PR? I've been seeing failures in the QEMU job (#6874) and would love to just merge this and not have to investigate them 😅 |
|
Does anyone have time to review this this week? I think it's valuable on its own, and more importantly this + #6872 would help unblock CI and get development flowing here again. |
|
Thank you! I'm going to merge this to help unblock CI. I'd be happy to make any changes in a follow-up if others find time to review later and have concerns. |
There was a problem hiding this comment.
I'm so sorry for the late review! Everything looks good to me! Great job, @jameslamb !
I just added the following line to the v4.6.0 release to not forget this in the future release
[ ] Manually attach aarch64 Linux wheel to the release
|
Thank you so much! Good idea adding that to the release checklist. |
fixes #6788
Proposes building manylinux aarch64 wheels on a real aarch64 machine, taking advantage of the new GitHub-hosted Arm runners.
Those wheels are currently built using emulation, on an x86_64 machine. That process, and especially testing in that environment, is super slow... that job takes around 60-90 minutes per run.
Making this change should speed up and simplify LightGBM's CI.... the new job introduced here builds the same wheel, runs MORE tests, and only takes about 15 minutes 😁
Other small changes in this PR:
pydistcheckfloor to the latest version (0.9.0)AZURElocal variable in.ci/setup.sh