-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add GitHub action to build wheels with cibuildwheel #8
Conversation
Thank you! I never even tried for
For One way to further reduce the time needed is to not run for more than 1 python version per platform. Both |
Thanks for your feedback. I tried to put the However it seems to be a new problem after I merged the master branch into my branch, see e.g https://github.com/finsberg/cppyy-backend/actions/runs/4101851124/jobs/7074102266#step:8:2624 Regarding Also we are currently not running any repair command for the windows wheels. One option would be to use https://pypi.org/project/delvewheel/ Regarding python versions, I am actually only building one python version per combination of OS and architecture. |
The new problem is most likely my switch to |
c7dc7d4
to
230004e
Compare
57f63ba
to
40a7af6
Compare
@wlav I have now finally managed to get this to work. I am now building the aarch64 wheel on CircleCI and the rest of the wheels on GitHub Actions. You can check out this run and see that all the wheels are successfully uploaded as artifacts. The way it works is that you need to first trigger a pipeline on CircleCI from GitHub actions and then make sure to get the job number using the CircleCI Rest API. This will start the pipeline on CircleCI. At the same time we start building the wheels on GitHub action and once that is done we assume that the pipeline on CircleCI is also done. Then we create a new job that fetches the wheel from Circle CI I have created a separate python script both for triggering the pipeline and for fetching the wheel. What remains is to add a new job for uploading the wheels to pypi, but for this we could use https://github.com/marketplace/actions/pypi-publish. You can also check out this pipeline as an example. Also, in order to trigger the pipeline you need to set up a token with admin rights (the different scopes are "Status", "Read only" and "Admin" and we need to do more than "Read"), and save this as a repository secret called You also want to change the default argument Final note is that I have turned off the |
I'm slowly working through the details, thanks for all the work! For auditwheel, if I try it interactively, I get |
@wlav The reason it is failing here: https://github.com/wlav/cppyy-backend/actions/runs/4422767181/jobs/7757608629 is becasue you need to set a secret with the name |
I've set the CIRCLE_API_TOKEN several times (regenerated, set again), but it never shows up in the job: it always shows an empty wheel_path b/c the token value is empty: I've completely run out of ideas how else to fix this ... do you know anything else that it could be? |
That is strange. Did make sure to set up the project on CIRCLECI? In my dashboard it looks as follows: Also, you need to use the Personal API token , i.e under User Settings, not Project Settings. I tried to run it using my account here: https://github.com/finsberg/cppyy-backend/actions/runs/6146025185/job/16674588355 and it is still working for me |
Thanks for the reference! Turns out the token is okay. What failed is that this After that, the copy still fails, still because (I still need to test all the produced wheels. The Mac ARM one wasn't usable b/c the full path to the compiler was embedded, which will require some fixing; others may suffer from the same.) |
I belive it is still the same problem. Changing
which might reveal some more information. And it might be useful to print out Line 72 in 8c774b1
|
The problem appears to be simpler, but I'm no closer to a solution (I tried playing with quotes, but saw no difference). The "Get wheel from CircleCI artifact" step, although marked as successful actually fails with:
so it's $JOB_NUMBER that is somehow the empty string. |
Just an update to show I'm not forgetting about this. :) Setting of the job number fails way earlier b/c the JSON result isn't properly formatted:
Still debugging ... I'm trying out the wheels at the same time, but so far, all of them are broken. The Linux ones are manylinux1, which can't be used as is b/c of an ABI change starting gcc5 (and manylinux1 isn't supported anymore anyway). I've yet to be able to convince the system to run manylinux2014 instead. The Mac ARM one for some reason builds binaries for the wrong architecture (x86_64). I have yet to test the Windows ones. |
The MacOS thing at least is understood (and similar issues are probably on some of the other builds): the runner is an x86_64 machine that cross compiles. Although cibuildwheel will setup the proper configuration, Cling is in turn build using cmake which re-derives the toolchain and picks the host as its target, ie. it selects x86_64. |
Doh, I finally figured out the job number issue:
which only exists in your repo. The error message is not in proper JSON format, after which a Python exception is raised and the job number data is empty. |
I'm going to punt on this one. It's been very useful to catch some compiler errors on different platforms early and to resolve the issues some people experienced on Mac M1/2. However, overall it's a net negative time sink as these CI platforms/builds have defaults that simply don't work for me. I've removed the PyPy builds as they're not necessary. The Linux builds default to the old pre-C++11 ABI and with no amount of work have I been able to change their mind. The MacOS ARM build is cross-compiled, which doesn't work b/c some of the final tools are used in the build process, i.e. they won't run if cross-compiled and aren't useful if not. The Windows builds fail b/c somewhere there's a compiler setting to prevent exporting symbols, so cppyy-backend won't link. Finally, I still can't get the circle-ci job to trigger and I suspect it will suffer from the same ABI problem regardless. (I haven't tried the Mac native build yet.) Thanks for the effort, though! It would have been nice, but contrary to other Python extension modules (which hide all symbols by design), cppyy needs external (C++11) linkage and it's clearly to difficult to convince these build systems otherwise. |
In this PR I have created a GitHub action to build wheels for
cling
. Currently the following OS + Architectures build successfullyThe following combinations failed due to timeout after 6 hours (this is apparently the limit https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits).
The reason these are so slow it because they are using emulation through QEMU. Another approach would be to use a native aarch64 runner (but I am not sure if this is available through GitHub Actions).
The follwing combinations failed due to some other reason
You can see the full output here: https://github.com/finsberg/cppyy-backend/actions/runs/4084705242
A few things worth noting
python create_src_directory.py
(specified in thepyproject.toml
), and in some cases this failed. I didn't really investigate this, but as a temporary solution I commented out the call tosys.exit
in order for the workflow to continue.auditwheel repair
to work, see https://github.com/finsberg/cppyy-backend/actions/runs/4048282754/jobs/6963318442#step:8:4430 for more info. As a result, I did disableauditwheel
, by settingtool.cibuildwheel.linux.repair-wheel-command = ""
inpyproject.toml
.python -c "import cppyy_backend"
. We might want to add a proper test to make sure some core functionality is working as expected.pyzmq
(https://github.com/zeromq/pyzmq/blob/main/.github/workflows/wheels.yml) as a guide. We could potentially get some inspiration from their setup scripts in order to better streamline this workflow.clingwrapper
should be very similar.This PR is related to wlav/cppyy#123