Add m1 support to conda build scripts

[seibert]

Minor adjustments needed to address differences in clang version and SDK version for osx-arm64.

[seibert]

There's no CI for this platform yet, but I tested this on my local M1.

[esc]

I guess the way to test this will be:

• build an llvmdev build 5 from this PR using an M1
• setup an M1 worker on the anaconda internal build-farm
• run this PR through the build-farm and check if the llvmlite build on M1 was fine

Finally: repeat for Numba and check if the M1 llvmdev&llvmlite combo is good for Numba too!

[esc]

Update: I tried to build this with aarch64 based miniforge but the binaries produced llvm-tblgen were so bad that I wasn't even able to debug them with lldb:

(build-n-upload) super@colo-osxarm-01 build % lldb bin/llvm-tblgen
(lldb) target create "bin/llvm-tblgen"
zsh: killed     lldb bin/llvm-tblgen

[esc]

I should note that, using the miniconda for M1 distribution available here:

https://docs.conda.io/en/latest/miniconda.html

seems like the build is now going through.

[esc]

Ok, I managed to build this, next step will be to setup an M1 worker on our internal build farm and do a buildout there.

[esc]

I built a first artifact here:

https://anaconda.org/numba/llvmdev/11.1.0/download/osx-arm64/llvmdev-11.1.0-h70deae4_5.tar.bz2

[esc]

I can confirm I am able to build llvmdev for the correct architecture (m1) now.

[esc]

It took some trickeries with the code recipe and the specification of the correct llvmdev (M1 needs build 5). But apparently I made some llvmlite 0.39dev0 versions for osx-arm64 for Pythons 3.8 3.9 and 3.10 (no 3.7, no M1 Python for that): if anyone would like to test these, they are here:

https://anaconda.org/numba/llvmlite/files?version=0.39.0dev0

If anyone has spare cycles, it would be great if you could test these somehow. Even a plain installation check on an M1 would be useful. These packages are brand new and experimental and I fully anticipate there to be issues. So thank you in advance for any of your time that you can contribute.

@seibert the following modifications had to be made to this patch, in order for llvmlite to build successfully:

#821

[esc]

OK, the packages are available from the numba/label/ci channel, so on an M1 with an appropriate miniconda, you can do:

conda create -c numba/label/ci -n llvmlite-test python=3.10 llvmlite=0.39

And then test with:

python -m llvmlite.tests

[esc]

@seibert I think we should cherry-pick my changes and then merge this PR. Having it on main will make it easier to build-out Numba packages too.

[esc]

@seibert as per our OOB conversation, I have pushed: dbe6ea0 -- which is the patch to select the correct version of llvmdev (M1 needs 5) for llvmlite.

Other than that, I think this is ready to review, there were a number of internal patches to the build farm, but we have taken care of those. If we can merge this to main, we will get a fresh set of llvmlite 0.39dev0 on numba/label/dev for folks to try and wheels would probably be next.

I think that I have provided everything for a review, files and suggestions etc... If anything is missing let me know and I'll see what I can do.

[esc]

looks like Numba was build-able but had some issues itself: numba/numba#7822

[esc]

I would hold off merging this until we are certain that no bugs in llvm/llvmlite need to be patched.

[programmingkidx]

I ran the test for llvm in Conda and it passed 😁

Here is what I saw:

$ conda activate llvmlite-test
(llvmlite-test) $ python -m llvmlite.tests
..s......................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 329 tests in 0.662s

OK (skipped=1)

My info:
M1 MacBook Air
Mac OS 11.1

[esc]

I ran the test for llvm in Conda and it passed 😁

Here is what I saw:

$ conda activate llvmlite-test
(llvmlite-test) $ python -m llvmlite.tests
..s......................................................................................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 329 tests in 0.662s

OK (skipped=1)

My info: M1 MacBook Air Mac OS 11.1

Thank you for confirming!

[programmingkidx]

Please let us know if there is anything else we could do to help.

[esc]

Please let us know if there is anything else we could do to help.

Thank you for offering to help. We are currently blocked by: numba/numba#7822 as we need to get Numba working on the M1 too.

[stuartarchibald]

Thanks for the patch, few minor comments but looks good otherwise. Obviously this needs testing on the build farm etc!

[stuartarchibald]

Does OSX-arm64 not need xar?

[seibert]

The package doesn't exist in the package repository for this platform. I'm not entirely certain what it is for, and the tests seem to pass without it.

[stuartarchibald]

Seems like it was added a while back as a workaround: #348 (specifically: 73cf559) it may not be applicable any more.

[stuartarchibald]

I guess the question is... does llvm-objdump work on M1, as from 73cf559 it seems like that was what triggered adding it. I suppose though that this is also somewhat moot if the package doesn't exist and further that this is a first build out on this architecture!

[programmingkidx]

On Mar 15, 2022, at 10:04 AM, stuartarchibald ***@***.***> wrote: @stuartarchibald commented on this pull request. In conda-recipes/llvmdev/meta.yaml: > @@ -54,7 +54,7 @@ requirements: host: # needed for llc at runtime - zlib # [not win] - - xar # [osx] + - xar # [osx and x86_64] I guess the question is... does llvm-objdump work on M1, as from 73cf559 it seems like that was what triggered adding it. I suppose though that this is also somewhat moot if the package doesn't exist and further that this is a first build out on this architecture!

I have objdump on my M1 MacBook Air. Just tell me what you want to test I could test it out for you. I would suggest send me the options you want to test and any file you want me to test it on.

[esc]

@stuartarchibald @seibert thank you for the review and updates. I am running this through the farm as Build llvmlite_137.

I also checked llvm-objdump and it appears to be fine.

[esc]

@stuartarchibald @seibert thank you for the review and updates. I am running this through the farm as Build llvmlite_137.

This passed fine again. I am marking this as RTM now.

[stuartarchibald]

Thanks for the patch, testing and fixes! Looks good.

[stuartarchibald]

I also checked llvm-objdump and it appears to be fine.

Many thanks @esc .

[esc]

Our build has completed, the files are now available in:

https://anaconda.org/numba/llvmlite/files

Development versions of llvmlite for M1 can now be installed on M1 too using:

conda install -c numba/label/dev llvmlite