Build native libraries for Mac OS X and Windows on aarch64 machines #135
Conversation
|
Any change to have this PR merged ? |
|
Hey there, sorry, saw this just yet. I'll have a look over the next few days. Thanks for the PR! |
|
If it is of any support, I am using a snapshot version of the branch on my computer since one month without any problems |
|
Any news on this PR ? |
|
@AlbanSeurat you said you're using this on your machine, is it for MacOS or Windows? |
|
MacOs Ventura 13 with m2max using a snapshot to be sure to have the arm build :) |
|
Hey Alban, alright, thanks for the info. Slowly getting back to having to look into this. I'll have to make sure it also runs on all the other systems, and often I find out that I need to bump versions and so on, so it takes a bit of time, but thanks for your patience! Looking forward to see how well this performs on Apple Silicon! Given how old the project is, this has been quite a ride, every new processor generation led to a big bump in performance for jblas :) |
|
OK, the config part for windows looks OK, but I've had a hard time compiling the dynamic libraries for Mac OS, I need to look more into this. One last thing, as a measure of precaution, I'm always a bit hesitant to merge binaries from external sources. @jiaminglu can I have a word with you to check a few things? |
|
Meanwhile, @AlbanSeurat, I just pushed aarch64 versions to main, if you want, can you check it out, build the jar and see whether it works for you? |
|
[ERROR] Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0 s <<< FAILURE! - in org.jblas.TestBlasFloat [ERROR] Failures: Values from the assert B.data =[0.482044, 0.707107, 0.517333; -0.731620, 0.000000, 0.681713; 0.482044, -0.707107, 0.517333] rather than : For some reason, this doesn't pass. Running the same tests from |
|
Thank you! Can you mvn package -Dmaven.test.skip and then print what java -jar target/jblas… and so on prints. There should be a message what linking error or the like happened. |
|
Here is the result of java -jar target/jblas-1.2.6-SNAPSHOT.jar Running sanity benchmarks. checking vector addition... ok Each benchmark will take about 5 seconds... Running benchmark "Java matrix multiplication, double precision". Running benchmark "Java matrix multiplication, single precision". Running benchmark "native matrix multiplication, double precision". Running benchmark "native matrix multiplication, single precision". |
|
Interesting, that seems to work 🤔 OK, I‘ll have to look into it further. Thanks for checking, Alban! |
|
If it is of any help, md5sum from md5sum src/main/resources/lib/static/Mac\ OS\ X/aarch64/libjblas_arch_flavor.jnilib md5sum from your build : |
|
Yeah, file sizes are also massively different. Weird. |
|
Is it going well? It seems you’ve successfully built your binary and the size is different. But I believe if it passes your test it will be ok.
Sent with Spark
2023年3月25日 +0800 06:00 Mikio L. Braun ***@***.***>，写道：
… Yeah, file sizes are also massively different. Weird.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
@jiaminglu would it be possible to have the bias library you are using on mac aarch64 and the command line arguments you are using for your configure script ? I tried to build the current project and it failed with java 8 (rosetta make the build think it is a x86 rather than an aarch64). |
|
It turns out that it depends whether you have an Intel or a ARM version of Java running. What I did was to install openjdk with brew and then use that, it should give you the right architecture. All of this should be documented somewhere... 😅 For reproducing configurations, everything ends up being written to configure.out, so if you copy that you should be able to run the same. I'd also be interested in the configure.out, @jiaminglu. For some reason, I had to manually add more static libs besides libquadmath... . Not a good situation. jblas is doing this so that the resulting shared lib has no external dependencies, but this ends up having low level dependencies on gcc and so on... . Without it, people need to install gfortran to run jblas, which is also not great... . I've been wanting to overhaul the configure scripts for years now, very brittle and error prone... |
|
I used temurin8 with brew but apparently they does not provide an aarch64 build, only a x86 and therefore rosetta 2 kick-in. I ready to change the script and document them if needed but for the same reason as you, I wanted to ease the work and start from @jiamlinglu build. |
|
Yeah, that javah bug is another thing I wanted to fix for years :) javac now supports a -h flag where it will output header files, but I think it will then also compile everything already. The whole build process with all those generated files is very complex. I was young and foolish, I guess :) |
|
I am using MacPorts as the package manager because it supports my old Mac and better supports multiple architecture.
Here is my configure.out:
```
BUILD_TYPE=openblas
CC=gcc
CCC=c99
CFLAGS=-fPIC
F77=gfortran
FOUND_JAVA=true
FOUND_NM=true
INCDIRS=-Iinclude -I/Library/Java/JavaVirtualMachines/openjdk17-graalvm/Contents/Home//include -I/Library/Java/JavaVirtualMachines/openjdk17-graalvm/Contents/Home//include/darwin
JAVA_HOME=/Library/Java/JavaVirtualMachines/openjdk17-graalvm/Contents/Home/
LAPACK_HOME=./lapack-lite-3.1.1
LD=gcc
LDFLAGS=-shared
LIB=lib
LINKAGE_TYPE=static
LOADLIBES=/opt/local/lib/libopenblas.a /opt/local/lib/gcc12/gcc/arm64-apple-darwin21/12.2.0/../../../libgfortran.a /opt/local/lib/gcc12/gcc/arm64-apple-darwin21/12.2.0/../../../libquadmath.a
MAKE=make
NM=nm
OS_ARCH=aarch64
OS_ARCH_WITH_FLAVOR=aarch64
OS_NAME=Mac\ OS\ X
RUBY=ruby
SO=jnilib
```
And here is the toolchain versions, and everything is in arm64:
```
gcc12 @12.2.0_0+stdlib_flag
OpenBLAS @0.3.21_1+gcc12+lapack
openjdk17-graalvm @22.2.0_0
```
But since it has been long, I am not sure whether the package got updated after the binary in this pr compiled.
在 2023年3月27日 +0800 PM7:23，Mikio L. Braun ***@***.***>，写道：
… It turns out that it depends whether you have an Intel or a ARM version of Java running. What I did was to install openjdk with brew and then use that, it should give you the right architecture. All of this should be documented somewhere... 😅
For reproducing configurations, everything ends up being written to configure.out, so if you copy that you should be able to run the same. I'd also be interested in the configure.out, @jiaminglu. For some reason, I had to manually add more static libs besides libquadmath... .
Not a good situation. jblas is doing this so that the resulting shared lib has no external dependencies, but this ends up having low level dependencies on gcc and so on... . Without it, people need to install gfortran to run jblas, which is also not great... .
I've been wanting to overhaul the configure scripts for years now, very brittle and error prone...
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
For the dependencies I believe it will be OK if it links statically.
But what libraries have you added other than libquadmath? That’s really strange. Actually I did not expect to see libquadmath either, since it’s not needed in x86 builds. I just don’t want to spend time on this minor issue.
Also, I don’t suggest to use x86_64 version of compilers with rosetta2 to build anything targeting arm. It may break things with build scripts not carefully written, link against static libraries of wrong architecture for example.
在 2023年3月27日 +0800 PM7:23，Mikio L. Braun ***@***.***>，写道：
… It turns out that it depends whether you have an Intel or a ARM version of Java running. What I did was to install openjdk with brew and then use that, it should give you the right architecture. All of this should be documented somewhere... 😅
For reproducing configurations, everything ends up being written to configure.out, so if you copy that you should be able to run the same. I'd also be interested in the configure.out, @jiaminglu. For some reason, I had to manually add more static libs besides libquadmath... .
Not a good situation. jblas is doing this so that the resulting shared lib has no external dependencies, but this ends up having low level dependencies on gcc and so on... . Without it, people need to install gfortran to run jblas, which is also not great... .
I've been wanting to overhaul the configure scripts for years now, very brittle and error prone...
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
my new configure.out (after changing build.xml to use javac task) BUILD_TYPE=openblas For some reason my version of gfortran need openMP for some calculs (and therefore I need to have libgomp). my configuration command line : Still having the error when passing test ... (I try not to use clang -> there is a wrapper of gcc) |
Build script modified to support build on windows/mac aarch64 platform.
Binary generated on the following environments.
Windows
Build environment: msys2
Toolchain:
mingw-w64-clang-aarch64-toolchain14.0.0Fortran compiler:
mingw-w64-clang-aarch64-flang14.0.4BLAS library:
mingw-w64-clang-aarch64-openblas640.3.20-3Mac OS X
C compiler: Apple clang version 13.1.6
Fortran compiler: MacPorts gcc12 12.1.0_6+stdlib_flag
BLAS library
On mac, a dependency of libquadmath is added, which is required when compiling by gfortran from MacPorts' gcc12 package