Upload osx_arm64 jars to maven repository

[paddyroddy]

When I run mvn clean install in https://github.com/flimlib/flimj-ui I cannot build it as the jars are not found in the maven repository

Could not resolve dependencies for project flimlib:flimj-ui:jar:1.1.1-SNAPSHOT: flimlib:flimlib:jar:natives-osx_${scijava.platform.bits}:2.1.1

Having a look around, it looks like the arm64 jars are needed to be built and uploaded to https://maven.scijava.org/content/repositories/releases/flimlib/flimlib/2.1.1/. If I can manage to compile this repo then happy to do this myself but if anyone has beat me to it please can you do this

[paddyroddy]

Okay, so I'm aware that the CI does the builds and that GH doesn't currently support M1 actions/runner-images#2187

I have now been able to compile flimlib. If you can tell me what needs to be done to upload to the maven repository, then I'm happy to do it or send you the files.

[ctrueden]

@paddyroddy Cool, thanks for pursuing that. Yeah, I guess we can upload them manually for now, sure. Wanna share them online temporarily, I can download them, and then deploy?

Alternately @paddyroddy if you're willing, I could make you a deploy account on maven.scijava.org and you could deploy them yourself using mvn deploy:deploy-file.

Let me know which you'd prefer!

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/how-does-scijava-platform-bits-expand-issues-on-m1-mac/62238/2

[paddyroddy]

Hi @ctrueden, I raised this issue before I understood the problem. Do you think the jar actually need uploading? In that maven repository I see no mention of architecture i.e. Intel/AMD etc.

[paddyroddy]

This is the contents of my ~/.m2/repository/flimlib by the way

[paddyroddy]

I've put the contents of ~/.m2/repository/flimlib here https://www.dropbox.com/sh/zyc8cfibywf2xss/AACfaUDGWu_rDzAb2ax11_Dla?dl=0 let me know if that works for you

[mkitti]

What are the build instructions for the native code?

Is it cmake based? https://github.com/flimlib/flimlib/blob/master/CMakeLists.txt ?

[ctrueden]

@mkitti Yes, it's Maven, which calls CMake via the exec-maven-plugin So the architecture value probably comes from Java's os.arch, if that's what you were wondering.

[mkitti]

To simplify the native component, this cmake workflow seems to work:

git clone https://github.com/flimlib/flimlib.git
mkdir flimlib/build
cd flimlib/build
cmake -DCMAKE_BUILD_TYPE=Release ..
cmake --build .

The main requirement seem to be C and C++ compilers, SWIG, and JDK 8.

Conda-Forge seems to have osx-arm64 builds for SWIG and JDK:
https://anaconda.org/conda-forge/swig/
https://anaconda.org/conda-forge/openjdk

As long as we can use clang, I think we could make a viable conda-forge recipe out of this.

[ctrueden]

@mkitti Yes, that's pretty much what the exec-maven-plugin is doing under the hood. There are reasons we use Maven, though: 1) package up the resultant artifact into a JAR file and deploy it to the SciJava Maven repository; 2) build the Java-specific SWIG wrapper code and customizations. So the Maven build is the single entry point into the build, just like usual with Java projects—it's just that some native code gets built under the hood too using CMake et al.

[mkitti]

Right, and I think we can use maven in a conda-forge recipe as well although I do not have it in my conda-forge virtual machine at the moment.

https://anaconda.org/conda-forge/maven

My understanding is that GitHubActions does not yet have a macos-arm build environment. My thought is that we could use conda-forge's build service to provide the osx-arm native libraries.

What I am less sure about is if we also want conda-forge to build the entire Java package as well.

[ctrueden]

@mkitti Ah, I see what you are thinking now, thanks!

But it makes me nervous to use the conda-forge infrastructure for this, unless we are adding value to the conda-forge ecosystem by actually publishing this project on conda-forge. We would still ultimately need to deploy the binaries to maven.scijava.org as well, so that they can be consumed by Maven-based tooling. conda-forge/conda-forge.github.io#590 may be tangentially related here...

Ideally, we'd just use GitHub Actions to do the arm64 builds as well. We could potentially cross-compile M1 arm64 from an x86_64 mac container.

[mkitti]

Yes, I've had a similar conversation with them regarding Julia packages: conda-forge/julia-feedstock#14 . Yet, we have julia building on conda-forge. The main contention is that they also want to encourage use of conda to distribute packages.

My opinion on the matter is that FIJI should be available on conda-forge and that people should be able to distribute ImageJ plugins on conda-forge for FIJI. However, we should not be obligated to package all plugins as individual conda-forge packages to start. That effort would need to come from the authors of the individual plugins. There should probably be a distinct ImageJ package, and then a separate FIJI package. Users can then ultimately decide how they want to install additional software.

In this case, we just need to compile the C library. The C library itself is a valuable contribution. Someone else can use it is a dependency in another piece of software. There are many C libraries or even C++ header only libraries on conda-forge.

From there, there is nothing stopping us from downloading the binary tarballs from conda-forge and repackaging it into a jar via maven using Github Actions.

[paddyroddy]

I have successfully got flimlib, flimj-ops and flimj-ui to build on M1. I'm not sure where best to raise this, but I'll add to this thread here to be consistent.

In the pom.xml in flimlib add the <scijava.platform.bits>some-name-here</scijava.platform.bits> in the <properties> section (overwriting this scijava/pom-scijava-base#30)

then in flimj-ops add the <scijava.platform.bits>some-name-here</scijava.platform.bits> in the <properties> section again and change the version here https://github.com/flimlib/flimj-ops/blob/1869916d7acb6ae9e4488fc1e35adde7ba59d7d8/pom.xml#L96 to what you built i.e. 2.1.2-SNAPSHOT (maven searches locally before looking in the cloud)

finally in flimj-ui add the <scijava.platform.bits>some-name-here</scijava.platform.bits> in the <properties> section again and change the appropriate versions here https://github.com/flimlib/flimj-ui/blob/88f01032de6899113ea9e51aae1cdebe6e04dc24/pom.xml#L97-L98 to i.e. 2.1.2-SNAPSHOT

Then build each in turn:

cd $REPO_LOCATION/flimlib
mvn clean install
cd $REPO_LOCATION/flimj-ops
mvn clean install -DskipTests
cd $REPO_LOCATION/flimj-ui
mvn clean install

For this to work, the tests in flimj-ops must be skipped as they fail and maven will not finish the build. I will raise a relevant issue there and reference here

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/how-does-scijava-platform-bits-expand-issues-on-m1-mac/62238/4

[paddyroddy]

also looks like M1 CI will be available soon actions/runner#805 (comment)

[imagesc-bot]

This issue has been mentioned on Image.sc Forum. There might be relevant details there:

https://forum.image.sc/t/cmake-fails-to-build-on-m1-mac-apple-silicon/61964/8

[marktsuchida]

It doesn't look like GitHub Actions hosted runners for Apple M1 will be available any time soon.

But cross-compiling for arm64 on an x86_64 host is not hard. We just need to pass -DCMAKE_OSX_ARCHITECTURES=arm64 to cmake. (A universal binary can be created using -DCMAKE_OSX_ARCHITECTURES=x86_64;arm64.) This works for flimlib when invoking cmake directly.

On macOS arm64, the following command successfully creates native packages containing x86_64 code and named with -natives-osx_64.jar.

mvn -Dcmake.generator=-DCMAKE_OSX_ARCHITECTURES=x86_64 -Dscijava.platform.bits=64 -DskipTests package

(The use of the cmake.generator property to pass in an arbitrary cmake configure flag being a temporary hack. Perhaps the property should be renamed if it will be used like this.)

Conversely, on macOS x86_64,

mvn -Dcmake.generator=-DCMAKE_OSX_ARCHITECTURES=arm64 -Dscijava.platform.bits=arm64 -DskipTests package

produces flimlib-2.1.2-SNAPSHOT-natives-osx_arm64.jar correctly.

Note that tests need to be skipped in both directions (although they should work on an arm64 host if an x86_64 JVM is used).

What I'm not sure about is how this should be made to work with the current CI scripts.

[mkitti]

I can make a runner available if needed.

[hinerm]

I wasn't aware of this issue so I independently built & uploaded arm64 artifacts for all the flimlib releases that I could (2.0.0 did not build on my system):

Closing for now. Please re-open if 2.0.0 binaries are strictly needed.

[ctrueden]

@hinerm Thanks! That's great as a stopgap. But will future flimlib releases include macos-arm64 binaries? If not, I think this issue cannot be considered fully resolved.