[SPARK-34341][BUILD] Skip zinc related operations on aarch64

[Yikun]

What changes were proposed in this pull request?

Skip the zinc related installation operations on aarch64 platform.

Why are the changes needed?

The standalone zinc is not supported well on aarch64, so that the error ouput, cannot execute binary file: Exec format error dumped after build/mvn is called.

This patch try to skip the zinc installation and related operations on aarch64 to make sure the error output doesn't print again on aarch64.

Does this PR introduce any user-facing change?

No

How was this patch tested?

simple cmd: build/mvn -v, see no error ouput again in aarch64, and nothing changed on x86

• on AArch64 Ubuntu

root@yikun-arm:~/dev/spark# uname -a
Linux yikun-arm 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:10 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
root@yikun-arm:~/dev/spark# uname -m
aarch64
root@yikun-arm:~/dev/spark# build/mvn -v
Using `mvn` from path: /root/dev/spark/build/apache-maven-3.6.3/bin/mvn
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /root/dev/spark/build/apache-maven-3.6.3
Java version: 1.8.0_222, vendor: Private Build, runtime: /usr/lib/jvm/java-8-openjdk-arm64/jre
Default locale: en, platform encoding: UTF-8
OS name: "linux", version: "4.15.0-70-generic", arch: "aarch64", family: "unix"

• on x86 Mac OS

# uname -a
Darwin MacBook.local 19.6.0 Darwin Kernel Version 19.6.0: Tue Nov 10 00:10:30 PST 2020; root:xnu-6153.141.10~1/RELEASE_X86_64 x86_64
# uname -m
x86_64
# build/mvn -v
Using `mvn` from path: /Users/jiangyikun/huawei/apache-maven-3.6.3/bin/mvn
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /Users/jiangyikun/huawei/apache-maven-3.6.3
Java version: 1.8.0_221, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk1.8.0_221.jdk/Contents/Home/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "mac os x", version: "10.15.7", arch: "x86_64", family: "mac"

• on x86 Ubuntu

root@yikun-x86:~/spark# uname -a
Linux yikun-x86 5.4.0-58-generic #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@yikun-x86:~/spark# uname -m
x86_64
root@yikun-x86:~/spark# ./build//mvn  -v
Using `mvn` from path: /root/spark/build/apache-maven-3.6.3/bin/mvn
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /root/spark/build/apache-maven-3.6.3
Java version: 1.8.0_275, vendor: Private Build, runtime: /usr/lib/jvm/java-8-openjdk-amd64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.4.0-58-generic", arch: "amd64", family: "unix"

[AmplabJenkins]

Can one of the admins verify this patch?

[HyukjinKwon]

@Yikun,

The standalone zinc is not supported well on aarch64, so that the error ouput, cannot execute binary file: Exec format error dumped after build/mvn is called.

Do you have any issue link in zinc for this? It would be great if we can confirm that zinc is not supported in aarch64.

[Yikun]

@HyukjinKwon Thanks for you super fast reply. : )

Do you have any issue link in zinc for this?

I think it's a little diffcult to get this link, because the latest standalone version zinc (0.3.15, spark used), it is already deprecated, so we have no chance to raise a issue to confirm the Arm support from zinc officially [1].

It would be great if we can confirm that zinc is not supported in aarch64.

But I can confirmed the 0.3.15 zinc, will call nailgun and finally call ng, but ng doesn't have completely arm support. [2][3] so that I got this cannot execute binary file: Exec format error here.

And another info is that I check the sbt/zinc [4], looks like we can only use the scalor-maven-plugin to enable zinc with maven, but not the stand alone style. I do some try on this plugin demo[5] on aarch64, it works.

IMO, maybe the solution is that first we skip the zinc operations on aarch64 in here , and switch standalone zinc to scalor-maven-plugin to enable zinc in future (if our spark really want).

[1] https://github.com/typesafehub/zinc
[2] https://github.com/typesafehub/zinc/blob/8522622/src/universal/bin/nailgun#L36-L43
[3] https://github.com/typesafehub/zinc/tree/8522622/src/universal/bin/ng
[4] https://github.com/sbt/zinc
[5] https://github.com/random-maven/scalor-maven-plugin/tree/master/demo

[HyukjinKwon]

cc @huangtianhua, do you have any input on this?

[huangtianhua]

Sorry reply late. Yes, we found standalone zinc is not supported well on aarch64 before, but it doesn't affect mvn building/testing, so we ignored it. To skip zinc related commands on aarch64 first is ok to me, so +1 :) thanks

[HyukjinKwon]

Merged to master.

[odidev]

@HyukjinKwon @huangtianhua, I was actually interested in enablement of "zinc" operations in aarch64 as well. Are you planning to add this in future? @Yikun I've tried modifying the pom.xml to to add scalor-maven-plugin to enable zinc, but since it is getting compiled and tests are running without zinc also, How can we test that it is running the build using zinc? Let me know if i am m missing something here.

[HyukjinKwon]

Manual test is fine (just showing the logs in the PR description). If we can fix it not invasively, I think it's fine to add zinc back in aarch64 too.

[Yikun]

How can we test that it is running the build using zinc?

@odidev Sorry for late reply, and looks like there is some misleading content in my last reply. Actually, the scala-maven-plugin v4.0.0 already support the new version Zinc 1.x with multi-arch support rather than scalor-maven-plugin. [1] So, you can just use the scala-maven-plugin v4.X to enable the Zinc incremental compiler in aarch64 and also x86.

If we can fix it not invasively, I think it's fine to add zinc back in aarch64 too.

@HyukjinKwon And when I do investigation on the Zinc multi-arch support, I found there is bug SPARK-34539, looks like the zinc server code can be cleanup completely after we bump the scala-maven-plugin to v4.X in SPARK-28759, because the zinc increamental compiler has been enabled by default[2] in scala-maven-plugin 4.X.

[1] davidB/scala-maven-plugin#321
[2] davidB/scala-maven-plugin@0f01f2f