[SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon by dongjoon-hyun · Pull Request #34548 · apache/spark

dongjoon-hyun · 2021-11-11T05:19:28Z

What changes were proposed in this pull request?

This PR aims to add ExtendedLevelDBTest to disable LevelDB selectively on Apple Silicon.

Add ExtendedLevelDBTest test tag
Disable LevelDB test cases from kvstore module/HistoryServerSuite/AppStatusStoreSuite.

Why are the changes needed?

Java 17 officially support Apple Silicon.

JEP 391: macOS/AArch64 Port
https://bugs.openjdk.java.net/browse/JDK-8251280

In addition, Oracle Java, Azul Zulu, and Eclipse Temurin Java 17 supports Apple Silicon natively.

/Users/dongjoon/.jenv/versions/oracle17/bin/java: Mach-O 64-bit executable arm64
/Users/dongjoon/.jenv/versions/zulu17/bin/java: Mach-O 64-bit executable arm64
/Users/dongjoon/.jenv/versions/temurin17/bin/java: Mach-O 64-bit executable arm64

However, LevelDBJNI still doesn't support Apple Silicon natively, the relevant test cases fail on M1.

Does this PR introduce any user-facing change?

No. This is a test-only PR.

How was this patch tested?

Run the above tests on M1.

…s on Apple Silicon

dongjoon-hyun · 2021-11-11T05:21:03Z

common/kvstore/src/test/java/org/apache/spark/util/kvstore/LevelDBIteratorSuite.java


  @Override
  protected KVStore createStore() throws Exception {
+    assumeFalse(SystemUtils.IS_OS_MAC_OSX && System.getProperty("os.arch").equals("aarch64"));


We cannot use ExtendedLevelDBTest in Java test.

dongjoon-hyun · 2021-11-11T05:21:09Z

common/kvstore/src/test/java/org/apache/spark/util/kvstore/LevelDBSuite.java


  @Before
  public void setup() throws Exception {
+    assumeFalse(SystemUtils.IS_OS_MAC_OSX && System.getProperty("os.arch").equals("aarch64"));


We cannot use ExtendedLevelDBTest in Java test.

dongjoon-hyun · 2021-11-11T05:22:19Z

core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala


    if (!inMemory) {
+      // LevelDB doesn't support Apple Silicon yet
+      assume(!(Utils.isMac && System.getProperty("os.arch").equals("aarch64")))


This test case will run in-memory test still on M1.

dongjoon-hyun · 2021-11-11T05:42:05Z

sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingSessionWindowSuite.scala

    // RocksDB doesn't support Apple Silicon yet
-    if (System.getProperty("os.name").equals("Mac OS X") &&
-        System.getProperty("os.arch").equals("aarch64")) {
+    if (Utils.isMac && System.getProperty("os.arch").equals("aarch64")) {


I piggy-back this replacement from System.getProperty("os.name").equals("Mac OS X") to Utils.isMac.

dongjoon-hyun · 2021-11-11T05:42:28Z

Thank you, @HyukjinKwon .

SparkQA · 2021-11-11T06:33:02Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49552/

dongjoon-hyun · 2021-11-11T07:03:37Z

Merged to master.

SparkQA · 2021-11-11T07:37:06Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49552/

SparkQA · 2021-11-11T08:01:51Z

Test build #145083 has finished for PR 34548 at commit 3c3605e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

…ndedLevelDBTest ### What changes were proposed in this pull request? This PR is a follow-up of #34548. This is missed due to `-Pyarn` profile. ### Why are the changes needed? This is required to pass `yarn` module on Apple Silicon. **BEFORE** ``` $ build/sbt "yarn/test" ... [info] YarnShuffleServiceSuite: [info] org.apache.spark.network.yarn.YarnShuffleServiceSuite *** ABORTED *** (20 milliseconds) [info] java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 ... ``` **AFTER** ``` $ build/sbt "yarn/test" -Pyarn -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest ... [info] Run completed in 4 minutes, 57 seconds. [info] Total number of tests run: 135 [info] Suites: completed 18, aborted 0 [info] Tests: succeeded 135, failed 0, canceled 1, ignored 0, pending 0 [info] All tests passed. [success] Total time: 319 s (05:19), completed Nov 12, 2021, 4:53:14 PM ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? A manual test on Apple Silicon. ``` $ build/sbt "yarn/test" -Pyarn -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest ``` Closes #34576 from dongjoon-hyun/SPARK-37282-2. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.com>

LuciferYang · 2021-11-18T06:09:24Z

Should we extract SystemUtils.IS_OS_MAC_OSX && SystemUtils.OS_ARCH.equals("aarch64") into an independent val, such as Utils.isAppleSilicon @dongjoon-hyun

HyukjinKwon · 2021-11-18T06:19:34Z

SGTM

LuciferYang · 2021-11-18T08:06:17Z

SGTM

Let me do this later

…sts on Apple Silicon ### What changes were proposed in this pull request? This PR aims to add `ExtendedLevelDBTest ` to disable `LevelDB` selectively on Apple Silicon. - Add `ExtendedLevelDBTest ` test tag - Disable LevelDB test cases from `kvstore` module/`HistoryServerSuite`/`AppStatusStoreSuite`. ### Why are the changes needed? Java 17 officially support Apple Silicon. - JEP 391: macOS/AArch64 Port - https://bugs.openjdk.java.net/browse/JDK-8251280 In addition, Oracle Java, Azul Zulu, and Eclipse Temurin Java 17 supports Apple Silicon natively. ``` /Users/dongjoon/.jenv/versions/oracle17/bin/java: Mach-O 64-bit executable arm64 /Users/dongjoon/.jenv/versions/zulu17/bin/java: Mach-O 64-bit executable arm64 /Users/dongjoon/.jenv/versions/temurin17/bin/java: Mach-O 64-bit executable arm64 ``` However, LevelDBJNI still doesn't support Apple Silicon natively, the relevant test cases fail on M1. ### Does this PR introduce _any_ user-facing change? No. This is a test-only PR. ### How was this patch tested? Run the above tests on M1. Closes apache#34548 from dongjoon-hyun/SPARK-37282. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit a116045) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

…ndedLevelDBTest ### What changes were proposed in this pull request? This PR is a follow-up of apache#34548. This is missed due to `-Pyarn` profile. ### Why are the changes needed? This is required to pass `yarn` module on Apple Silicon. **BEFORE** ``` $ build/sbt "yarn/test" ... [info] YarnShuffleServiceSuite: [info] org.apache.spark.network.yarn.YarnShuffleServiceSuite *** ABORTED *** (20 milliseconds) [info] java.lang.UnsatisfiedLinkError: Could not load library. Reasons: [no leveldbjni64-1.8 ... ``` **AFTER** ``` $ build/sbt "yarn/test" -Pyarn -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest ... [info] Run completed in 4 minutes, 57 seconds. [info] Total number of tests run: 135 [info] Suites: completed 18, aborted 0 [info] Tests: succeeded 135, failed 0, canceled 1, ignored 0, pending 0 [info] All tests passed. [success] Total time: 319 s (05:19), completed Nov 12, 2021, 4:53:14 PM ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? A manual test on Apple Silicon. ``` $ build/sbt "yarn/test" -Pyarn -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest ``` Closes apache#34576 from dongjoon-hyun/SPARK-37282-2. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Kousuke Saruta <sarutak@oss.nttdata.com> (cherry picked from commit 8b45a08) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

[SPARK-37282][TESTS] Add ExtendedLevelDBTest and disable LevelDB test…

3c3605e

…s on Apple Silicon

github-actions bot added CORE SQL STRUCTURED STREAMING labels Nov 11, 2021

dongjoon-hyun commented Nov 11, 2021

View reviewed changes

HyukjinKwon approved these changes Nov 11, 2021

View reviewed changes

dongjoon-hyun commented Nov 11, 2021

View reviewed changes

dongjoon-hyun closed this in a116045 Nov 11, 2021

dongjoon-hyun deleted the SPARK-37282 branch November 11, 2021 07:03

dongjoon-hyun mentioned this pull request Nov 13, 2021

[SPARK-37282][TESTS][FOLLOWUP] Mark YarnShuffleServiceSuite as ExtendedLevelDBTest #34576

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon#34548

[SPARK-37282][TESTS] Add `ExtendedLevelDBTest` and disable LevelDB tests on Apple Silicon#34548
dongjoon-hyun wants to merge 1 commit intoapache:masterfrom
dongjoon-hyun:SPARK-37282

dongjoon-hyun commented Nov 11, 2021 •

edited

Loading

Uh oh!

dongjoon-hyun Nov 11, 2021

Uh oh!

dongjoon-hyun Nov 11, 2021

Uh oh!

dongjoon-hyun Nov 11, 2021

Uh oh!

dongjoon-hyun Nov 11, 2021

Uh oh!

dongjoon-hyun commented Nov 11, 2021

Uh oh!

SparkQA commented Nov 11, 2021

Uh oh!

dongjoon-hyun commented Nov 11, 2021

Uh oh!

SparkQA commented Nov 11, 2021

Uh oh!

SparkQA commented Nov 11, 2021

Uh oh!

LuciferYang commented Nov 18, 2021

Uh oh!

HyukjinKwon commented Nov 18, 2021

Uh oh!

LuciferYang commented Nov 18, 2021 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

dongjoon-hyun commented Nov 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

dongjoon-hyun Nov 11, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Nov 11, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Nov 11, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Nov 11, 2021

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Nov 11, 2021

Uh oh!

SparkQA commented Nov 11, 2021

Uh oh!

dongjoon-hyun commented Nov 11, 2021

Uh oh!

SparkQA commented Nov 11, 2021

Uh oh!

SparkQA commented Nov 11, 2021

Uh oh!

LuciferYang commented Nov 18, 2021

Uh oh!

HyukjinKwon commented Nov 18, 2021

Uh oh!

LuciferYang commented Nov 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dongjoon-hyun commented Nov 11, 2021 •

edited

Loading

LuciferYang commented Nov 18, 2021 •

edited

Loading