[SPARK-46205][CORE] Improve `PersistenceEngine` performance with `KryoSerializer` #44113

dongjoon-hyun · 2023-12-01T22:55:07Z

What changes were proposed in this pull request?

This PR aims to improve PersistenceEngine performance with KryoSerializer via introducing a new configuration, spark.deploy.recoverySerializer.

Why are the changes needed?

Allow users to choose a better serializer to get a better performance in their environment. Especially, KryoSerializer is about 3x faster than JavaSerializer with FileSystemPersistenceEngine.

================================================================================================
PersistenceEngineBenchmark
================================================================================================

OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1051-azure
AMD EPYC 7763 64-Core Processor
1000 Workers:                                    Best Time(ms)   Avg Time(ms)   Stdev(ms)    Rate(M/s)   Per Row(ns)   Relative
-------------------------------------------------------------------------------------------------------------------------------
ZooKeeperPersistenceEngine with JavaSerializer            1202           1298         138          0.0     1201614.2       1.0X
ZooKeeperPersistenceEngine with KryoSerializer             951           1004          48          0.0      950559.0       1.3X
FileSystemPersistenceEngine with JavaSerializer            212            217           6          0.0      211623.2       5.7X
FileSystemPersistenceEngine with KryoSerializer             79             81           2          0.0       79132.5      15.2X
BlackHolePersistenceEngine                                   0              0           0         30.9          32.4   37109.8X

Does this PR introduce any user-facing change?

No. The default behavior is the same.

How was this patch tested?

Pass the CIs with the new added test cases.

Was this patch authored or co-authored using generative AI tooling?

No.

…oSerializer`

dongjoon-hyun · 2023-12-01T23:01:35Z

~~I'm going to update the benchmark files.~~ This is done.

https://github.com/dongjoon-hyun/spark/actions/runs/7066098422 (Java 17)
https://github.com/dongjoon-hyun/spark/actions/runs/7066103306 (Java 21)

dongjoon-hyun · 2023-12-01T23:24:35Z

core/src/main/scala/org/apache/spark/deploy/master/FileSystemPersistenceEngine.scala

      if (out != null) {
        out.close()
      }
+      fileOut.close()


Without this reordering, KryoSerializer complains.

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala

cc @sadikovi

dongjoon-hyun · 2023-12-02T00:08:09Z

Could you review this when you have a chance, @yaooqinn , @beliefer , @LuciferYang ?

beliefer

LGTM except some comments.

beliefer · 2023-12-02T02:04:02Z

core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala

+      val e = master.invokePrivate(_persistenceEngine()).asInstanceOf[FileSystemPersistenceEngine]
+      assert(e.serializer.isInstanceOf[KryoSerializer])


Suggested change

val e = master.invokePrivate(_persistenceEngine()).asInstanceOf[FileSystemPersistenceEngine]

assert(e.serializer.isInstanceOf[KryoSerializer])

val persistenceEngine = master.invokePrivate(_persistenceEngine()).asInstanceOf[FileSystemPersistenceEngine]

assert(persistenceEngine.serializer.isInstanceOf[KryoSerializer])

Thank you but this was intention in order to avoid Scalastyle failure. Line 338 has 99 char already.

beliefer · 2023-12-02T02:12:44Z

core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineBenchmark.scala

@@ -60,15 +61,23 @@ object PersistenceEngineBenchmark extends BenchmarkBase {
    runBenchmark("PersistenceEngineBenchmark") {
      val benchmark = new Benchmark(s"$numWorkers Workers", numWorkers, output = output)

-      benchmark.addCase("ZooKeeperPersistenceEngine", numIters) { _ =>
+      benchmark.addCase("ZooKeeperPersistenceEngine with JavaSerializer", numIters) { _ =>


How about

Seq(serializerJava, serializerKryo).foreach { serializer => benchmark.addCase("ZooKeeperPersistenceEngine with ${serializer.getClass.getSimpleName}", numIters) { _ => val engine = new ZooKeeperPersistenceEngine(conf, serializer) ... }

Sorry, but this is also intentional for the ordering like the following.

OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1051-azure AMD EPYC 7763 64-Core Processor 1000 Workers: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------- ZooKeeperPersistenceEngine with JavaSerializer 1202 1298 138 0.0 1201614.2 1.0X ZooKeeperPersistenceEngine with KryoSerializer 951 1004 48 0.0 950559.0 1.3X FileSystemPersistenceEngine with JavaSerializer 212 217 6 0.0 211623.2 5.7X FileSystemPersistenceEngine with KryoSerializer 79 81 2 0.0 79132.5 15.2X BlackHolePersistenceEngine 0 0 0 30.9 32.4 37109.8X

Oh, for this one, it seems that I misunderstood your previous comment. Let me address it at the next PR (or a follow-up), @beliefer . Sorry for my mistake.

beliefer · 2023-12-02T02:13:00Z

core/src/test/scala/org/apache/spark/deploy/master/PersistenceEngineSuite.scala

 import org.apache.spark.util.Utils

+


Please revert this line.

Thank you! Sure.

dongjoon-hyun · 2023-12-02T02:29:01Z

Thank you so much, @beliefer .

Merged to master for Apache Spark 4.0.0.

### What changes were proposed in this pull request? This is a follow-up of #44113 to address a comment about simplifying the benchmark. ### Why are the changes needed? To simplify. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually. ``` $ build/sbt "core/Test/runMain org.apache.spark.deploy.master.PersistenceEngineBenchmark" ... [info] OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Mac OS X 14.2 [info] Apple M1 Max [info] 1000 Workers: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative [info] ------------------------------------------------------------------------------------------------------------------------------- [info] ZooKeeperPersistenceEngine with JavaSerializer 12602 12847 253 0.0 12601717.5 1.0X [info] ZooKeeperPersistenceEngine with KryoSerializer 12116 12130 13 0.0 12116373.8 1.0X [info] FileSystemPersistenceEngine with JavaSerializer 429 435 6 0.0 429374.9 29.3X [info] FileSystemPersistenceEngine with KryoSerializer 179 180 2 0.0 178795.7 70.5X [info] BlackHolePersistenceEngine 0 0 0 46.6 21.5 587273.6X [success] Total time: 126 s (02:06), completed Dec 1, 2023, 7:59:25 PM ``` ### Was this patch authored or co-authored using generative AI tooling? Closes #44118 from dongjoon-hyun/SPARK-46205-2. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…oSerializer` ### What changes were proposed in this pull request? This PR aims to improve `PersistenceEngine` performance with `KryoSerializer` via introducing a new configuration, `spark.deploy.recoverySerializer`. ### Why are the changes needed? Allow users to choose a better serializer to get a better performance in their environment. Especially, `KryoSerializer` is about **3x faster** than `JavaSerializer` with `FileSystemPersistenceEngine`. ``` ================================================================================================ PersistenceEngineBenchmark ================================================================================================ OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1051-azure AMD EPYC 7763 64-Core Processor 1000 Workers: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------- ZooKeeperPersistenceEngine with JavaSerializer 1202 1298 138 0.0 1201614.2 1.0X ZooKeeperPersistenceEngine with KryoSerializer 951 1004 48 0.0 950559.0 1.3X FileSystemPersistenceEngine with JavaSerializer 212 217 6 0.0 211623.2 5.7X FileSystemPersistenceEngine with KryoSerializer 79 81 2 0.0 79132.5 15.2X BlackHolePersistenceEngine 0 0 0 30.9 32.4 37109.8X ``` ### Does this PR introduce _any_ user-facing change? No. The default behavior is the same. ### How was this patch tested? Pass the CIs with the new added test cases. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#44113 from dongjoon-hyun/SPARK-46205. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

### What changes were proposed in this pull request? This is a follow-up of apache#44113 to address a comment about simplifying the benchmark. ### Why are the changes needed? To simplify. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually. ``` $ build/sbt "core/Test/runMain org.apache.spark.deploy.master.PersistenceEngineBenchmark" ... [info] OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Mac OS X 14.2 [info] Apple M1 Max [info] 1000 Workers: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative [info] ------------------------------------------------------------------------------------------------------------------------------- [info] ZooKeeperPersistenceEngine with JavaSerializer 12602 12847 253 0.0 12601717.5 1.0X [info] ZooKeeperPersistenceEngine with KryoSerializer 12116 12130 13 0.0 12116373.8 1.0X [info] FileSystemPersistenceEngine with JavaSerializer 429 435 6 0.0 429374.9 29.3X [info] FileSystemPersistenceEngine with KryoSerializer 179 180 2 0.0 178795.7 70.5X [info] BlackHolePersistenceEngine 0 0 0 46.6 21.5 587273.6X [success] Total time: 126 s (02:06), completed Dec 1, 2023, 7:59:25 PM ``` ### Was this patch authored or co-authored using generative AI tooling? Closes apache#44118 from dongjoon-hyun/SPARK-46205-2. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

### What changes were proposed in this pull request? This PR aims to update `Spark Standalone` cluster recovery configurations. ### Why are the changes needed? We need to document - #44173 - #44129 - #44113 ![Screenshot 2023-12-06 at 7 15 24 PM](https://github.com/apache/spark/assets/9700541/04f0be6f-cdfb-4d87-b1b5-c4bf131f460a) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #44227 from dongjoon-hyun/SPARK-46299. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…oSerializer` ### What changes were proposed in this pull request? This PR aims to improve `PersistenceEngine` performance with `KryoSerializer` via introducing a new configuration, `spark.deploy.recoverySerializer`. ### Why are the changes needed? Allow users to choose a better serializer to get a better performance in their environment. Especially, `KryoSerializer` is about **3x faster** than `JavaSerializer` with `FileSystemPersistenceEngine`. ``` ================================================================================================ PersistenceEngineBenchmark ================================================================================================ OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1051-azure AMD EPYC 7763 64-Core Processor 1000 Workers: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------- ZooKeeperPersistenceEngine with JavaSerializer 1202 1298 138 0.0 1201614.2 1.0X ZooKeeperPersistenceEngine with KryoSerializer 951 1004 48 0.0 950559.0 1.3X FileSystemPersistenceEngine with JavaSerializer 212 217 6 0.0 211623.2 5.7X FileSystemPersistenceEngine with KryoSerializer 79 81 2 0.0 79132.5 15.2X BlackHolePersistenceEngine 0 0 0 30.9 32.4 37109.8X ``` ### Does this PR introduce _any_ user-facing change? No. The default behavior is the same. ### How was this patch tested? Pass the CIs with the new added test cases. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#44113 from dongjoon-hyun/SPARK-46205. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

### What changes were proposed in this pull request? This is a follow-up of apache#44113 to address a comment about simplifying the benchmark. ### Why are the changes needed? To simplify. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually. ``` $ build/sbt "core/Test/runMain org.apache.spark.deploy.master.PersistenceEngineBenchmark" ... [info] OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Mac OS X 14.2 [info] Apple M1 Max [info] 1000 Workers: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative [info] ------------------------------------------------------------------------------------------------------------------------------- [info] ZooKeeperPersistenceEngine with JavaSerializer 12602 12847 253 0.0 12601717.5 1.0X [info] ZooKeeperPersistenceEngine with KryoSerializer 12116 12130 13 0.0 12116373.8 1.0X [info] FileSystemPersistenceEngine with JavaSerializer 429 435 6 0.0 429374.9 29.3X [info] FileSystemPersistenceEngine with KryoSerializer 179 180 2 0.0 178795.7 70.5X [info] BlackHolePersistenceEngine 0 0 0 46.6 21.5 587273.6X [success] Total time: 126 s (02:06), completed Dec 1, 2023, 7:59:25 PM ``` ### Was this patch authored or co-authored using generative AI tooling? Closes apache#44118 from dongjoon-hyun/SPARK-46205-2. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

### What changes were proposed in this pull request? This PR aims to update `Spark Standalone` cluster recovery configurations. ### Why are the changes needed? We need to document - apache#44173 - apache#44129 - apache#44113 ![Screenshot 2023-12-06 at 7 15 24 PM](https://github.com/apache/spark/assets/9700541/04f0be6f-cdfb-4d87-b1b5-c4bf131f460a) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#44227 from dongjoon-hyun/SPARK-46299. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

…oSerializer` ### What changes were proposed in this pull request? This PR aims to improve `PersistenceEngine` performance with `KryoSerializer` via introducing a new configuration, `spark.deploy.recoverySerializer`. ### Why are the changes needed? Allow users to choose a better serializer to get a better performance in their environment. Especially, `KryoSerializer` is about **3x faster** than `JavaSerializer` with `FileSystemPersistenceEngine`. ``` ================================================================================================ PersistenceEngineBenchmark ================================================================================================ OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Linux 5.15.0-1051-azure AMD EPYC 7763 64-Core Processor 1000 Workers: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------------- ZooKeeperPersistenceEngine with JavaSerializer 1202 1298 138 0.0 1201614.2 1.0X ZooKeeperPersistenceEngine with KryoSerializer 951 1004 48 0.0 950559.0 1.3X FileSystemPersistenceEngine with JavaSerializer 212 217 6 0.0 211623.2 5.7X FileSystemPersistenceEngine with KryoSerializer 79 81 2 0.0 79132.5 15.2X BlackHolePersistenceEngine 0 0 0 30.9 32.4 37109.8X ``` ### Does this PR introduce _any_ user-facing change? No. The default behavior is the same. ### How was this patch tested? Pass the CIs with the new added test cases. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#44113 from dongjoon-hyun/SPARK-46205. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

### What changes were proposed in this pull request? This is a follow-up of apache#44113 to address a comment about simplifying the benchmark. ### Why are the changes needed? To simplify. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manually. ``` $ build/sbt "core/Test/runMain org.apache.spark.deploy.master.PersistenceEngineBenchmark" ... [info] OpenJDK 64-Bit Server VM 17.0.9+9-LTS on Mac OS X 14.2 [info] Apple M1 Max [info] 1000 Workers: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative [info] ------------------------------------------------------------------------------------------------------------------------------- [info] ZooKeeperPersistenceEngine with JavaSerializer 12602 12847 253 0.0 12601717.5 1.0X [info] ZooKeeperPersistenceEngine with KryoSerializer 12116 12130 13 0.0 12116373.8 1.0X [info] FileSystemPersistenceEngine with JavaSerializer 429 435 6 0.0 429374.9 29.3X [info] FileSystemPersistenceEngine with KryoSerializer 179 180 2 0.0 178795.7 70.5X [info] BlackHolePersistenceEngine 0 0 0 46.6 21.5 587273.6X [success] Total time: 126 s (02:06), completed Dec 1, 2023, 7:59:25 PM ``` ### Was this patch authored or co-authored using generative AI tooling? Closes apache#44118 from dongjoon-hyun/SPARK-46205-2. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

### What changes were proposed in this pull request? This PR aims to update `Spark Standalone` cluster recovery configurations. ### Why are the changes needed? We need to document - apache#44173 - apache#44129 - apache#44113 ![Screenshot 2023-12-06 at 7 15 24 PM](https://github.com/apache/spark/assets/9700541/04f0be6f-cdfb-4d87-b1b5-c4bf131f460a) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual review. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#44227 from dongjoon-hyun/SPARK-46299. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

### What changes were proposed in this pull request? This is a logical revert of SPARK-46205 - #44113 - #44118 ### Why are the changes needed? The initial implementation didn't handle the class initialization logic properly. Until we have a fix, I'd like to revert this from `master` branch. ### Does this PR introduce _any_ user-facing change? No, this is not released yet. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #46087 from dongjoon-hyun/SPARK-47875. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

[SPARK-46205][CORE] Improve PersistenceEngine performance with `Kry…

abd2afd

…oSerializer`

github-actions bot added the CORE label Dec 1, 2023

dongjoon-hyun commented Dec 1, 2023

View reviewed changes

Update benchmark results

aa57ce9

beliefer approved these changes Dec 2, 2023

View reviewed changes

Address comments

204fbab

dongjoon-hyun closed this in 0d40b1a Dec 2, 2023

dongjoon-hyun deleted the SPARK-46205 branch December 2, 2023 02:53

dongjoon-hyun mentioned this pull request Dec 2, 2023

[SPARK-46205][CORE][TESTS][FOLLOWUP] Simplify PersistenceEngineBenchmark #44118

Closed

dongjoon-hyun mentioned this pull request Dec 7, 2023

[SPARK-46299][DOCS] Make spark.deploy.recovery* docs up-to-date #44227

Closed

dongjoon-hyun mentioned this pull request Dec 11, 2023

[SPARK-46216][CORE] Improve FileSystemPersistenceEngine to support compressions #44129

Closed

dongjoon-hyun mentioned this pull request Apr 16, 2024

[SPARK-47875][CORE] Remove spark.deploy.recoverySerializer #46087

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-46205][CORE] Improve `PersistenceEngine` performance with `KryoSerializer` #44113

[SPARK-46205][CORE] Improve `PersistenceEngine` performance with `KryoSerializer` #44113

dongjoon-hyun commented Dec 1, 2023 •

edited

Loading

dongjoon-hyun commented Dec 1, 2023 •

edited

Loading

dongjoon-hyun Dec 1, 2023 •

edited

Loading

dongjoon-hyun Dec 11, 2023

dongjoon-hyun commented Dec 2, 2023

beliefer left a comment

beliefer Dec 2, 2023

dongjoon-hyun Dec 2, 2023

beliefer Dec 2, 2023

dongjoon-hyun Dec 2, 2023

dongjoon-hyun Dec 2, 2023

beliefer Dec 2, 2023

dongjoon-hyun Dec 2, 2023

dongjoon-hyun commented Dec 2, 2023

		val e = master.invokePrivate(_persistenceEngine()).asInstanceOf[FileSystemPersistenceEngine]
		assert(e.serializer.isInstanceOf[KryoSerializer])

[SPARK-46205][CORE] Improve PersistenceEngine performance with KryoSerializer #44113

[SPARK-46205][CORE] Improve PersistenceEngine performance with KryoSerializer #44113

Conversation

dongjoon-hyun commented Dec 1, 2023 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

dongjoon-hyun commented Dec 1, 2023 • edited Loading

dongjoon-hyun Dec 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun commented Dec 2, 2023

beliefer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dongjoon-hyun commented Dec 2, 2023

[SPARK-46205][CORE] Improve `PersistenceEngine` performance with `KryoSerializer` #44113

[SPARK-46205][CORE] Improve `PersistenceEngine` performance with `KryoSerializer` #44113

dongjoon-hyun commented Dec 1, 2023 •

edited

Loading

dongjoon-hyun commented Dec 1, 2023 •

edited

Loading

dongjoon-hyun Dec 1, 2023 •

edited

Loading