[SPARK-58513][CORE] UnifiedMemoryManager wrongly reports INVALID_DRIVER_MEMORY on the executor side by AngersZhuuuu · Pull Request #57716 · apache/spark

AngersZhuuuu · 2026-08-03T10:58:33Z

What changes were proposed in this pull request?

Make UnifiedMemoryManager report role-appropriate errors when the JVM heap is too small:

SparkEnv passes the process role (isDriver) to UnifiedMemoryManager.
The system memory check throws INVALID_DRIVER_MEMORY.SYSTEM_MEMORY on the driver and the new INVALID_EXECUTOR_MEMORY.SYSTEM_MEMORY on executors, each pointing to its own memory option.
The SPARK-12759 fail-fast check of spark.executor.memory is unchanged and runs on both roles, now as INVALID_EXECUTOR_MEMORY.CONFIG_MEMORY.
The two error conditions are restructured with sub-conditions: INVALID_DRIVER_MEMORY.SYSTEM_MEMORY, INVALID_EXECUTOR_MEMORY.{CONFIG_MEMORY, SYSTEM_MEMORY}.

Why are the changes needed?

When spark.executor.memory is set to 500m, the executor may fail with:

Exception in thread "main" java.lang.IllegalArgumentException: System memory 466092032 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration.

The message is misleading: the check runs in the executor, whose heap is controlled by spark.executor.memory, but it reports driver configuration guidance.

Does this PR introduce any user-facing change?

Yes, error reporting only; no pass/fail behavior change.

An executor with insufficient heap now fails with INVALID_EXECUTOR_MEMORY.SYSTEM_MEMORY, pointing to --executor-memory / spark.executor.memory, instead of INVALID_DRIVER_MEMORY.
Existing conditions become sub-conditions: INVALID_DRIVER_MEMORY -> INVALID_DRIVER_MEMORY.SYSTEM_MEMORY, INVALID_EXECUTOR_MEMORY -> INVALID_EXECUTOR_MEMORY.CONFIG_MEMORY; messages gain an "Insufficient driver/executor memory:" prefix.

How was this patch tested?

Added and updated UTs in UnifiedMemoryManagerSuite; error definitions are validated by SparkThrowableSuite.

Was this patch authored or co-authored using generative AI tooling?

Yes.
Generated-by: Claude Code (Claude Fable 5)

…nfig in driver , check executor config in executor

uros-b · 2026-08-03T11:46:55Z

Thank you @AngersZhuuuu!

AngersZhuuuu · 2026-08-04T02:27:29Z

This fix have a problem that may cause gluten failed to init, since we change teh method's parameter. So there have two way to fix this:

Add a compatible method and set its default value to true.

def apply(conf: SparkConf, numCores: Int): UnifiedMemoryManager = {
    apply(conf, numCores, isDriver = true)
  }

  def apply(
      conf: SparkConf,
      numCores: Int,
      isDriver: Boolean = true): UnifiedMemoryManager = {
      isDriver: Boolean): UnifiedMemoryManager = {
    val maxMemory = getMaxMemory(conf, isDriver)
    new UnifiedMemoryManager(
      conf,
      maxHeapMemory = maxMemory,
      onHeapStorageRegionSize =
        (maxMemory * conf.get(config.MEMORY_STORAGE_FRACTION)).toLong,
      numCores = numCores)
  }

add

// Compatible with legacy calls, such as Gluten
def apply(conf: SparkConf, numCores: Int): UnifiedMemoryManager = {
  applyInternal(conf, numCores, None)
}


// Spark internally explicitly passes in roles
private def applyInternal(
    conf: SparkConf,
    numCores: Int,
    isDriver: Option[Boolean]): UnifiedMemoryManager = {
  val maxMemory = getMaxMemory(conf, isDriver)
  ...
}

isDriver match {
  case Some(true) =>
    // Only check Driver memory

  case Some(false) =>
    // Only check Executor memory

  case None =>
    // both check：Driver & Executor 
}

Which one do you think is better? @uros-b

shrirangmhalgi

Thankyou @AngersZhuuuu for working on this. LGTM.

What I Verified

SparkContext.isDriver(executorId) is DRIVER_IDENTIFIER == executorId (string constant comparison, no SparkContext instance needed)
apply(conf, numCores) 2-arg overload preserved -- delegates to isDriver = None which runs both checks, identical to current behavior
isDriver = Some(true) → driver check only; Some(false) → executor check only
HashedRelation.scala uses new UnifiedMemoryManager(...) directly (bypasses apply()/getMaxMemory()) -- unaffected by this change

Re: Gluten compatibility question

The existing apply(conf, numCores) signature is preserved with identical semantics - it delegates to isDriver = None which runs both checks, exactly as before. External callers (Gluten, etc.) will continue to work without changes. Neither Option 1 nor Option 2 is needed -- the current code already handles it.

nit: PR title has "Memeory". Could you please change it to "Memory".

pan3793 · 2026-08-06T10:26:41Z

    val minSystemMemory = (reservedMemory * 1.5).ceil.toLong
-    if (systemMemory < minSystemMemory) {
+    val checkDriverMemory = isDriver.isEmpty || isDriver.contains(true)
+    val checkExecutorMemory = isDriver.isEmpty || isDriver.contains(false)


This reverts the SPARK-12759 fail-fast: the driver no longer validates spark.executor.memory at SparkContext startup, so an undersized value now launches executors that all fail until the cluster manager gives up. The executor-memory check is cheap and correct on both roles; only the INVALID_DRIVER_MEMORY check needs gating:

if (isDriver.getOrElse(true) && systemMemory < minSystemMemory) { ... } // SPARK-12759: keep unconditional if (conf.contains(config.EXECUTOR_MEMORY)) { ... }

How about current?

pan3793

Overall this looks good to me. The wiring is consistent with the existing SparkContext.isDriver(executorId) usage already in SparkEnv (e.g. the streaming-shuffle-output-tracker selection), local mode keeps isDriver=true, and the behavior change is purely which exception/message is thrown on the executor side — no pass/fail condition changes, so no valid configuration newly fails or passes. The 2-arg apply defaulting to isDriver=true keeps every existing/test caller intact.

A few nits below. Also worth noting for the stale description: the executor still runs both checks — the runtime-heap check (the point of this PR) and the SPARK-12759 spark.executor.memory config check — same on the driver. So "validates spark.executor.memory only" isn't quite right.

pan3793 · 2026-08-07T09:59:16Z

+  def apply(
+      conf: SparkConf,
+      numCores: Int,
+      isDriver: Boolean): UnifiedMemoryManager = {


This declaration fits the 100-char limit on one line — def apply(conf: SparkConf, numCores: Int, isDriver: Boolean): UnifiedMemoryManager = { is ~88 chars. Suggest collapsing to a single line for consistency with the other one-liners in this companion object.

pan3793 · 2026-08-07T09:59:16Z

+        throw new SparkIllegalArgumentException(
+          errorClass = "INVALID_EXECUTOR_MEMORY",
+          messageParameters = Map(
+            "executorMemory" -> systemMemory.toString,


INVALID_EXECUTOR_MEMORY now means two different "memory" values depending on which branch fires. The existing SPARK-12759 check below (~line 492) reports the configured spark.executor.memory through this same error class/parameter, whereas here the executor branch passes systemMemory (the observed JVM heap, Runtime.maxMemory) as executorMemory. The rendered message ("Executor memory 466092032 must be at least 471859200") therefore shows the runtime heap rather than the 500m the user actually configured — still a bit misleading, just differently than before.

Consider a dedicated error class framed as "System memory ... increase --executor-memory" (mirroring how INVALID_DRIVER_MEMORY says "System memory"), or surface both the observed and configured values.

pan3793 · 2026-08-07T09:59:16Z

+    assert(exception.getMessage.contains("increase executor memory"))
+  }
+
+  test("SPARK-58513: driver validates executor memory") {


This test exercises pre-existing behavior, not the new code path. With systemMemory=1MB > minSystemMemory, the first check passes and this hits the pre-existing SPARK-12759 config check — effectively a duplicate of the existing "insufficient executor memory" test (line 243) but with explicit isDriver=true. The test that actually validates the new behavior is "SPARK-58513: executor validates executor heap" above.

Consider dropping this one, or repurposing it (e.g. assert the driver small-heap path emits "increase heap size", which is distinct from the executor message).

pan3793

LGTM. @uros-b @shrirangmhalgi, do you want to have another look? since things get changed after your approval

pan3793 · 2026-08-10T03:46:41Z

@AngersZhuuuu, the PySpark CI failure is unrelated, could you refresh the PR description to match? It still describes the previous revision.

AngersZhuuuu · 2026-08-10T03:57:16Z

@AngersZhuuuu, the PySpark CI failure is unrelated, could you refresh the PR description to match? It still describes the previous revision.

Done

Fold INVALID_EXECUTOR_SYSTEM_MEMORY into INVALID_EXECUTOR_MEMORY as SYSTEM_MEMORY/CONFIG_MEMORY sub-conditions, and restructure INVALID_DRIVER_MEMORY with a single SYSTEM_MEMORY sub-condition. Update throw sites and tests accordingly. Assisted-by: Claude Fable 5

…ER_MEMORY on the executor side ### What changes were proposed in this pull request? Make `UnifiedMemoryManager` report role-appropriate errors when the JVM heap is too small: - `SparkEnv` passes the process role (`isDriver`) to `UnifiedMemoryManager`. - The system memory check throws `INVALID_DRIVER_MEMORY.SYSTEM_MEMORY` on the driver and the new `INVALID_EXECUTOR_MEMORY.SYSTEM_MEMORY` on executors, each pointing to its own memory option. - The SPARK-12759 fail-fast check of `spark.executor.memory` is unchanged and runs on both roles, now as `INVALID_EXECUTOR_MEMORY.CONFIG_MEMORY`. - The two error conditions are restructured with sub-conditions: `INVALID_DRIVER_MEMORY.SYSTEM_MEMORY`, `INVALID_EXECUTOR_MEMORY.{CONFIG_MEMORY, SYSTEM_MEMORY}`. ### Why are the changes needed? When `spark.executor.memory` is set to 500m, the executor may fail with: ``` Exception in thread "main" java.lang.IllegalArgumentException: System memory 466092032 must be at least 471859200. Please increase heap size using the --driver-memory option or spark.driver.memory in Spark configuration. ``` The message is misleading: the check runs in the executor, whose heap is controlled by `spark.executor.memory`, but it reports driver configuration guidance. ### Does this PR introduce _any_ user-facing change? Yes, error reporting only; no pass/fail behavior change. - An executor with insufficient heap now fails with `INVALID_EXECUTOR_MEMORY.SYSTEM_MEMORY`, pointing to `--executor-memory` / `spark.executor.memory`, instead of `INVALID_DRIVER_MEMORY`. - Existing conditions become sub-conditions: `INVALID_DRIVER_MEMORY` -> `INVALID_DRIVER_MEMORY.SYSTEM_MEMORY`, `INVALID_EXECUTOR_MEMORY` -> `INVALID_EXECUTOR_MEMORY.CONFIG_MEMORY`; messages gain an "Insufficient driver/executor memory:" prefix. ### How was this patch tested? Added and updated UTs in `UnifiedMemoryManagerSuite`; error definitions are validated by `SparkThrowableSuite`. ### Was this patch authored or co-authored using generative AI tooling? Yes. Generated-by: Claude Code (Claude Fable 5) Closes #57716 from AngersZhuuuu/SPARK-58513. Lead-authored-by: Angerszhuuuu <angers.zhu@gmail.com> Co-authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Cheng Pan <chengpan@apache.org> (cherry picked from commit f2815e0) Signed-off-by: Cheng Pan <chengpan@apache.org>

pan3793 · 2026-08-10T08:59:18Z

Merge Summary:

merged into master f2815e0
merged into branch-4.x 1983dd5
merged into branch-4.3 c252290

Posted by merge_spark_pr.py

AngersZhuuuu added 2 commits August 3, 2026 18:52

[SPARK-58513][CORE] Init UnifiedMemeoryManager should check driver co…

2299270

…nfig in driver , check executor config in executor

Update UnifiedMemoryManagerSuite.scala

66d311d

uros-b approved these changes Aug 3, 2026

View reviewed changes

Update UnifiedMemoryManager.scala

d72521b

update

3ac4643

shrirangmhalgi approved these changes Aug 5, 2026

View reviewed changes

AngersZhuuuu changed the title ~~[SPARK-58513][CORE] Init UnifiedMemeoryManager should check driver config in driver , check executor config in executor~~ [SPARK-58513][CORE] Init UnifiedMemoryManager should check driver config in driver , check executor config in executor Aug 5, 2026

pan3793 reviewed Aug 6, 2026

View reviewed changes

Update

b89e33e

AngersZhuuuu changed the title ~~[SPARK-58513][CORE] Init UnifiedMemoryManager should check driver config in driver , check executor config in executor~~ [SPARK-58513][CORE] Init UnifiedMemoryManager should only check executor config in executor Aug 7, 2026

update

193c48c

pan3793 reviewed Aug 7, 2026

View reviewed changes

AngersZhuuuu added 2 commits August 7, 2026 18:03

Update UnifiedMemoryManager.scala

4747cb1

update

95fe15a

pan3793 approved these changes Aug 7, 2026

View reviewed changes

pan3793 changed the title ~~[SPARK-58513][CORE] Init UnifiedMemoryManager should only check executor config in executor~~ [SPARK-58513][CORE] UnifiedMemoryManager wrongly reports INVALID_DRIVER_MEMORY on the executor side Aug 10, 2026

pan3793 mentioned this pull request Aug 10, 2026

Restructure memory error conditions into sub-conditions AngersZhuuuu/spark#7

Merged

pan3793 closed this in f2815e0 Aug 10, 2026

Navigation Menu

Uh oh!

[SPARK-58513][CORE] UnifiedMemoryManager wrongly reports INVALID_DRIVER_MEMORY on the executor side - #57716

[SPARK-58513][CORE] UnifiedMemoryManager wrongly reports INVALID_DRIVER_MEMORY on the executor side#57716
AngersZhuuuu wants to merge 9 commits into
apache:masterfrom
AngersZhuuuu:SPARK-58513

AngersZhuuuu commented Aug 3, 2026 •

edited by pan3793

Loading

Uh oh!

uros-b commented Aug 3, 2026

Uh oh!

AngersZhuuuu commented Aug 4, 2026

Uh oh!

shrirangmhalgi left a comment

Uh oh!

pan3793 Aug 6, 2026

Uh oh!

AngersZhuuuu Aug 7, 2026

Uh oh!

pan3793 left a comment •

edited

Loading

Uh oh!

pan3793 Aug 7, 2026

Uh oh!

pan3793 Aug 7, 2026

Uh oh!

pan3793 Aug 7, 2026

Uh oh!

pan3793 left a comment

Uh oh!

pan3793 commented Aug 10, 2026

Uh oh!

AngersZhuuuu commented Aug 10, 2026

Uh oh!

pan3793 commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

AngersZhuuuu commented Aug 3, 2026 • edited by pan3793 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

uros-b commented Aug 3, 2026

Uh oh!

AngersZhuuuu commented Aug 4, 2026

Uh oh!

shrirangmhalgi left a comment

Choose a reason for hiding this comment

What I Verified

Re: Gluten compatibility question

Uh oh!

pan3793 Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

AngersZhuuuu Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

pan3793 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

pan3793 Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

pan3793 Aug 7, 2026

Choose a reason for hiding this comment

Uh oh!

pan3793 left a comment

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Aug 10, 2026

Uh oh!

AngersZhuuuu commented Aug 10, 2026

Uh oh!

pan3793 commented Aug 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

AngersZhuuuu commented Aug 3, 2026 •

edited by pan3793

Loading

pan3793 left a comment •

edited

Loading