[MINOR][DOCS] Document minimum executor memory requirement #53017
Conversation
|
@dongjoon-hyun Could you please review? Thanks! |
There was a problem hiding this comment.
Thank you for making a PR. However, I'm not sure what is hard for users because we have a directive error message already, as you mentioned, like at least 471859200.
The lack of documentation makes it hard for users to configure off-heap memory (spark.memory.offHeap.size) along with executor memory, leading to validation failures.
org.apache.spark.SparkIllegalArgumentException:
[INVALID_EXECUTOR_MEMORY] Executor memory 1048576 must be at least 471859200.
Yes I agree that we have an error msg indicating users of this minimum requirement, But just thought it would be better if we document it as well so that users are aware of it beforehand. |
|
Got it. |
| private[spark] val EXECUTOR_MEMORY = ConfigBuilder(SparkLauncher.EXECUTOR_MEMORY) | ||
| .doc("Amount of memory to use per executor process, in MiB unless otherwise specified.") | ||
| .doc("Amount of memory to use per executor process, in MiB unless otherwise specified, " + | ||
| "and with minimum allowable value of 450M") |
There was a problem hiding this comment.
with minimum allowable value -> with the minimum value
docs/configuration.md
Outdated
| Amount of memory to use per executor process, in the same format as JVM memory strings with | ||
| a size unit suffix ("k", "m", "g" or "t") (e.g. <code>512m</code>, <code>2g</code>). | ||
| a size unit suffix ("k", "m", "g" or "t") (e.g. <code>512m</code>, <code>2g</code>, | ||
| and with a minimum allowable value of <code>450M</code>). |
There was a problem hiding this comment.
ditt. with minimum allowable value -> with the minimum value
|
Thank you for your comments. Have addressed them. |
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM. Thank you, @pratham76 . Merged to master for Apache Spark 4.2.0.
### What changes were proposed in this pull request? This PR adds documentation for the minimum allowable executor memory value (450M) to prevent configuration errors especially when using offHeap memory configurations. ### Why are the changes needed? Currently, Spark enforces a minimum executor memory of 450M as reserved memory to prevent OOMs, through validation logic in the below snippets https://github.com/apache/spark/blob/06013a90b5857b9981cedac099875e8d95b9f59e/core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala#L264 https://github.com/apache/spark/blob/06013a90b5857b9981cedac099875e8d95b9f59e/core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala#L466 However, this constraint is not documented in the configuration guide or in the code-level configuration definition. The lack of documentation makes it hard for users to configure off-heap memory (spark.memory.offHeap.size) along with executor memory, leading to validation failures. ### Does this PR introduce _any_ user-facing change? Only Doc changes ### How was this patch tested? CI ### Was this patch authored or co-authored using generative AI tooling? No Closes apache#53017 from pratham76/fix-docs. Authored-by: Pratham Manja <prathammanja@ibm.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
2 participants
What changes were proposed in this pull request?
This PR adds documentation for the minimum allowable executor memory value (450M) to prevent configuration errors especially when using offHeap memory configurations.
Why are the changes needed?
Currently, Spark enforces a minimum executor memory of 450M as reserved memory to prevent OOMs, through validation logic in the below snippets
spark/core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala
Line 264 in 1a802e3
spark/core/src/main/scala/org/apache/spark/memory/UnifiedMemoryManager.scala
Line 466 in 1a802e3
However, this constraint is not documented in the configuration guide or in the code-level configuration definition. The lack of documentation makes it hard for users to configure off-heap memory (spark.memory.offHeap.size) along with executor memory, leading to validation failures.
Does this PR introduce any user-facing change?
Only Doc changes
How was this patch tested?
CI
Was this patch authored or co-authored using generative AI tooling?
No