From 8f360b97e9f0afff7e6627df15cc17f103a5c2b3 Mon Sep 17 00:00:00 2001 From: Huang Tengfei Date: Tue, 28 Aug 2018 15:28:06 +0800 Subject: [PATCH 1/3] update configuration.md, correct the default unit of spark.executor.memory and spark.driver.memory. Change-Id: I2935b93559aa3e016bbab7a083c8c24bbdc6f685 --- docs/configuration.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 0270dc2cfaf45..2f46db8cced22 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -152,7 +152,7 @@ of the most common options to set are: spark.driver.memory 1g - Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in MiB + Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in bytes unless otherwise specified (e.g. 1g, 2g).
Note: In client mode, this config must not be set through the SparkConf @@ -175,7 +175,7 @@ of the most common options to set are: spark.executor.memory 1g - Amount of memory to use per executor process, in MiB unless otherwise specified. + Amount of memory to use per executor process, in bytes unless otherwise specified. (e.g. 2g, 8g). From 3eb3b66a52435366f258cbb40d01d8f3b0141bff Mon Sep 17 00:00:00 2001 From: huangtengfei02 Date: Tue, 28 Aug 2018 16:29:25 +0800 Subject: [PATCH 2/3] fix style issue Change-Id: I73e82b8bd07064d874bf76df52cb661097532884 --- docs/configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/configuration.md b/docs/configuration.md index 2f46db8cced22..31ec89d98a4b4 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -152,7 +152,7 @@ of the most common options to set are: spark.driver.memory 1g - Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in bytes + Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in bytes unless otherwise specified (e.g. 1g, 2g).
Note: In client mode, this config must not be set through the SparkConf From 294ee6d630f1758f78f6496317e744455746527f Mon Sep 17 00:00:00 2001 From: huangtengfei02 Date: Thu, 30 Aug 2018 15:27:37 +0800 Subject: [PATCH 3/3] update docs Change-Id: I60994d4e2904e2c2c9f548fa7e6b3c8c1e027077 --- docs/configuration.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/configuration.md b/docs/configuration.md index 31ec89d98a4b4..951bdefe6ddfc 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -152,8 +152,9 @@ of the most common options to set are: spark.driver.memory 1g - Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in bytes - unless otherwise specified (e.g. 1g, 2g). + Amount of memory to use for the driver process, i.e. where SparkContext is initialized, in the + same format as JVM memory strings with a size unit suffix ("k", "m", "g" or "t") + (e.g. 512m, 2g).
Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. @@ -175,8 +176,8 @@ of the most common options to set are: spark.executor.memory 1g - Amount of memory to use per executor process, in bytes unless otherwise specified. - (e.g. 2g, 8g). + Amount of memory to use per executor process, in the same format as JVM memory strings with + a size unit suffix ("k", "m", "g" or "t") (e.g. 512m, 2g).