-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-6120] [mllib] Warnings about memory in tree, ensemble model save #4864
Conversation
… with too small a Java heap size
Test build #28211 has started for PR 4864 at commit
|
// SPARK-6120: We do a hacky check here so users understand why save() is failing | ||
// when they run the ML guide example. | ||
// TODO: Fix this issue for real. | ||
val driverMemory = sc.getConf.getOption("spark.driver.memory") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Move this into the the if (sc.isLocal)
branch.
Test build #28213 has started for PR 4864 at commit
|
Test build #28213 has finished for PR 4864 at commit
|
Test FAILed. |
Test build #28211 has finished for PR 4864 at commit
|
Test PASSed. |
Issue: When the Python DecisionTree example in the programming guide is run, it runs out of Java Heap Space when using the default memory settings for the spark shell. This prints a warning. CC: mengxr Author: Joseph K. Bradley <joseph@databricks.com> Closes #4864 from jkbradley/dt-save-heap and squashes the following commits: 02e8daf [Joseph K. Bradley] fixed based on code review 7ecb1ed [Joseph K. Bradley] Added warnings about memory when calling tree and ensemble model save with too small a Java heap size (cherry picked from commit c2fe3a6) Signed-off-by: Xiangrui Meng <meng@databricks.com>
LGTM. Merged into master and branch-1.3. Thanks! |
Issue: When the Python DecisionTree example in the programming guide is run, it runs out of Java Heap Space when using the default memory settings for the spark shell.
This prints a warning.
CC: @mengxr