-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Printing variable value in notebook when I set value to a variable #17
Comments
Thanks a lot for report, I've been aware of this for a long time. The major cause is that Spark interpreter only expose its API to retrieve the last variable being set, regardless of whether its the last line of a code or not. It's possible to use deeper non-API functions to retrieve it but the gain doesn't justify the risk. Could you append it with ;"" to override it with an empty output? |
Yes I did that, but I don't know this workaround solved this issue or not. |
ISpark has much higher memory consumption than spark-shell for obvious reason (message queue, visualization etc.) Make sure you increase both --driver-memory and --executor-memory to accomodate your dataset. I doubt if display too much is the cause of out of memory: in this case the websocket will timeout first before that happens. |
Hm... Strange because I set driver-memory to 32G and the actual usage was about 23G then I started to save object file and the notebook died with oom. |
IMHO save object file is done on workers/executors. SO that's expected |
Printing variable value in notebook is very annoying when I set large data to a variable.
Actually freezes happen often.
For example:
val test = [LARGE DATA]
The text was updated successfully, but these errors were encountered: