New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SW-1755] Don't need to stop worker nodes in internal backend, spark takes care of it as it shutdowns the executors #1661
Conversation
@@ -196,7 +196,7 @@ def getOrCreate(spark, conf=None, verbose=True, pre_create_hook=None, h2o_connec | |||
|
|||
def stop(self): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is used only for explicit stop by the user, if the user does not call this stop() method in pysparkling, the shutdown hook is called and stop() method on scala side is invoked
…takes care of it as it shutdowns the executors
@@ -364,6 +364,14 @@ External backend configuration properties | |||
| | | suffix "k", "m" or "g" (e.g. | | |||
| | | ``450k``, ``3m``) | | |||
+-------------------------------------------------------+----------------+-------------------------------------+ | |||
| ``sspark.ext.h2o.external.backend.stop.timeout`` | ``10000ms`` | How long we wait before for proper | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo: sspark.ext.h2o.external.backend.stop.timeout
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
@@ -364,6 +364,14 @@ External backend configuration properties | |||
| | | suffix "k", "m" or "g" (e.g. | | |||
| | | ``450k``, ``3m``) | | |||
+-------------------------------------------------------+----------------+-------------------------------------+ | |||
| ``sspark.ext.h2o.external.backend.stop.timeout`` | ``10000ms`` | How long we wait before for proper | |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How long we wait before for proper external backend before we give up on confirmation from worker nodes.
Would it be possible to rephrase the sentence?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup, done
…takes care of it as it shutdowns the executors (#1661)
No description provided.