[SPARK-3398] [EC2] Have spark-ec2 intelligently wait for specific cluster states #2339

nchammas · 2014-09-09T23:17:56Z

Instead of waiting arbitrary amounts of time for the cluster to reach a specific state, this patch lets spark-ec2 explicitly wait for a cluster to reach a desired state.

This is useful in a couple of situations:

The cluster is launching and you want to wait until SSH is available before installing stuff.
The cluster is being terminated and you want to wait until all the instances are terminated before trying to delete security groups.

This patch removes the need for the --wait option and removes some of the time-based retry logic that was being used.

nchammas · 2014-09-09T23:21:52Z

Depending on what the reviewers think, there are some additional lines that can be removed, like:

the wait_for_cluster function
the wait_for_instances function
the --wait option

SparkQA · 2014-09-09T23:51:52Z

QA tests have started for PR 2339 at commit 7969265.

This patch merges cleanly.

SparkQA · 2014-09-10T00:52:50Z

QA tests have finished for PR 2339 at commit 7969265.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

nchammas · 2014-09-10T03:05:17Z

@JoshRosen and @davies: This PR is ready for review.

davies · 2014-09-10T17:52:05Z

ec2/spark_ec2.py

+            if all(i.state == 'running' for i in cluster_instances) and \
+               all(is_ssh_available(host=i.ip_address, opts=opts) for i in cluster_instances):
+                print ""  # so that next line of output starts on new line
+                return


maybe we could put "break" here, and put 'sys.stdout.write("\n")' at the end of this funciton

Makes sense. I'll do that but put print "" at the end instead of sys.stdout.write("\n"), since we want to flush the output immediately. Does that sound good to you?

The long comment sounds boring than 'sys.stdout.write("\n")'. With 'print ""', you also need to flush manually. I think we do not need to flush here, because it has no visually changes until other loggings coming in.

Either is fine to me.

Hmm, I'll get back to you on this. Definitely a minor point either way, but I'd like to get it right. Gotta run for now but will check on this later tonight.

OK, I see your point. I've made the change per your recommendation.

davies · 2014-09-10T18:08:22Z

This patch look good to me, just some minor comments.

I think we should keep --wait option (do not break user) and deprecate it. The dead code ( wait_for_cluster and wait_for_instances) should be removed.

@JoshRosen how do you think?

nchammas · 2014-09-10T19:03:22Z

Alright, I've updated things per all the feedback I've gotten with one minor exception, which I will revisit later tonight.

nchammas · 2014-09-10T21:19:48Z

Jenkins, retest this please.

SparkQA · 2014-09-10T21:49:29Z

QA tests have started for PR 2339 at commit 26c5ed0.

This patch merges cleanly.

SparkQA · 2014-09-10T21:53:00Z

QA tests have started for PR 2339 at commit 26c5ed0.

This patch merges cleanly.

SparkQA · 2014-09-10T22:58:32Z

QA tests have finished for PR 2339 at commit 26c5ed0.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-10T23:01:56Z

QA tests have finished for PR 2339 at commit 26c5ed0.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-10T23:53:20Z

QA tests have started for PR 2339 at commit 9a9e035.

This patch merges cleanly.

SparkQA · 2014-09-11T00:53:34Z

QA tests have finished for PR 2339 at commit 9a9e035.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

nchammas · 2014-09-11T00:58:39Z

Flume test failed. My most recent commit just removed a couple of comment lines.

Jenkins, retest this please.

nchammas · 2014-09-11T01:54:00Z

Jenkins, retest this please.

SparkQA · 2014-09-11T01:54:32Z

QA tests have started for PR 2339 at commit 9a9e035.

This patch merges cleanly.

SparkQA · 2014-09-11T02:53:11Z

QA tests have started for PR 2339 at commit 9a9e035.

This patch merges cleanly.

SparkQA · 2014-09-11T02:55:09Z

QA tests have finished for PR 2339 at commit 9a9e035.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

nchammas · 2014-09-11T02:57:50Z

@davies This PR is ready for another review. I believe I've covered all the feedback given so far.

davies · 2014-09-11T03:19:21Z

@nchammas LGTM, thanks!

SparkQA · 2014-09-11T03:53:43Z

QA tests have finished for PR 2339 at commit 9a9e035.

This patch fails unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-26T00:29:40Z

QA tests have started for PR 2339 at commit 0e648bc.

This patch merges cleanly.

nchammas · 2014-09-26T00:35:45Z

I might remove some of the extra linbreaks/whitespace here.

Ah, OK, I just took care of that. An earlier version of that text would've put the line at over 100 characters, which is why it originally had those breaks in there.

SparkQA · 2014-09-26T00:39:38Z

QA tests have started for PR 2339 at commit 8b701d1.

This patch merges cleanly.

SparkQA · 2014-09-26T01:38:16Z

QA tests have finished for PR 2339 at commit 0e648bc.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-09-26T01:38:19Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20832/

SparkQA · 2014-09-26T01:46:24Z

QA tests have finished for PR 2339 at commit 8b701d1.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-09-26T01:46:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20834/

nchammas · 2014-09-26T18:39:52Z

Hey @pwendell, is there anything else you'd like to change about this PR?

nchammas · 2014-09-30T19:03:35Z

BTW @shivaram I took your suggestion of checking the status checks before checking SSH itself.

AmplabJenkins · 2014-09-30T22:52:23Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21071/

nchammas · 2014-10-01T01:55:39Z

Dunno where this most recent failure came from. The build output just shows a failure to checkout the patch.

pwendell · 2014-10-01T23:57:38Z

Jenkins, retest this please.

SparkQA · 2014-10-02T00:04:36Z

QA tests have started for PR 2339 at commit 43a69f0.

This patch merges cleanly.

SparkQA · 2014-10-02T01:13:13Z

QA tests have finished for PR 2339 at commit 43a69f0.

This patch passes unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- println(s"Failed to load main class $childMainClass.")

AmplabJenkins · 2014-10-02T01:13:17Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/21163/

nchammas · 2014-10-02T01:15:50Z

This patch adds the following public classes (experimental):
println(s"Failed to load main class $childMainClass.")

FYI: I believe I have these phantom new class notes finally sorted out in #2606.

nchammas · 2014-10-06T21:02:07Z

@pwendell Is this good to merge in?

nchammas · 2014-10-06T21:24:29Z

BTW, related side note: I just used Packer to create an example image from a template, and it looks like Packer follows a similar pattern of first waiting for the instance to come up, and then waiting for SSH to become available.

==> amazon-ebs: Launching a source AWS instance...
    amazon-ebs: Instance ID: i-1fbfcff1
==> amazon-ebs: Waiting for instance (i-1fbfcff1) to become ready...
==> amazon-ebs: Waiting for SSH to become available...
==> amazon-ebs: Connected to SSH!
==> amazon-ebs: Stopping the source instance...
==> amazon-ebs: Waiting for the instance to stop...

Just thought that was cool.

JoshRosen · 2014-10-07T23:54:13Z

This looks good to me, so I'm going to merge it. Thanks for doing this!

This PR re-introduces [0e648bc](0e648bc) from PR #2339, which somehow never made it into the codebase. Additionally, it removes a now-unnecessary linear backoff on the SSH checks since we are blocking on EC2 status checks before testing SSH. Author: Nicholas Chammas <nicholas.chammas@gmail.com> Closes #3195 from nchammas/remove-ec2-ssh-backoff and squashes the following commits: efb29e1 [Nicholas Chammas] Revert "Remove linear backoff." ef3ca99 [Nicholas Chammas] reuse conn adb4eaa [Nicholas Chammas] Remove linear backoff. 55caa24 [Nicholas Chammas] Check EC2 status checks before SSH.

nchammas added 2 commits September 9, 2014 19:13

wait for specific cluster states

126e4cf

fix long line (PEP 8)

7969265

davies reviewed Sep 10, 2014
View reviewed changes

deprecate wait option; remove dead code

bb67c06

replace print with write()

26c5ed0

remove extraneous comment

9a9e035

nchammas force-pushed the spark-ec2-wait-properly branch from 8b701d1 to 43a69f0 Compare September 30, 2014 22:41

asfgit closed this in 5912ca6 Oct 7, 2014

nchammas deleted the spark-ec2-wait-properly branch October 8, 2014 01:05

nchammas mentioned this pull request Nov 11, 2014

[SPARK-3398] [SPARK-4325] [EC2] Use EC2 status checks. #3195

Closed

[SPARK-3398] [EC2] Have spark-ec2 intelligently wait for specific cluster states #2339

[SPARK-3398] [EC2] Have spark-ec2 intelligently wait for specific cluster states #2339

Conversation

nchammas commented Sep 9, 2014

nchammas commented Sep 9, 2014

SparkQA commented Sep 9, 2014

SparkQA commented Sep 10, 2014

nchammas commented Sep 10, 2014

davies Sep 10, 2014

Choose a reason for hiding this comment

nchammas Sep 10, 2014

Choose a reason for hiding this comment

davies Sep 10, 2014

Choose a reason for hiding this comment

nchammas Sep 10, 2014

Choose a reason for hiding this comment

nchammas Sep 10, 2014

Choose a reason for hiding this comment

davies commented Sep 10, 2014

nchammas commented Sep 10, 2014

nchammas commented Sep 10, 2014

SparkQA commented Sep 10, 2014

SparkQA commented Sep 10, 2014

SparkQA commented Sep 10, 2014

SparkQA commented Sep 10, 2014

SparkQA commented Sep 10, 2014

SparkQA commented Sep 11, 2014

nchammas commented Sep 11, 2014

nchammas commented Sep 11, 2014

SparkQA commented Sep 11, 2014

SparkQA commented Sep 11, 2014

SparkQA commented Sep 11, 2014

nchammas commented Sep 11, 2014

davies commented Sep 11, 2014

SparkQA commented Sep 11, 2014

SparkQA commented Sep 26, 2014

nchammas commented Sep 26, 2014

SparkQA commented Sep 26, 2014

SparkQA commented Sep 26, 2014

AmplabJenkins commented Sep 26, 2014

SparkQA commented Sep 26, 2014

AmplabJenkins commented Sep 26, 2014

nchammas commented Sep 26, 2014

nchammas commented Sep 30, 2014

AmplabJenkins commented Sep 30, 2014

nchammas commented Oct 1, 2014

pwendell commented Oct 1, 2014

SparkQA commented Oct 2, 2014

SparkQA commented Oct 2, 2014

AmplabJenkins commented Oct 2, 2014

nchammas commented Oct 2, 2014

nchammas commented Oct 6, 2014

nchammas commented Oct 6, 2014

JoshRosen commented Oct 7, 2014