Permalink
Browse files

SPARK-1156: allow user to login into a cluster without slaves

Reported in https://spark-project.atlassian.net/browse/SPARK-1156

The current spark-ec2 script doesn't allow user to login to a cluster without slaves. One of the issues brought by this behaviour is that when all the worker died, the user cannot even login to the cluster for debugging, etc.

Author: CodingCat <zhunansjtu@gmail.com>

Closes #58 from CodingCat/SPARK-1156 and squashes the following commits:

104af07 [CodingCat] output ERROR to stderr
9a71769 [CodingCat] do not allow user to start 0-slave cluster
24a7c79 [CodingCat] allow user to login into a cluster without slaves
  • Loading branch information...
1 parent cda381f commit 3eb009f362993dbe43028419c2d48011111a200d @CodingCat CodingCat committed with mateiz Mar 6, 2014
Showing with 6 additions and 5 deletions.
  1. +6 −5 ec2/spark_ec2.py
View
@@ -398,15 +398,13 @@ def get_existing_cluster(conn, opts, cluster_name, die_on_error=True):
if any((master_nodes, slave_nodes)):
print ("Found %d master(s), %d slaves" %
(len(master_nodes), len(slave_nodes)))
- if (master_nodes != [] and slave_nodes != []) or not die_on_error:
+ if master_nodes != [] or not die_on_error:
return (master_nodes, slave_nodes)
else:
if master_nodes == [] and slave_nodes != []:
- print "ERROR: Could not find master in group " + cluster_name + "-master"
- elif master_nodes != [] and slave_nodes == []:
- print "ERROR: Could not find slaves in group " + cluster_name + "-slaves"
+ print >> sys.stderr, "ERROR: Could not find master in group " + cluster_name + "-master"
else:
- print "ERROR: Could not find any existing cluster"
+ print >> sys.stderr, "ERROR: Could not find any existing cluster"
sys.exit(1)
@@ -680,6 +678,9 @@ def real_main():
opts.zone = random.choice(conn.get_all_zones()).name
if action == "launch":
+ if opts.slaves <= 0:
+ print >> sys.stderr, "ERROR: You have to start at least 1 slave"
+ sys.exit(1)
if opts.resume:
(master_nodes, slave_nodes) = get_existing_cluster(
conn, opts, cluster_name)

0 comments on commit 3eb009f

Please sign in to comment.