[SPARK-3584] sbin/slaves doesn't work when we use password authentication for SSH #2444

sarutak · 2014-09-18T09:16:23Z

No description provided.

Modified sbin/slaves to choose localhost as a default host list Renamed conf/slaves to conf/slaves.template Added entries about slaves and slaves.template to .rat-excludes Added entries about slaves to .gitignore

SparkQA · 2014-09-18T09:19:17Z

QA tests have started for PR 2444 at commit 297e75d.

This patch merges cleanly.

SparkQA · 2014-09-18T10:25:57Z

QA tests have finished for PR 2444 at commit 297e75d.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

mattf · 2014-09-18T14:42:48Z

sbin/slaves.sh

  else
-    export HOSTLIST="${SPARK_SLAVES}"
+    HOSTLIST=`cat "${SPARK_SLAVES}"`


why cat here and echo later?

This is to use HOSTLIST as List of Host, not file.
It's to use localhost as a default host list entry.

thanks for pointing that out. i didn't read closely enough.

mattf · 2014-09-19T12:45:21Z

+1 lgtm

pwendell · 2014-09-19T18:02:13Z

sbin/slaves.sh

+    if [ -f "${SPARK_CONF_DIR}/slaves" ]; then
+      HOSTLIST=`cat "${SPARK_CONF_DIR}/slaves"`
+    else
+      HOSTLIST=localhost


We should change the docs in spark-standalone.md to explain two new features:

You can set SSH_FOREGROUND if you cannot use paswordless SSH (currently, it says this is required).

If there is no slaves file in existence, it will launch a single slave at localhost by default.

yes, i was moving too quickly this morning. definitely need something to allow for background ssh.

O.K, I'll add SSH_FOREGROUND variable and add description.

pwendell · 2014-09-19T18:03:25Z

Made some comments. We need to guard this with a config parameter because otherwise it will regress behavior on large clusters where serial vs parallel ssh makes a big difference.

…ts-modification

SparkQA · 2014-09-22T04:54:18Z

QA tests have started for PR 2444 at commit 1bba8a9.

This patch merges cleanly.

SparkQA · 2014-09-22T05:34:24Z

QA tests have started for PR 2444 at commit 7120a0c.

This patch merges cleanly.

SparkQA · 2014-09-22T05:49:29Z

QA tests have started for PR 2444 at commit e570431.

This patch merges cleanly.

SparkQA · 2014-09-22T06:01:50Z

QA tests have finished for PR 2444 at commit 1bba8a9.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-22T06:42:37Z

QA tests have finished for PR 2444 at commit 7120a0c.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2014-09-22T06:56:46Z

QA tests have finished for PR 2444 at commit e570431.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

pwendell · 2014-09-25T06:25:14Z

sbin/slaves.sh

-   sleep $SPARK_SLAVE_SLEEP
- fi
+for slave in `echo "$HOSTLIST"|sed  "s/#.*$//;/^$/d"`; do
+  if [ "${SPARK_SSH_FOREGROUND}" = "y" ] || [ "${SPARK_SSH_FOREGROUND}" = "yes" ]; then


Typically for these types of options we just check whether it's defined or not. For example elsewhere we do:

if [ -n "$SPARK_PRINT_LAUNCH_COMMAND" ]; then

Can you make it consistent with that?

pwendell · 2014-09-25T06:25:57Z

This looks good, just had a minor comment, then I think it's ready to merge.

…ts-modification

…ROUND" as a flag

sarutak · 2014-09-25T10:41:16Z

Thanks @pwendell , I've modified what you mentioned.

SparkQA · 2014-09-25T10:44:19Z

QA tests have started for PR 2444 at commit 7858225.

This patch merges cleanly.

SparkQA · 2014-09-25T11:52:21Z

QA tests have finished for PR 2444 at commit 7858225.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-09-25T11:52:24Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20799/

mattf · 2014-09-25T12:08:31Z

docs/spark-standalone.md

+does not exist, the launch scripts use a list which contains single hostname `localhost`. This can be used for testing.
+The master machine must be able to access each of the slave machines via `ssh`. By default, `ssh` is executed in the background for parallel execution for each slave machine.
+If you would like to use password authentication instead of password-less(using a private key) for `ssh`, `ssh` does not work well in the background.
+To avoid this, you can set a environment variable `SPARK_SSH_FOREGROUND` to something like `yes` or `y` to execute `ssh` in the foreground.



what about -

To launch a Spark standalone cluster with the launch scripts, you should create a file called conf/slaves in your Spark directory, which must contain the hostnames of all the machines where you intend to start Spark workers, one per line. If conf/slaves does not exist, the launch scripts defaults to a single machine (localhost), which is useful for testing. Note, the master machine accesses each of the worker machines via ssh. By default, ssh is run in parallel and requires password-less (using a private key) access to be setup. If you do not have a password-less setup, you can set the environment variable SPARK_SSH_FOREGROUND and serially provide a password for each worker.

@mattf Thank you for reviewing. It makes sense.

…ndalone.md

SparkQA · 2014-09-25T12:39:22Z

QA tests have started for PR 2444 at commit eff7394.

This patch merges cleanly.

SparkQA · 2014-09-25T13:47:23Z

QA tests have finished for PR 2444 at commit eff7394.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2014-09-25T13:47:26Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/20805/

pwendell · 2014-09-25T23:47:13Z

Thanks @sarutak and @mattf for the review. I'll pull this in.

jameszhouyi · 2014-09-28T01:53:47Z

Hi @pwendell ,
After this commit, for spark-perf will complain 'not found slaves' when run ./bin/run... so have to modify from slaves.template to slaves manually ?

JoshRosen · 2014-10-02T20:24:19Z

I also noticed that make-distribution.sh fails when trying to copy the now-nonexistent conf/slaves file. I'm going to push a hotfix commit to fix that.

JoshRosen · 2014-10-02T20:26:08Z

Actually, nevermind: this was fixed in #2549.

Modified sbin/slaves not to export HOSTLIST

297e75d

Modified sbin/slaves to choose localhost as a default host list Renamed conf/slaves to conf/slaves.template Added entries about slaves and slaves.template to .rat-excludes Added entries about slaves to .gitignore

mattf reviewed Sep 18, 2014
View reviewed changes

pwendell reviewed Sep 19, 2014
View reviewed changes

sarutak added 2 commits September 22, 2014 13:15

Merge branch 'master' of git://git.apache.org/spark into slaves-scrip…

88e2f17

…ts-modification

Added SPARK_SSH_FOREGROUND flag to sbin/slaves

1bba8a9

Added a description about default host for sbin/slaves

7120a0c

Added a description for SPARK_SSH_FOREGROUND variable

e570431

pwendell reviewed Sep 25, 2014
View reviewed changes

sarutak added 2 commits September 25, 2014 18:23

Merge branch 'master' of git://git.apache.org/spark into slaves-scrip…

53d7121

…ts-modification

Modified sbin/slaves to use the environment variable "SPARK_SSH_FOREG…

7858225

…ROUND" as a flag

mattf reviewed Sep 25, 2014
View reviewed changes

Improve the description about Cluster Launch Script in docs/spark-sta…

eff7394

…ndalone.md

asfgit closed this in 0dc868e Sep 25, 2014

sarutak deleted the slaves-scripts-modification branch April 11, 2015 05:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-3584] sbin/slaves doesn't work when we use password authentication for SSH #2444

[SPARK-3584] sbin/slaves doesn't work when we use password authentication for SSH #2444

sarutak commented Sep 18, 2014

SparkQA commented Sep 18, 2014

SparkQA commented Sep 18, 2014

mattf Sep 18, 2014

sarutak Sep 19, 2014

mattf Sep 19, 2014

mattf commented Sep 19, 2014

pwendell Sep 19, 2014

mattf Sep 19, 2014

sarutak Sep 21, 2014

pwendell commented Sep 19, 2014

SparkQA commented Sep 22, 2014

SparkQA commented Sep 22, 2014

SparkQA commented Sep 22, 2014

SparkQA commented Sep 22, 2014

SparkQA commented Sep 22, 2014

SparkQA commented Sep 22, 2014

pwendell Sep 25, 2014

pwendell commented Sep 25, 2014

sarutak commented Sep 25, 2014

SparkQA commented Sep 25, 2014

SparkQA commented Sep 25, 2014

AmplabJenkins commented Sep 25, 2014

mattf Sep 25, 2014

sarutak Sep 25, 2014

SparkQA commented Sep 25, 2014

SparkQA commented Sep 25, 2014

AmplabJenkins commented Sep 25, 2014

pwendell commented Sep 25, 2014

jameszhouyi commented Sep 28, 2014

JoshRosen commented Oct 2, 2014

JoshRosen commented Oct 2, 2014

[SPARK-3584] sbin/slaves doesn't work when we use password authentication for SSH #2444

[SPARK-3584] sbin/slaves doesn't work when we use password authentication for SSH #2444

Conversation

sarutak commented Sep 18, 2014

SparkQA commented Sep 18, 2014

SparkQA commented Sep 18, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mattf commented Sep 19, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pwendell commented Sep 19, 2014

SparkQA commented Sep 22, 2014

SparkQA commented Sep 22, 2014

SparkQA commented Sep 22, 2014

SparkQA commented Sep 22, 2014

SparkQA commented Sep 22, 2014

SparkQA commented Sep 22, 2014

Choose a reason for hiding this comment

pwendell commented Sep 25, 2014

sarutak commented Sep 25, 2014

SparkQA commented Sep 25, 2014

SparkQA commented Sep 25, 2014

AmplabJenkins commented Sep 25, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Sep 25, 2014

SparkQA commented Sep 25, 2014

AmplabJenkins commented Sep 25, 2014

pwendell commented Sep 25, 2014

jameszhouyi commented Sep 28, 2014

JoshRosen commented Oct 2, 2014

JoshRosen commented Oct 2, 2014