Add SSH and SCP command utility and refactor shell utility #10658

jiacheliu3 · 2019-12-19T01:03:25Z

This change adds a wrapper to SSH executing a command to ShellUtils, together with SCP commands.

ShellUtils class refactored with inner classed extracted to standalone classes.

PR #10596 relies on this utility as it extensively invokes commands over SSH.

alluxio-bot · 2019-12-19T01:03:30Z

Automated checks report:

AmplabJenkins build check: PENDING
- We were not able to detect AmplabJenkins test results on this PR. Status will update when testing completes.
Commits associated with Github account: PASS
PR title follows the conventions: PASS

Some checks failed. Please fix the reported issues and reply 'alluxio-bot, check this please' to re-run checks.

AmplabJenkins · 2019-12-19T01:07:12Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-12-19T01:07:13Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7356/
Test FAILed.

AmplabJenkins · 2019-12-19T01:23:42Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-12-19T01:23:44Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7357/
Test FAILed.

AmplabJenkins · 2019-12-19T01:32:46Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-12-19T01:32:47Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7358/
Test FAILed.

AmplabJenkins · 2019-12-19T01:41:17Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-12-19T01:41:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7359/
Test FAILed.

AmplabJenkins · 2019-12-19T01:59:24Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-12-19T01:59:26Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7361/
Test FAILed.

AmplabJenkins · 2019-12-19T02:07:47Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-12-19T02:07:48Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7362/
Test FAILed.

AmplabJenkins · 2019-12-19T19:55:40Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-12-19T19:56:30Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7369/
Test FAILed.

AmplabJenkins · 2019-12-19T21:34:46Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-12-19T21:34:48Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7372/
Test FAILed.

jiacheliu3 · 2019-12-19T22:54:16Z

core/common/src/main/java/alluxio/util/ShellUtils.java

-  private static final class Command {
-    private String[] mCommand;
+  private static class Command {
+    protected String[] mCommand;


Changed to protected to be inherited

core/common/src/main/java/alluxio/util/ShellUtils.java

jiacheliu3 · 2019-12-19T22:55:19Z

core/common/src/main/java/alluxio/util/ShellUtils.java

@@ -148,7 +149,7 @@ private Command(String[] execString) {
     * @return the output
     * @throws ExitCodeException if the command returns a non-zero exit code
     */
-    private String run() throws ExitCodeException, IOException {
+    protected String run() throws ExitCodeException, IOException {


Enable inheritance

jiacheliu3 · 2019-12-19T22:57:50Z

core/common/src/main/java/alluxio/util/ShellUtils.java

+     *
+     * @return {@link CommandReturn} object representation of stdout, stderr and exit code
+     */
+    protected CommandReturn runTolerateFailure() throws IOException {


This method is similar to run().

The differences are:

Not redirecting stderr to stdout.

Keeping exitCode, stderr and stdout separated and wrapped in an object CommandReturn.

Not throwing exception when exitcode is non-zero. Return all information instead.

AmplabJenkins · 2020-01-03T01:17:24Z

Merged build finished. Test FAILed.

AmplabJenkins · 2020-01-03T01:17:25Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7447/
Test FAILed.

AmplabJenkins · 2020-01-03T01:25:35Z

Merged build finished. Test FAILed.

AmplabJenkins · 2020-01-03T01:25:36Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7448/
Test FAILed.

AmplabJenkins · 2020-01-03T01:55:01Z

Merged build finished. Test PASSed.

AmplabJenkins · 2020-01-03T01:55:03Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7449/
Test PASSed.

AmplabJenkins · 2020-01-03T06:20:24Z

Merged build finished. Test PASSed.

AmplabJenkins · 2020-01-03T06:20:26Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7451/
Test PASSed.

gpang

@jiacheliu3 Thanks! I left a few comments.

core/common/src/main/java/alluxio/util/ShellUtils.java

core/common/src/main/java/alluxio/shell/ShellCommand.java

core/common/src/main/java/alluxio/shell/SshCommand.java

core/common/src/main/java/alluxio/shell/ScpCommand.java

core/common/src/main/java/alluxio/shell/ShellCommand.java

AmplabJenkins · 2020-01-03T22:54:20Z

Merged build finished. Test FAILed.

AmplabJenkins · 2020-01-03T22:54:22Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7462/
Test FAILed.

gpang

@jiacheliu3 Just a minor comment, then looks good.

core/common/src/main/java/alluxio/util/ShellUtils.java

AmplabJenkins · 2020-01-03T23:50:55Z

Merged build finished. Test PASSed.

AmplabJenkins · 2020-01-03T23:50:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7465/
Test PASSed.

AmplabJenkins · 2020-01-04T00:08:17Z

Merged build finished. Test PASSed.

AmplabJenkins · 2020-01-04T00:08:18Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7468/
Test PASSed.

gpang

LGTM

gpang · 2020-01-07T20:00:27Z

alluxio-bot, merge this please

add ssh execution

3b0c9f0

better test and access modifier

d669950

jiacheliu3 added 2 commits December 18, 2019 17:29

checkstyle

f6b4826

testfile checkstyle

e120069

fix style

4afa9b8

fix checkstyle link

546a530

field modifiers

49da48a

close resources properly

fdfe3c6

resolved findbug error and tests

d1c73de

jiacheliu3 commented Dec 19, 2019

View reviewed changes

core/common/src/main/java/alluxio/util/ShellUtils.java Outdated Show resolved Hide resolved

jiacheliu3 commented Dec 19, 2019

View reviewed changes

jiacheliu3 added 2 commits January 2, 2020 17:11

redirect stderr to stdout

861a634

format errors

d484e44

more format

8749601

comments

efac11e

jiacheliu3 changed the title ~~Add SSH execution utility and refactor shell utility~~ Add SSH and SCP command utility and refactor shell utility Jan 3, 2020

jiacheliu3 requested a review from gpang January 3, 2020 05:55

gpang suggested changes Jan 3, 2020

View reviewed changes

resolve review comments

625d07f

whitespace issue

6940dd5

jiacheliu3 requested a review from gpang January 3, 2020 23:10

gpang suggested changes Jan 3, 2020

View reviewed changes

core/common/src/main/java/alluxio/util/ShellUtils.java Outdated Show resolved Hide resolved

rename methods

caf243a

jiacheliu3 requested a review from gpang January 3, 2020 23:44

gpang approved these changes Jan 6, 2020

View reviewed changes

alluxio-bot merged commit dcdb571 into Alluxio:master Jan 7, 2020

Add SSH and SCP command utility and refactor shell utility #10658

Add SSH and SCP command utility and refactor shell utility #10658

Conversation

jiacheliu3 commented Dec 19, 2019 • edited

alluxio-bot commented Dec 19, 2019

AmplabJenkins commented Dec 19, 2019

AmplabJenkins commented Dec 19, 2019

AmplabJenkins commented Dec 19, 2019

AmplabJenkins commented Dec 19, 2019

AmplabJenkins commented Dec 19, 2019

AmplabJenkins commented Dec 19, 2019

AmplabJenkins commented Dec 19, 2019

AmplabJenkins commented Dec 19, 2019

AmplabJenkins commented Dec 19, 2019

AmplabJenkins commented Dec 19, 2019

AmplabJenkins commented Dec 19, 2019

AmplabJenkins commented Dec 19, 2019

AmplabJenkins commented Dec 19, 2019

AmplabJenkins commented Dec 19, 2019

AmplabJenkins commented Dec 19, 2019

AmplabJenkins commented Dec 19, 2019

jiacheliu3 Dec 19, 2019

Choose a reason for hiding this comment

jiacheliu3 Dec 19, 2019

Choose a reason for hiding this comment

jiacheliu3 Dec 19, 2019

Choose a reason for hiding this comment

AmplabJenkins commented Jan 3, 2020

AmplabJenkins commented Jan 3, 2020

AmplabJenkins commented Jan 3, 2020

AmplabJenkins commented Jan 3, 2020

AmplabJenkins commented Jan 3, 2020

AmplabJenkins commented Jan 3, 2020

AmplabJenkins commented Jan 3, 2020

AmplabJenkins commented Jan 3, 2020

gpang left a comment

Choose a reason for hiding this comment

AmplabJenkins commented Jan 3, 2020

AmplabJenkins commented Jan 3, 2020

gpang left a comment

Choose a reason for hiding this comment

AmplabJenkins commented Jan 3, 2020

AmplabJenkins commented Jan 3, 2020

AmplabJenkins commented Jan 4, 2020

AmplabJenkins commented Jan 4, 2020

gpang left a comment

Choose a reason for hiding this comment

gpang commented Jan 7, 2020

jiacheliu3 commented Dec 19, 2019 •

edited