Add collectInfo command #10596

jiacheliu3 · 2019-12-09T19:43:42Z

Add Alluxio CollectInfo command set to Alluxio command line. This command will collect information on every host in the Alluxio cluster, tarball the information into a local tarball file, then gather all the tarballs back to where the command is issued.

This command is designed to help troubleshooting and collect information from Alluxio clusters. The collected information include logs, config files and it also executes a set of commands on each of the hosts and include outputs in the tarball.

Added commands:
This will run one or all checks on the local machine and write the output into files in targetDir. At the end of execution a tarball will be produced containing all the information collected, put in the targetDir as well.
bin/alluxio collectInfo --local [all/collectConfig/collectEnv/collectLog/collectMetrics/collectAlluxioInfo] targetDir

This will run the corresponding collectInfo --local command on all hosts in the Alluxio cluster defined in conf/masters and conf/workers via SSH. After that the tarballs generated by each individual infoBundle call will be SCPed into a local temp directory, and then put into one final tarball in the targetDir.
bin/alluxio collectInfo [all/collectConfig/collectEnv/collectLog/collectMetrics/collectAlluxioInfo] targetDir

Phase 2:
This PR is the Phase 1 of CollectInfo feature. This PR provides basic implementations and defines the structure.

Some features are not implemented in this PR and scheduled to a separate Phase 2 PR. These features include the following:

Skipping a command when pre-existing work are found. This is because if the cluster is large, running infoBundle can take a long time and fail in the middle.
Add a -f option to force overwrite existing work.
Add a -c or --components option to copy only certain logs.
Add a -h or --hosts option to only invoke collectInfo on certain hosts.
Add security check to avoid copying conf/ files that contain credentials.

alluxio-bot · 2019-12-09T19:43:47Z

Automated checks report:

AmplabJenkins build check: PENDING
- We were not able to detect AmplabJenkins test results on this PR. Status will update when testing completes.
Commits associated with Github account: PASS
PR title follows the conventions: PASS

Some checks failed. Please fix the reported issues and reply 'alluxio-bot, check this please' to re-run checks.

AmplabJenkins · 2019-12-09T19:59:06Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-12-09T19:59:08Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7217/
Test FAILed.

AmplabJenkins · 2019-12-10T06:01:05Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-12-10T06:01:06Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7231/
Test FAILed.

AmplabJenkins · 2019-12-11T22:46:32Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-12-11T22:46:34Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7264/
Test FAILed.

AmplabJenkins · 2019-12-16T22:36:43Z

Merged build finished. Test FAILed.

AmplabJenkins · 2019-12-16T22:36:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7300/
Test FAILed.

This change adds a wrapper to SSH executing a command to ShellUtils, together with SCP commands. `ShellUtils` class refactored with inner classed extracted to standalone classes. PR #10596 relies on this utility as it extensively invokes commands over SSH. pr-link: #10658 change-id: cid-d4d74b1d4b1aff513fe298effee509f620e5821c

AmplabJenkins · 2020-01-29T07:08:54Z

Merged build finished. Test FAILed.

AmplabJenkins · 2020-01-29T07:08:55Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7830/
Test FAILed.

AmplabJenkins · 2020-01-29T08:55:41Z

Merged build finished. Test FAILed.

AmplabJenkins · 2020-01-29T08:55:43Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7831/
Test FAILed.

AmplabJenkins · 2020-02-01T13:15:12Z

Merged build finished. Test FAILed.

AmplabJenkins · 2020-02-01T13:15:14Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/7886/
Test FAILed.

jiacheliu3 · 2020-02-12T16:41:12Z

@jiacheliu3 have you considered having a single command , and just using an option like --local for just collecting the local info? It is confusing to have 2 commands collectInfo and collectInfoAll

Good idea. Now CollectInfoAll has been merged into CollectInfo and added a --local option as the switch. @gpang PTAL at the new logic now. I may add more unit test coverage tmr.

AmplabJenkins · 2020-02-12T16:47:12Z

Merged build finished. Test FAILed.

AmplabJenkins · 2020-02-12T16:47:14Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/8150/
Test FAILed.

AmplabJenkins · 2020-02-12T17:11:14Z

Merged build finished. Test PASSed.

AmplabJenkins · 2020-02-12T17:11:15Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/8151/
Test PASSed.

gpang · 2020-02-12T22:31:04Z

@jiacheliu3 Thanks! Can you update the PR title and description with the updated syntax?

ns1123 · 2020-02-11T19:24:09Z

shell/src/main/java/alluxio/cli/bundler/command/CollectAlluxioInfoCommand.java

+            new AlluxioCommand(mAlluxioPath, "fs mount"), null);
+    registerCommand("version",
+            new AlluxioCommand(mAlluxioPath, "version -r"), null);
+    registerCommand("job",


@bradyoo is there a better api for getting JobService information instead of dumping the list of JobID's? If not, I'd rather we just collect the number of jobs instead of the jobs themselves. A list of JobId's will have almost not use to an external reviewer.

job ID's might not be immediately useful, but a string dump of the Job type and config for each would be the most useful IMO

FYI @ns1123 job ls in 2.2 prints Job Type

ns1123 · 2020-02-11T19:25:01Z

shell/src/main/java/alluxio/cli/bundler/command/CollectAlluxioInfoCommand.java

+    registerCommand("job",
+            new AlluxioCommand(mAlluxioPath, "job ls"), null);
+    registerCommand("journal",
+            new AlluxioCommand(mAlluxioPath, String.format("fs ls -R %s",


Assuming the journal is local not UFS. The journal is pretty important for our review so I'd definitely try to handle at least HDFS journal.

Added logic to check if the journal dir is local or hdfs.
Local -> ls -R journalDir
Hdfs -> hdfs dfs -ls -R journalDir
Are there any more logic that can be helpful?

This pretty much covers all scenarios that we want to.

shell/src/main/java/alluxio/cli/bundler/command/CollectEnvCommand.java

ns1123 · 2020-02-11T19:28:14Z

shell/src/main/java/alluxio/cli/bundler/command/CollectEnvCommand.java

+
+  @Override
+  protected void registerCommands() {
+    registerCommand("ps", new ShellCommand(new String[]{"ps", "-ef", "|grep alluxio*"}), null);


Do we want to also grab information about a presto/spark/yarn process if it's colocated?

Added attempts to grab Spark/Yarn/Hdfs/Presto.

ns1123 · 2020-02-11T19:29:51Z

shell/src/main/java/alluxio/cli/bundler/command/CollectJvmInfoCommand.java

+    Map<String, String> procs = new HashMap<>();
+
+    // Get Jps output
+    String[] jpsCommand = new String[]{"jps"};


Interestingly, jstack will only work if run within the same cgroup. Even sudo will not be able to jstack another user process correctly. WRT too many java processes, that is one of the pieces of information we want to be aware of. I think it's perfectly fine to grab all java processes on the machine.

shell/src/main/java/alluxio/cli/bundler/command/CollectMetricsCommand.java

jiacheliu3 · 2020-02-13T01:52:28Z

@jiacheliu3 Thanks! Can you update the PR title and description with the updated syntax?

This is done. Thanks for the catch!

bf8086

Overall looks good to me. Only a couple minor comments.

shell/src/main/java/alluxio/cli/bundler/CollectInfo.java

jiacheliu3 · 2020-02-13T10:19:37Z

@ns1123 PTAL now the CollectJvmInfoCommand attempts sudo.

AmplabJenkins · 2020-02-13T10:21:40Z

Merged build finished. Test FAILed.

AmplabJenkins · 2020-02-13T10:21:42Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/8187/
Test FAILed.

AmplabJenkins · 2020-02-13T10:30:42Z

Merged build finished. Test FAILed.

AmplabJenkins · 2020-02-13T10:30:44Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/8188/
Test FAILed.

AmplabJenkins · 2020-02-13T11:15:35Z

Merged build finished. Test PASSed.

AmplabJenkins · 2020-02-13T11:15:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Alluxio-Pull-Request-Builder/8189/
Test PASSed.

jiacheliu3 · 2020-02-13T15:23:40Z

@bf8086 @ns1123 if you have great ideas on improvements that are not low-hanging fruits (like a smarter extra command option), I would propose we leave a mark there and I can address it in the phase 2 improvement on this command. This PR is getting a little too long to keep track and keep focused on all aspects.

gpang

LGTM

ns1123 · 2020-02-13T16:57:43Z

LGTM. Thanks @jiacheliu3 for doing this! looking forward to getting to use it.

bf8086

LGTM.

calvinjia · 2020-02-13T20:00:46Z

alluxio-bot, merge this please.

jiacheliu3 changed the title ~~Log bundle command~~ [WIP] Log bundle command Dec 9, 2019

jiacheliu3 changed the title ~~[WIP] Log bundle command~~ [WIP] Info bundle command Dec 11, 2019

jiacheliu3 mentioned this pull request Dec 19, 2019

Add SSH and SCP command utility and refactor shell utility #10658

Merged

jiacheliu3 added 6 commits January 29, 2020 12:32

init commit of basic functions

42e6bbf

this compiles

f397c73

added features and compiles

c53705e

more functions and unit tests

0878687

Delegate changes to ShellUtils

dfd7d45

Apply new SSH and SCP utils

fc2d17e

jiacheliu3 force-pushed the log_bundle branch from b2f984b to fc2d17e Compare January 29, 2020 07:05

update tar utils

47cff23

jiacheliu3 added 2 commits January 30, 2020 18:12

rewrite util for generate tar.gz files

5dfec6e

passed basic unit tests

172ed2f

include unit tests

39d3973

checkstyle

0a9ff3d

ns1123 suggested changes Feb 12, 2020

View reviewed changes

jiacheliu3 changed the title ~~Add collectInfo and collectInfoAll command~~ Add collectInfo command Feb 13, 2020

bf8086 suggested changes Feb 13, 2020

View reviewed changes

shell/src/main/java/alluxio/cli/bundler/CollectInfo.java Outdated Show resolved Hide resolved

shell/src/main/java/alluxio/cli/bundler/CollectInfo.java Outdated Show resolved Hide resolved

shell/src/main/java/alluxio/cli/bundler/CollectInfo.java Outdated Show resolved Hide resolved

try sudo for jps and jstack

cd6b2c8

spotbug

418a318

correct test

7612580

jiacheliu3 requested review from bf8086 and ns1123 February 13, 2020 15:19

gpang approved these changes Feb 13, 2020

View reviewed changes

ns1123 approved these changes Feb 13, 2020

View reviewed changes

bf8086 approved these changes Feb 13, 2020

View reviewed changes

alluxio-bot merged commit 1807acf into Alluxio:master Feb 13, 2020

Add collectInfo command #10596

Add collectInfo command #10596

Conversation

jiacheliu3 commented Dec 9, 2019 • edited

alluxio-bot commented Dec 9, 2019

AmplabJenkins commented Dec 9, 2019

AmplabJenkins commented Dec 9, 2019

AmplabJenkins commented Dec 10, 2019

AmplabJenkins commented Dec 10, 2019

AmplabJenkins commented Dec 11, 2019

AmplabJenkins commented Dec 11, 2019

AmplabJenkins commented Dec 16, 2019

AmplabJenkins commented Dec 16, 2019

AmplabJenkins commented Jan 29, 2020

AmplabJenkins commented Jan 29, 2020

AmplabJenkins commented Jan 29, 2020

AmplabJenkins commented Jan 29, 2020

AmplabJenkins commented Feb 1, 2020

AmplabJenkins commented Feb 1, 2020

jiacheliu3 commented Feb 12, 2020

AmplabJenkins commented Feb 12, 2020

AmplabJenkins commented Feb 12, 2020

AmplabJenkins commented Feb 12, 2020

AmplabJenkins commented Feb 12, 2020

gpang commented Feb 12, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jiacheliu3 commented Feb 13, 2020

bf8086 left a comment

Choose a reason for hiding this comment

jiacheliu3 commented Feb 13, 2020

AmplabJenkins commented Feb 13, 2020

AmplabJenkins commented Feb 13, 2020

AmplabJenkins commented Feb 13, 2020

AmplabJenkins commented Feb 13, 2020

AmplabJenkins commented Feb 13, 2020

AmplabJenkins commented Feb 13, 2020

jiacheliu3 commented Feb 13, 2020

gpang left a comment

Choose a reason for hiding this comment

ns1123 commented Feb 13, 2020

bf8086 left a comment

Choose a reason for hiding this comment

calvinjia commented Feb 13, 2020

jiacheliu3 commented Dec 9, 2019 •

edited