[SPARK-8405][Doc] Add how to view logs on Web UI when yarn log aggregation is enabled #7463
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Test build #37617 has finished for PR 7463 at commit
|
| @@ -68,9 +68,9 @@ In YARN terminology, executors and application masters run inside "containers". | |||
|
|
|||
| yarn logs -applicationId <app ID> | |||
|
|
|||
| will print out the contents of all log files from all containers from the given application. You can also view the container log files directly in HDFS using the HDFS shell or API. The directory where they are located can be found by looking at your YARN configs (`yarn.nodemanager.remote-app-log-dir` and `yarn.nodemanager.remote-app-log-dir-suffix`). | |||
| will print out the contents of all log files from all containers from the given application. You can also view the container log files directly in HDFS using the HDFS shell or API. The directory where they are located can be found by looking at your YARN configs (`yarn.nodemanager.remote-app-log-dir` and `yarn.nodemanager.remote-app-log-dir-suffix`). The logs are also available on Web UI. You need have both the Spark history server and the MR history server running and configure `yarn.log.server.url` in yarn-site.xml properly. The log url on the Spark history server UI will redirect you to the MR history server to show the aggregated logs. | |||
There was a problem hiding this comment.
Minor text updates: MR history server -> MapReduce History Server. I'd hyperlink one of them too. url -> URL. back-tick yarn-site.xml It would be nice to link through to YARN docs that would describe these parameters too, if possible.
|
Thanks @srowen . I updated the text. But I didn't find proper docs for MapReduce job history server and the |
|
Test build #37789 has finished for PR 7463 at commit
|
| @@ -68,9 +68,9 @@ In YARN terminology, executors and application masters run inside "containers". | |||
|
|
|||
| yarn logs -applicationId <app ID> | |||
|
|
|||
| will print out the contents of all log files from all containers from the given application. You can also view the container log files directly in HDFS using the HDFS shell or API. The directory where they are located can be found by looking at your YARN configs (`yarn.nodemanager.remote-app-log-dir` and `yarn.nodemanager.remote-app-log-dir-suffix`). | |||
| will print out the contents of all log files from all containers from the given application. You can also view the container log files directly in HDFS using the HDFS shell or API. The directory where they are located can be found by looking at your YARN configs (`yarn.nodemanager.remote-app-log-dir` and `yarn.nodemanager.remote-app-log-dir-suffix`). The logs are also available on Web UI. You need have both the Spark history server and the MapReduce history server running and configure `yarn.log.server.url` in `yarn-site.xml` properly. The log URL on the Spark history server UI will redirect you to the MapReduce history server to show the aggregated logs. | |||
There was a problem hiding this comment.
"The logs are also available on Web UI" is pretty generic. Can we make this more specific like on the Spark Web UI under the Executors Tab
|
Sounds great, @tgravescs . The text was updated as you suggested. |
|
Test build #38044 has finished for PR 7463 at commit
|
| @@ -68,9 +68,9 @@ In YARN terminology, executors and application masters run inside "containers". | |||
|
|
|||
| yarn logs -applicationId <app ID> | |||
|
|
|||
| will print out the contents of all log files from all containers from the given application. You can also view the container log files directly in HDFS using the HDFS shell or API. The directory where they are located can be found by looking at your YARN configs (`yarn.nodemanager.remote-app-log-dir` and `yarn.nodemanager.remote-app-log-dir-suffix`). | |||
| will print out the contents of all log files from all containers from the given application. You can also view the container log files directly in HDFS using the HDFS shell or API. The directory where they are located can be found by looking at your YARN configs (`yarn.nodemanager.remote-app-log-dir` and `yarn.nodemanager.remote-app-log-dir-suffix`). The logs are also available on the Spark Web UI under the Executors Tab. You need have both the Spark history server and the MapReduce history server running and configure `yarn.log.server.url` in `yarn-site.xml` properly. The log URL on the Spark history server UI will redirect you to the MapReduce history server to show the aggregated logs. | |||
There was a problem hiding this comment.
can you change "You need have" to "You need to" have"
|
one minor nit, otherwise looks good. @srowen do you have any other comments, I want to not sure your previous comment was addressed? |
|
OK by me |
|
Sorry for the delay. Updated! |
|
Test build #38427 has finished for PR 7463 at commit
|
|
+1. Thanks! |
asfgit
pushed a commit
that referenced
this pull request
Jul 27, 2015
…gation is enabled Some users may not be aware that the logs are available on Web UI even if Yarn log aggregation is enabled. Update the doc to make this clear and what need to be configured. Author: Carson Wang <carson.wang@intel.com> Closes #7463 from carsonwang/YarnLogDoc and squashes the following commits: 274c054 [Carson Wang] Minor text fix 74df3a1 [Carson Wang] address comments 5a95046 [Carson Wang] Update the text in the doc e5775c1 [Carson Wang] Update doc about how to view the logs on Web UI when yarn log aggregation is enabled (cherry picked from commit 6228381) Signed-off-by: Tom Graves <tgraves@yahoo-inc.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some users may not be aware that the logs are available on Web UI even if Yarn log aggregation is enabled. Update the doc to make this clear and what need to be configured.