[LIVY-472][Server] Improve the logs for fail-to-create session #96
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@zjffdu Would you please help to review? Thanks. |
zjffdu
reviewed
May 23, 2018
| s" $appTagLowerCase in ${livyConf.getTimeAsMs(LivyConf.YARN_APP_LOOKUP_TIMEOUT)/1000}" + | ||
| " seconds. This may be because 1) spark-submit fail to submit application to YARN; " + | ||
| "or 2) YARN cluster doesn't have enough resources to start the application in time. " + | ||
| "Please check Livy log and YARN log to know the details.") |
There was a problem hiding this comment.
Could we include the yarn appId or yarn app webl url in this log ?
There was a problem hiding this comment.
No, we can't. Usually when we met such issue, yarn doesn't actually launch the application, so there's no web URL, or app id is invalid.
There was a problem hiding this comment.
Then could we include the livy session web link so that user that check this livy session to see why session is failed ?
There was a problem hiding this comment.
I can try to add (but I don't want to change a lot for just a link), also Livy session web link is no different from the returning json data.
There was a problem hiding this comment.
Double check the code, seems this is not easy to get a session UI link here, this will involve unnecessary parameter passing between different classes.
Codecov Report
@@ Coverage Diff @@
## master #96 +/- ##
============================================
+ Coverage 71.33% 71.46% +0.13%
Complexity 796 796
============================================
Files 97 97
Lines 5424 5428 +4
Branches 807 809 +2
============================================
+ Hits 3869 3879 +10
+ Misses 1028 1023 -5
+ Partials 527 526 -1
Continue to review full report at Codecov.
|
|
Any more comments? @zjffdu . |
|
LGTM |
|
Thanks for the review @zjffdu ! Merging to master and branch 0.5. |
asfgit
pushed a commit
that referenced
this pull request
May 25, 2018
## What changes were proposed in this pull request? Livy currently doesn't give a very clear log about the fail-to-create session, it only says that session related app tag cannot be found in RM, but doesn't tell user how to search and get the true root cause. So here change the logs to make it more clear. ## How was this patch tested? Local verification. Author: jerryshao <sshao@hortonworks.com> Closes #96 from jerryshao/LIVY-472. (cherry picked from commit ca4cad2) Signed-off-by: jerryshao <sshao@hortonworks.com>
tkakantousis
pushed a commit
to tkakantousis/incubator-livy
that referenced
this pull request
Oct 23, 2018
## What changes were proposed in this pull request? Livy currently doesn't give a very clear log about the fail-to-create session, it only says that session related app tag cannot be found in RM, but doesn't tell user how to search and get the true root cause. So here change the logs to make it more clear. ## How was this patch tested? Local verification. Author: jerryshao <sshao@hortonworks.com> Closes apache#96 from jerryshao/LIVY-472. (cherry picked from commit ca4cad2) Signed-off-by: jerryshao <sshao@hortonworks.com>
tkakantousis
added a commit
to logicalclocks/incubator-livy
that referenced
this pull request
Oct 24, 2018
* [MINOR] Addressed Docs dependency security vulnerability followup Followup to 26428c5 Author: Alex Bozarth <ajbozart@us.ibm.com> Closes apache#82 from ajbozarth/gem. (cherry picked from commit 6d2ffdd) Signed-off-by: Alex Bozarth <ajbozart@us.ibm.com> * [MINOR] Fix travis builds Each individual commit has a more detailed description of what's being changed and why. At the moment, Travis builds don't work: * https://travis-ci.org/mineo/incubator-livy/builds/359324523 - the `sudo pip3 install --upgrade pip "setuptools < 36"` command fails with `sudo: pip3: command not found`. * fixing that, the `failing the org.apache.rat:apache-rat-plugin:0.12:check` maven goal fails because it sees `.pytest_cache` folders that it doesn't know about (pytest-dev/pytest#3286, failure in https://travis-ci.org/mineo/incubator-livy/jobs/359326261, check the raw log). This pull request works around the pip3 failures by just using pip as a callable module (possible since Python 3.4) and adding `.pytest_cache` to raw-excludes, as well as showing the contents of `rat.txt` files in the `after_failure` step. I concede that the pip change is more of a workaround, but I don't know how else to fix it and unblock testing. Running the tests on travis. Please review https://livy.incubator.apache.org/community/ before opening a pull request. Author: Wieland Hoffmann <WHoffman@de.ibm.com> Closes apache#84 from mineo/rat. (cherry picked from commit 06cfa7b) Signed-off-by: jerryshao <sshao@hortonworks.com> * [LIVY-455][REPL] Fix json4s doesn't support java.math.BigDecimal issue ## What changes were proposed in this pull request? Livy's SQLInterpreter will throw exception when rows contain java.math.BigDecimal data. This is because current version of json4s doesn't treat java.math.BigDecimal type as primitive type. On the contrary, json4s supports Scala BigDecimal as primitive type. So the fix is to convert java BigDecimal to Scala BigDecimal. ## How was this patch tested? Unit test is added. Author: jerryshao <sshao@hortonworks.com> Closes apache#85 from jerryshao/LIVY-455. (cherry picked from commit 7e4bb3b) Signed-off-by: jerryshao <sshao@hortonworks.com> * [LIVY-457][REPL] Fix SQLContext is not initialized correctly issue ## What changes were proposed in this pull request? The signature of SQLContext's constructor is changed in Spark2, but we're still using the Spark1's signature, which will throw an exception when using this object. ## How was this patch tested? UT and local verification. Author: jerryshao <sshao@hortonworks.com> Closes apache#86 from jerryshao/LIVY-457. (cherry picked from commit cd8b112) Signed-off-by: jerryshao <sshao@hortonworks.com> * [LIVY-466][RSC] Fix RSCDriver exception during RPC shutdown ## What changes were proposed in this pull request? During RSCDriver's shutdown, it will first shutdown RPC server, and then all the RPC clients. When RPC client is closed, it will register a timeout to avoid orphaned RSCDriver, but this is not necessary during RSCDriver's shutdown, so here fixing this issue. The details can be seen in [JIRA](https://issues.apache.org/jira/browse/LIVY-466). ## How was this patch tested? Local verification. Author: jerryshao <sshao@hortonworks.com> Closes apache#90 from jerryshao/LIVY-466. (cherry picked from commit e3f45a0) Signed-off-by: jerryshao <sshao@hortonworks.com> * [LIVY-472][SERVER] Improve the logs for fail-to-create session ## What changes were proposed in this pull request? Livy currently doesn't give a very clear log about the fail-to-create session, it only says that session related app tag cannot be found in RM, but doesn't tell user how to search and get the true root cause. So here change the logs to make it more clear. ## How was this patch tested? Local verification. Author: jerryshao <sshao@hortonworks.com> Closes apache#96 from jerryshao/LIVY-472. (cherry picked from commit ca4cad2) Signed-off-by: jerryshao <sshao@hortonworks.com> * [Security] Update to support pyspark and sparkr changes in Spark 2.3.1 * [LIVY-498][REPL] Fix Windows CRLF line ending issue in SparkR interpreter ## What changes were proposed in this pull request? If the issued query contains CRLF EOL, it will be failed to execute on *nix machine. This happens when submitting queries from Windows machine and executing on Linux machine. So here propose to convert statement to match system's EOL. ## How was this patch tested? New UT added. Author: jerryshao <sshao@hortonworks.com> Closes apache#105 from jerryshao/LIVY-498. (cherry picked from commit 8027ca7) Signed-off-by: jerryshao <sshao@hortonworks.com> * upgrade to work with spark 2.3.2
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Livy currently doesn't give a very clear log about the fail-to-create session, it only says that session related app tag cannot be found in RM, but doesn't tell user how to search and get the true root cause. So here change the logs to make it more clear.
How was this patch tested?
Local verification.