New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
(CDAP-17287) Collects Spark execution event logs #12782
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Minor comments
throw new UnsupportedOperationException("Spark event logs collection is not enabled"); | ||
} | ||
|
||
if (!"application/octet-stream".equals(request.headers().get(HttpHeaderNames.CONTENT_TYPE))) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we can use javax.ws.rs.core.MediaType.APPLICATION_OCTET_STREAM?
cdap-spark-core-base/src/main/java/io/cdap/cdap/app/runtime/spark/SparkRuntimeUtils.java
Outdated
Show resolved
Hide resolved
cdap-spark-core-base/src/main/java/io/cdap/cdap/app/runtime/spark/SparkRuntimeUtils.java
Show resolved
Hide resolved
HttpURLConnection urlConn = remoteClient.openConnection(HttpMethod.POST, path); | ||
try { | ||
urlConn.setChunkedStreamingMode(CHUNK_SIZE); | ||
urlConn.setRequestProperty(HttpHeaders.CONTENT_TYPE, "application/octet-stream"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we can use javax.ws.rs.core.MediaType.APPLICATION_OCTET_STREAM?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
cdap-spark-core-base/src/main/java/io/cdap/cdap/app/runtime/spark/SparkRuntimeUtils.java
Outdated
Show resolved
Hide resolved
cdap-spark-core-base/src/main/java/io/cdap/cdap/app/runtime/spark/SparkRuntimeUtils.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM after minor fix
cdap-spark-core-base/src/main/java/io/cdap/cdap/app/runtime/spark/SparkRuntimeUtils.java
Outdated
Show resolved
Hide resolved
0277c88
to
83144b4
Compare
- Introducing the Spark event logging and upload of logs through the runtime service. Based on the cConf settings, the event logging collection will be on/off regardless of the Spark job spark.eventLog.* configurations. - Refactoring of the SparkRuntimeEnv class to simplify the logic by using CompletableFuture. - Make runtime system services shutdown after the SparkExecutionContext stopped. Currently they stop concurrently. It is not a problem before this PR since we are not doing any important work at the context stop. However, this PR introduces uploading the event logs at the context stop, which needs to be completed before the system services shutdown and process termination.
83144b4
to
63aa9a7
Compare
Build passed https://builds.cask.co/browse/CDAP-DUT7352-7 |
This PR has three commits:
SparkExecutionContext
stopped. Currently they stop concurrently. It is not a problem before this PR since we are not doing any important work at the context stop. However, this PR introduces uploading the event logs at the context stop, which needs to be completed before the system services shutdown and process termination.SparkRuntimeEnv
class to simplify the logic.spark.eventLog.*
configurations.