[BEAM-1405] Spark runner should not close the SparkContext when the context is provided#1934
Conversation
|
R: @amitsela |
|
Refer to this link for build results (access rights to CI server needed): Build result: FAILURE[...truncated 7504 lines...] at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused by: org.apache.maven.plugin.MojoExecutionException: Archetype IT 'basic' failed: Execution failure: exit code = 1 at org.apache.maven.archetype.mojos.IntegrationTestMojo.execute(IntegrationTestMojo.java:269) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) ... 31 more2017-02-07T11:03:09.543 [ERROR] 2017-02-07T11:03:09.543 [ERROR] Re-run Maven using the -X switch to enable full debug logging.2017-02-07T11:03:09.543 [ERROR] 2017-02-07T11:03:09.543 [ERROR] For more information about the errors and possible solutions, please read the following articles:2017-02-07T11:03:09.543 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException2017-02-07T11:03:09.543 [ERROR] 2017-02-07T11:03:09.543 [ERROR] After correcting the problems, you can resume the build with the command2017-02-07T11:03:09.543 [ERROR] mvn -rf :beam-sdks-java-maven-archetypes-starterchannel stoppedSetting status of 3b29d67 to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7139/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install--none-- |
| * Provided Spark Context tests. | ||
| */ | ||
| public class ProvidedSparkContextTest { | ||
| private static final String[] WORDS_ARRAY = { |
There was a problem hiding this comment.
Those should stay up here I think (all static members declarations + getSparkContextOptions)
There was a problem hiding this comment.
Ok, I won't move the declarations.
| public void testWithProvidedContext() throws Exception { | ||
| JavaSparkContext jsc = new JavaSparkContext("local[*]", "Existing_Context"); | ||
| testWithValidProvidedContext(jsc); | ||
| if (!jsc.sc().isStopped()) { |
There was a problem hiding this comment.
Asserting here that the context is not stopped would be good, and make testReuseProvidedContext redundant I believe - you want to make sure that context remains alive after executing a pipeline with a provided context & that also guarantees it's reusable..
| * @throws Exception | ||
| */ | ||
| @Test | ||
| public void testReuseProvidedContext() throws Exception { |
There was a problem hiding this comment.
This could be avoided by asserting the context stays alive after executing a provided context pipeline (mentioned above).
|
Failure seems to be unrelated, let's wait for the next build (after relating to review comments). |
|
CC: @jbonofre |
|
@jbonofre stop! be on vacation, we're fine here 😉 |
|
Doing my best 😀 |
|
@jbonofre Amit is right, profit of the beach ! |
iemejia
force-pushed
the
BEAM-1405-spark-stop-context
branch
from
3b29d67 to
5edcdff
Compare
|
I did the changes and rebased as asked. |
|
LGTM pending tests. |
|
Refer to this link for build results (access rights to CI server needed): |
iemejia
changed the title
iemejia
changed the title
6 participants
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
[BEAM-<Jira issue #>] Description of pull requestmvn clean verify. (Even better, enableTravis-CI on your fork and ensure the whole test matrix passes).
<Jira issue #>in the title with the actual Jira issuenumber, if there is one.
Individual Contributor License Agreement.