[BEAM-646] Use Surgery in the Dataflow Runner #2039
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Run Dataflow RunnableOnService |
|
Refer to this link for build results (access rights to CI server needed): Build result: FAILURE[...truncated 4596 lines...] at hudson.remoting.UserRequest.perform(UserRequest.java:153) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:332) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused by: org.apache.maven.plugin.MojoFailureException: You have 2 Checkstyle violations. at org.apache.maven.plugin.checkstyle.CheckstyleViolationCheckMojo.execute(CheckstyleViolationCheckMojo.java:588) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) ... 31 more2017-02-17T23:12:21.425 [ERROR] 2017-02-17T23:12:21.425 [ERROR] Re-run Maven using the -X switch to enable full debug logging.2017-02-17T23:12:21.425 [ERROR] 2017-02-17T23:12:21.425 [ERROR] For more information about the errors and possible solutions, please read the following articles:2017-02-17T23:12:21.425 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException2017-02-17T23:12:21.426 [ERROR] 2017-02-17T23:12:21.426 [ERROR] After correcting the problems, you can resume the build with the command2017-02-17T23:12:21.426 [ERROR] mvn -rf :beam-runners-direct-javachannel stoppedSetting status of 46d6b1e to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7564/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install--none-- |
tgroh
force-pushed
the
dataflow_runner_surgery
branch
from
46d6b1e
to
580243e
Compare
|
Refer to this link for build results (access rights to CI server needed): Failed Tests: 3beam_PreCommit_Java_MavenInstall/org.apache.beam:beam-examples-java: 3
--none-- |
tgroh
force-pushed
the
dataflow_runner_surgery
branch
2 times, most recently
from
96beeb7
to
b9742ca
Compare
|
Refer to this link for build results (access rights to CI server needed): Failed Tests: 3beam_PreCommit_Java_MavenInstall/org.apache.beam:beam-examples-java: 3
--none-- |
|
Refer to this link for build results (access rights to CI server needed): Failed Tests: 3beam_PreCommit_Java_MavenInstall/org.apache.beam:beam-examples-java: 3
--none-- |
|
Refer to this link for build results (access rights to CI server needed): Failed Tests: 3beam_PreCommit_Java_MavenInstall/org.apache.beam:beam-examples-java: 3
--none-- |
tgroh
force-pushed
the
dataflow_runner_surgery
branch
from
1229c97
to
5c56e8a
Compare
4 tasks
|
Refer to this link for build results (access rights to CI server needed): Build result: FAILURE[...truncated 620.03 KB...] at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:336) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused by: org.apache.maven.plugin.MojoFailureException: You have 1 Checkstyle violation. at org.apache.maven.plugin.checkstyle.CheckstyleViolationCheckMojo.execute(CheckstyleViolationCheckMojo.java:588) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) ... 31 more2017-02-21T18:20:18.663 [ERROR] 2017-02-21T18:20:18.663 [ERROR] Re-run Maven using the -X switch to enable full debug logging.2017-02-21T18:20:18.663 [ERROR] 2017-02-21T18:20:18.663 [ERROR] For more information about the errors and possible solutions, please read the following articles:2017-02-21T18:20:18.663 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException2017-02-21T18:20:18.663 [ERROR] 2017-02-21T18:20:18.663 [ERROR] After correcting the problems, you can resume the build with the command2017-02-21T18:20:18.663 [ERROR] mvn -rf :beam-runners-google-cloud-dataflow-javachannel stoppedSetting status of 5c56e8a to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7634/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install--none-- |
tgroh
force-pushed
the
dataflow_runner_surgery
branch
from
5c56e8a
to
09e2278
Compare
|
Refer to this link for build results (access rights to CI server needed): Failed Tests: 4beam_PreCommit_Java_MavenInstall/org.apache.beam:beam-examples-java: 3
beam_PreCommit_Java_MavenInstall/org.apache.beam:beam-runners-spark: 1--none-- |
tgroh
force-pushed
the
dataflow_runner_surgery
branch
from
09e2278
to
777ea60
Compare
|
Refer to this link for build results (access rights to CI server needed): Failed Tests: 2beam_PreCommit_Java_MavenInstall/org.apache.beam:beam-examples-java: 1beam_PreCommit_Java_MavenInstall/org.apache.beam:beam-runners-spark: 1--none-- |
tgroh
force-pushed
the
dataflow_runner_surgery
branch
from
777ea60
to
1af755f
Compare
|
Run Dataflow RunnableOnService |
|
Refer to this link for build results (access rights to CI server needed): |
|
Refer to this link for build results (access rights to CI server needed): Build result: FAILURE[...truncated 1.11 MB...] at org.apache.rat.analysis.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:46) at org.apache.rat.analysis.DefaultAnalyserFactory$DefaultAnalyser.analyse(DefaultAnalyserFactory.java:60) at org.apache.rat.document.impl.util.DocumentAnalyserMultiplexer.analyse(DocumentAnalyserMultiplexer.java:36) at org.apache.rat.report.claim.util.ClaimReporterMultiplexer.report(ClaimReporterMultiplexer.java:42) ... 39 moreCaused by: java.io.FileNotFoundException: src/test/java/org/apache/beam/sdk/TestUtils.java (No such file or directory) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.(FileInputStream.java:138) at org.apache.rat.mp.FilesReportable$FileDocument.reader(FilesReportable.java:88) at org.apache.rat.analysis.DocumentHeaderAnalyser.analyse(DocumentHeaderAnalyser.java:41) ... 42 more2017-02-22T01:01:42.800 [ERROR] 2017-02-22T01:01:42.801 [ERROR] Re-run Maven using the -X switch to enable full debug logging.2017-02-22T01:01:42.801 [ERROR] 2017-02-22T01:01:42.801 [ERROR] For more information about the errors and possible solutions, please read the following articles:2017-02-22T01:01:42.801 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException2017-02-22T01:01:42.801 [ERROR] 2017-02-22T01:01:42.801 [ERROR] After correcting the problems, you can resume the build with the command2017-02-22T01:01:42.801 [ERROR] mvn -rf :beam-sdks-java-corechannel stoppedSending e-mails to: commits@beam.apache.org jbonofre@apache.org tgroh@google.com klk@google.com altay@google.comSetting status of 1af755f to FAILURE with url https://builds.apache.org/job/beam_PostCommit_Java_RunnableOnService_Dataflow/2362/ and message: 'Build finished. 'Using context: Jenkins: Google Cloud Dataflow Runner RunnableOnService Tests--none-- |
|
Run Dataflow RunnableOnService |
|
Refer to this link for build results (access rights to CI server needed): |
|
Code looks good, but I want to see what happens between now and green ROS. There's a possibility you'll need a new worker image. |
|
Run Dataflow RunnableOnService |
|
Refer to this link for build results (access rights to CI server needed): Build result: FAILURE[...truncated 543.18 KB...] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused by: org.apache.maven.plugin.compiler.CompilationFailureException: Compilation failure/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/core-java/src/main/java/org/apache/beam/runners/core/SideInputHandler.java:[119,58] cannot find symbol symbol: method getCoderInternal() location: variable sideInput of type org.apache.beam.sdk.values.PCollectionView<?> at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:1029) at org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:137) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) ... 31 more2017-02-23T02:47:43.842 [ERROR] 2017-02-23T02:47:43.842 [ERROR] Re-run Maven using the -X switch to enable full debug logging.2017-02-23T02:47:43.842 [ERROR] 2017-02-23T02:47:43.842 [ERROR] For more information about the errors and possible solutions, please read the following articles:2017-02-23T02:47:43.842 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException2017-02-23T02:47:43.842 [ERROR] 2017-02-23T02:47:43.842 [ERROR] After correcting the problems, you can resume the build with the command2017-02-23T02:47:43.842 [ERROR] mvn -rf :beam-runners-core-javachannel stoppedSetting status of ec18c91 to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7733/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install--none-- |
tgroh
force-pushed
the
dataflow_runner_surgery
branch
from
ec18c91
to
15e3c17
Compare
|
Run Dataflow RunnableOnService |
|
Refer to this link for build results (access rights to CI server needed): |
|
Refer to this link for build results (access rights to CI server needed): |
|
@kennknowles PTAL (Dataflow ROS is green, woop woop) |
kennknowles
reviewed
Feb 24, 2017
| @@ -440,6 +441,7 @@ public void testDiskSizeGbConfig() throws IOException { | |||
| } | |||
|
|
|||
| @Test | |||
| @Ignore("TODO: Reenable when the hierarchy is copied") | |||
There was a problem hiding this comment.
For each TODO: I think some are fixed, some should be just deleted, and the rest should have JIRAs if possible.
There was a problem hiding this comment.
All are fixed, with exposure of performSurgery
|
LGTM pending changes just to |
tgroh
force-pushed
the
dataflow_runner_surgery
branch
from
2165852
to
d200b38
Compare
|
Refer to this link for build results (access rights to CI server needed): Build result: ABORTED[...truncated 792.59 KB...] ... 31 moreCaused by: java.lang.RuntimeException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?Command was /bin/sh -c cd /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/io/hbase && /usr/local/asfpackages/java/jdk1.8.0_121/jre/bin/java org.apache.maven.surefire.booter.ForkedBooter /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/io/hbase/target/surefire/surefire5316712535186464581tmp /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/io/hbase/target/surefire/surefire_81718212360316678872tmp at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:590) at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:460) at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:229) at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:201) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1026) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:862) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:755) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) ... 32 more2017-02-24T21:13:58.618 [ERROR] 2017-02-24T21:13:58.618 [ERROR] Re-run Maven using the -X switch to enable full debug logging.2017-02-24T21:13:58.618 [ERROR] 2017-02-24T21:13:58.619 [ERROR] For more information about the errors and possible solutions, please read the following articles:2017-02-24T21:13:58.619 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException2017-02-24T21:13:58.619 [ERROR] 2017-02-24T21:13:58.619 [ERROR] After correcting the problems, you can resume the build with the command2017-02-24T21:13:58.619 [ERROR] mvn -rf :beam-sdks-java-io-hbaseBuild was abortedchannel stoppedSetting status of 2165852 to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7824/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install--none-- |
|
Refer to this link for build results (access rights to CI server needed): Build result: ABORTED[...truncated 793.34 KB...] ... 31 moreCaused by: java.lang.RuntimeException: The forked VM terminated without properly saying goodbye. VM crash or System.exit called?Command was /bin/sh -c cd /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/sdks/java/io/hbase && /usr/local/asfpackages/java/jdk1.8.0_121/jre/bin/java org.apache.maven.surefire.booter.ForkedBooter /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/sdks/java/io/hbase/target/surefire/surefire8763274207289909236tmp /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/sdks/java/io/hbase/target/surefire/surefire_8682168825656325026tmp at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:590) at org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:460) at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:229) at org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:201) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1026) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:862) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:755) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) ... 32 more2017-02-24T21:45:16.870 [ERROR] 2017-02-24T21:45:16.870 [ERROR] Re-run Maven using the -X switch to enable full debug logging.2017-02-24T21:45:16.870 [ERROR] 2017-02-24T21:45:16.870 [ERROR] For more information about the errors and possible solutions, please read the following articles:2017-02-24T21:45:16.870 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException2017-02-24T21:45:16.871 [ERROR] 2017-02-24T21:45:16.871 [ERROR] After correcting the problems, you can resume the build with the command2017-02-24T21:45:16.871 [ERROR] mvn -rf :beam-sdks-java-io-hbaseBuild was abortedchannel stoppedSetting status of d200b38 to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7826/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install--none-- |
tgroh
force-pushed
the
dataflow_runner_surgery
branch
from
d200b38
to
92977f6
Compare
Use Pipeline Surgery in the Dataflow Runner Add additional override factories for Dataflow overrides.
tgroh
force-pushed
the
dataflow_runner_surgery
branch
from
92977f6
to
3b0c64d
Compare
|
Changes Unknown when pulling 3b0c64d on tgroh:dataflow_runner_surgery into ** on apache:master**. |
|
Refer to this link for build results (access rights to CI server needed): Failed Tests: 1beam_PreCommit_Java_MavenInstall/org.apache.beam:beam-runners-google-cloud-dataflow-java: 1--none-- |
|
Changes Unknown when pulling 62e02c0 on tgroh:dataflow_runner_surgery into ** on apache:master**. |
|
Run Dataflow RunnableOnService |
|
Refer to this link for build results (access rights to CI server needed): |
|
Refer to this link for build results (access rights to CI server needed): |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
[BEAM-<Jira issue #>] Description of pull requestmvn clean verify. (Even better, enableTravis-CI on your fork and ensure the whole test matrix passes).
<Jira issue #>in the title with the actual Jira issuenumber, if there is one.
Individual Contributor License Agreement.
Couple of cleanups and underlying PRs still have to be submitted prior to actual review.
I do want to use this PR to run some RunnableOnService Tests, though.