New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BEAM-1182] Clone Checkpoints before resuming in the DirectRunner #2323
Conversation
The coder can do no work, but should always be specified.
This exercises the CheckpointMarkCoder of all Unbounded Sources in the DirectRunner.
R: @dhalperi |
0de7717
to
6489585
Compare
Refer to this link for build results (access rights to CI server needed): Build result: FAILURE[...truncated 608.56 KB...] at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused by: org.apache.maven.plugin.compiler.CompilationFailureException: Compilation failure/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/runners/direct-java/src/main/java/org/apache/beam/runners/direct/UnboundedReadEvaluatorFactory.java:[51,24] package sun.reflect.misc does not exist at org.apache.maven.plugin.compiler.AbstractCompilerMojo.execute(AbstractCompilerMojo.java:1029) at org.apache.maven.plugin.compiler.CompilerMojo.execute(CompilerMojo.java:137) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) ... 31 more2017-03-24T21:45:25.929 [ERROR] 2017-03-24T21:45:25.929 [ERROR] Re-run Maven using the -X switch to enable full debug logging.2017-03-24T21:45:25.929 [ERROR] 2017-03-24T21:45:25.929 [ERROR] For more information about the errors and possible solutions, please read the following articles:2017-03-24T21:45:25.929 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException2017-03-24T21:45:25.929 [ERROR] 2017-03-24T21:45:25.929 [ERROR] After correcting the problems, you can resume the build with the command2017-03-24T21:45:25.929 [ERROR] mvn -rf :beam-runners-direct-javachannel stoppedSetting status of 0de7717 to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/8777/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install--none-- |
@@ -458,6 +458,9 @@ private TestUnboundedSource(Coder<T> coder, boolean throwOnClose, List<T> elems) | |||
@Override | |||
public UnboundedSource.UnboundedReader<T> createReader( | |||
PipelineOptions options, @Nullable TestCheckpointMark checkpointMark) { | |||
checkState( | |||
checkpointMark == null || checkpointMark.decoded, | |||
"Cannot resume from a checkpoint that has not been decoded"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should probably separate the if
into two separate useful errors.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a checkState
- there's either no checkpoint or there's a decoded checkpoint; if we get a non-null non-decoded checkpoint, then we're resuming from a state we don't expect and the test should fail.
@@ -573,6 +577,10 @@ private TestCheckpointMark(int index) { | |||
public void finalizeCheckpoint() throws IOException { | |||
checkState( | |||
!finalized, "%s was finalized more than once", TestCheckpointMark.class.getSimpleName()); | |||
checkState( | |||
!decoded, | |||
"%s was finalized after being deccoded", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
decoded
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done.
LGTM, please self-merge at will after addressing minor issues |
Refer to this link for build results (access rights to CI server needed): |
Refer to this link for build results (access rights to CI server needed): |
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
[BEAM-<Jira issue #>] Description of pull request
mvn clean verify
. (Even better, enableTravis-CI on your fork and ensure the whole test matrix passes).
<Jira issue #>
in the title with the actual Jira issuenumber, if there is one.
Individual Contributor License Agreement.