[BEAM-79] Sync gearpump-runner branch with master #2241
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…aph structure Condense FunctionSpec and remove SdkFunctionSpec, merging data and params Inline PTransform to GraphNode, removing generic design
Also, allow direct installations from PyPI (once the package is published.) This will eventually make sdk_location an optional argument.
This also improves the class documentation.
…y after encoding/decoding
…t hierarchies in Graphite metric sink
Use Pipeline Surgery in the Dataflow Runner Add additional override factories for Dataflow overrides.
* add DEFLATE to CompressedSource * update docs in CompressedSource.java * update CompressedSourceTest * add DELATE to TextIO * update TextIOTest * add DEFLATE to FileBasedSink * update FileBasedSinkTest
This permits creates that cannot infer a coder based on the input elements to pass a Type Descriptor and use Coder Inference, rather than requiring a coder be provided.
Remove exception suppression from PAssert.SideInputCheckerDoFn Added assertion failure tests for `PAssert#thatSingleton` Added a test of default PAssert failure reason Javadoc changes [BEAM-1551] Allow `PAssert`s to take a message
… their resources' This reverts the portion of commit 3115dbd that touches BigQueryIO, which caused a double close bug.
…o gearpump-runner
manuzhang
changed the title
|
Refer to this link for build results (access rights to CI server needed): Build result: FAILURE[...truncated 1.95 MB...] at hudson.remoting.UserRequest.perform(UserRequest.java:153) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:336) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused by: org.apache.maven.plugin.MojoExecutionException: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. at org.apache.maven.plugins.enforcer.EnforceMojo.execute(EnforceMojo.java:237) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) ... 31 more2017-03-14T01:19:07.956 [ERROR] 2017-03-14T01:19:07.956 [ERROR] Re-run Maven using the -X switch to enable full debug logging.2017-03-14T01:19:07.956 [ERROR] 2017-03-14T01:19:07.956 [ERROR] For more information about the errors and possible solutions, please read the following articles:2017-03-14T01:19:07.956 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException2017-03-14T01:19:07.956 [ERROR] 2017-03-14T01:19:07.957 [ERROR] After correcting the problems, you can resume the build with the command2017-03-14T01:19:07.957 [ERROR] mvn -rf :beam-runners-gearpumpchannel stoppedSetting status of 0cdc5ad to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/8374/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install--none-- |
|
Refer to this link for build results (access rights to CI server needed): Build result: FAILURE[...truncated 1.98 MB...] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused by: org.apache.maven.plugin.MojoFailureException: There are test failures.Please refer to /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/gearpump/target/surefire-reports for the individual test results. at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution(SurefireHelper.java:91) at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:320) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:892) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:755) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) ... 31 more2017-03-14T05:39:09.533 [ERROR] 2017-03-14T05:39:09.533 [ERROR] Re-run Maven using the -X switch to enable full debug logging.2017-03-14T05:39:09.533 [ERROR] 2017-03-14T05:39:09.533 [ERROR] For more information about the errors and possible solutions, please read the following articles:2017-03-14T05:39:09.533 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException2017-03-14T05:39:09.533 [ERROR] 2017-03-14T05:39:09.533 [ERROR] After correcting the problems, you can resume the build with the command2017-03-14T05:39:09.533 [ERROR] mvn -rf :beam-runners-gearpumpchannel stoppedSetting status of e4f5d4a to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/8375/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install--none-- |
manuzhang
force-pushed
the
gearpump-runner-merge
branch
from
e4f5d4a
to
b326d84
Compare
|
Refer to this link for build results (access rights to CI server needed): Build result: FAILURE[...truncated 1.95 MB...] at hudson.remoting.UserRequest.perform(UserRequest.java:153) at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:336) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused by: org.apache.maven.plugin.MojoFailureException: You have 1 Checkstyle violation. at org.apache.maven.plugin.checkstyle.CheckstyleViolationCheckMojo.execute(CheckstyleViolationCheckMojo.java:588) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) ... 31 more2017-03-14T12:53:56.085 [ERROR] 2017-03-14T12:53:56.085 [ERROR] Re-run Maven using the -X switch to enable full debug logging.2017-03-14T12:53:56.085 [ERROR] 2017-03-14T12:53:56.085 [ERROR] For more information about the errors and possible solutions, please read the following articles:2017-03-14T12:53:56.085 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException2017-03-14T12:53:56.086 [ERROR] 2017-03-14T12:53:56.086 [ERROR] After correcting the problems, you can resume the build with the command2017-03-14T12:53:56.086 [ERROR] mvn -rf :beam-runners-gearpumpchannel stoppedSetting status of b326d84 to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/8382/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install--none-- |
|
Refer to this link for build results (access rights to CI server needed): Build result: FAILURE[...truncated 1.98 MB...] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused by: org.apache.maven.plugin.MojoFailureException: There are test failures.Please refer to /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/gearpump/target/surefire-reports for the individual test results. at org.apache.maven.plugin.surefire.SurefireHelper.reportExecution(SurefireHelper.java:91) at org.apache.maven.plugin.surefire.SurefirePlugin.handleSummary(SurefirePlugin.java:320) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:892) at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:755) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) ... 31 more2017-03-14T14:56:39.458 [ERROR] 2017-03-14T14:56:39.458 [ERROR] Re-run Maven using the -X switch to enable full debug logging.2017-03-14T14:56:39.458 [ERROR] 2017-03-14T14:56:39.458 [ERROR] For more information about the errors and possible solutions, please read the following articles:2017-03-14T14:56:39.458 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException2017-03-14T14:56:39.458 [ERROR] 2017-03-14T14:56:39.458 [ERROR] After correcting the problems, you can resume the build with the command2017-03-14T14:56:39.458 [ERROR] mvn -rf :beam-runners-gearpumpchannel stoppedSetting status of 060bc15 to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/8384/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install--none-- |
|
@kennknowles merge done and I fixed most test failures except those like This is because the Gearpump cluster has already been shutdown before returning the |
|
I don't have any suggestion - as far as I know that test should run the same as any other. The test method itself doesn't have a call to |
manuzhang
force-pushed
the
gearpump-runner-merge
branch
from
060bc15
to
b269bd2
Compare
|
It's because those tests call |
|
I think the best behavior for |
|
Changes Unknown when pulling b269bd2 on manuzhang:gearpump-runner-merge into ** on apache:gearpump-runner**. |
|
Refer to this link for build results (access rights to CI server needed): |
manuzhang
force-pushed
the
gearpump-runner-merge
branch
from
b269bd2
to
333becd
Compare
|
Refer to this link for build results (access rights to CI server needed): Build result: FAILURE[...truncated 813.21 KB...] at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593) at sun.security.ssl.InputRecord.read(InputRecord.java:532) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) at org.apache.maven.wagon.providers.http.httpclient.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136) at org.apache.maven.wagon.providers.http.httpclient.impl.io.SessionInputBufferImpl.read(SessionInputBufferImpl.java:195) at org.apache.maven.wagon.providers.http.httpclient.impl.io.ContentLengthInputStream.read(ContentLengthInputStream.java:178) at org.apache.maven.wagon.providers.http.httpclient.conn.EofSensorInputStream.read(EofSensorInputStream.java:137) at org.apache.maven.wagon.AbstractWagon.transfer(AbstractWagon.java:531) at org.apache.maven.wagon.AbstractWagon.getTransfer(AbstractWagon.java:353) ... 13 more2017-03-15T07:11:51.374 [ERROR] 2017-03-15T07:11:51.374 [ERROR] Re-run Maven using the -X switch to enable full debug logging.2017-03-15T07:11:51.374 [ERROR] 2017-03-15T07:11:51.374 [ERROR] For more information about the errors and possible solutions, please read the following articles:2017-03-15T07:11:51.374 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException2017-03-15T07:11:51.374 [ERROR] 2017-03-15T07:11:51.374 [ERROR] After correcting the problems, you can resume the build with the command2017-03-15T07:11:51.374 [ERROR] mvn -rf :beam-sdks-java-io-hbasechannel stoppedSetting status of 333becd to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/8440/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install--none-- |
manuzhang
force-pushed
the
gearpump-runner-merge
branch
from
333becd
to
3eab6a6
Compare
|
Changes Unknown when pulling 3eab6a6 on manuzhang:gearpump-runner-merge into ** on apache:gearpump-runner**. |
|
Refer to this link for build results (access rights to CI server needed): |
|
@kennknowles Done ! |
|
LGTM |
asfgit
pushed a commit
that referenced
this pull request
Mar 17, 2017
[BEAM-79] Fix gearpump-runner merge conflicts and test failure Revert BigQueryIO bit of 'Make all uses of CountingOutputStream close their resources' [BEAM-1629] Init metrics/aggregators accumulators before traversing pipeline Generate zip distribution for pyhthon Ignore results from the tox clean up phase Add README to python tarball. Remove exception suppression from PAssert.SideInputCheckerDoFn Remove duplicated dependency from Dataflow runner pom.xml Added assertion failure tests for `PAssert#thatSingleton` Added a test of default PAssert failure reason Javadoc changes [BEAM-1551] Allow `PAssert`s to take a message add unicode type to the typeDict attribute in Python SDK Remove Pipeline.getRunner [BEAM-1686] Use random MQTT clientID when not defined to avoid NPE Properly deal with late processing-time timers [BEAM-1661] Shade guava in the JdbcIO [BEAM-797] A PipelineVisitor that creates a Spark-native pipeline. Introduce Flink-specific state GC implementations Move GC timer checking to StatefulDoFnRunner.CleanupTimer Move pipeline context and add more tests. Add license to new files. Runner API translation of triggers and windowing strategies. Runner API encoding of WindowFns. Runner API context helper classes. Auto-generated runner api proto bindings. HadoopInputFormatIO with junits Test runner to stop on EOT watermark, or timeout. [BEAM-1184] Add integration tests to ElasticsearchIO Jdbc k8 script: postgres data store only accessible inside test project Jdbc k8 & data loading: add teardown and update names/docs Bump Dataflow ROS timeout to 120 minutes Fixup typo in WindowingStrategies Update archetypes [BEAM-1649] Fix unresolved references in Python SDK Upgrade Dataflow container version to beam-master-20170307 Explicitly GBK before stateful ParDo in Dataflow batch Add ServicesResourceTransformer to all shading configuration Fix typo in proto: widow -> window. [BEAM-1646] Remove duplicated bigquery dependency Update python SDK version to the next version. Change Json parsing from gson to jackson for ElasticsearchIO [maven-release-plugin] prepare for next development iteration [maven-release-plugin] prepare branch release-0.6.0 [BEAM-1633] Move .tox/ directory under target/ in Python SDK Make all uses of CountingOutputStream close their resources Flink: register known IOChannelFactories [BEAM-1546] Specify exact version for Python in the SDK [BEAM-1635] TypeError in AfterWatermark class's __repr__ method [BEAM-1636] UnboundedDataset action() does not materialize RDD [BEAM-1556] Make PipelineOptions a lazy-singleton and init IOs as part of it. Add tests for serialization of BigQueryIO.TableRowInfoCoder Fix tox warning for non-whitelisted find command Revert "Implement Single-Output ParDo as a composite" [BEAM-1623] Transform Reshuffle directly in Spark runner [BEAM-1626] Remove cache of MapWithStateDStream on read. Revert "DataflowRunner: experimental support for issuing FnAPI based jobs" Java examples: move shade plugin into default lifecycle Java 8 examples: add shade configuration for producing bundled/fat jar Java 8 examples: add SNAPSHOT repository to the pom.xml file Update Guava version from 19.0 to 20.0 in example projects [BEAM-1625] BoundedDataset action() does not materialize RDD Revert "[maven-release-plugin] prepare branch release-0.6.0" Updating Dataflow API client protobufs Implement Single-Output ParDo as a composite Add a Test for windowed CombineGloballyAsSingletonView Remove SingletonCombine Only Override CreatePCollectionView in Streaming [BEAM-1310] Add running integration tests in JdbcIO on Spark and Dataflow runners DataflowRunner: experimental support for issuing FnAPI based jobs Updates Python SDK source API so that sources can report limited parallelism signals. [BEAM-1188] Python Bigquery Verifier For E2E Test Deprecate Pipeline.getOptions BEAM-1567 hashStream should be closed in PackageUtil#createPackageAttributes() [BEAM-1565] Update Spark runner PostCommit Jenkins job. [maven-release-plugin] prepare branch release-0.6.0 Do not Reassign Windows when WindowFn is null Fix DataflowRunner message about uploaded vs cached files Update javadoc ant to include runners/ and exclude modules with a wildcard Adding per-stage matching to metrics filters Upgrade dill to 0.2.6 and pin it Remove PipelineRunner#apply [BEAM-111] Move WritableCoder to hadoop-common [BEAM-1297] Update maven shade plugin, fix typo and remove unneeded version [BEAM-351] Add DisplayData to KafkaIO Inline rather than reference FunctionSpecs. [BEAM-1517] Garbage collect user state in Flink Runner BEAM-1417 Count should comply with PTransform style guide BEAM-1419 Flatten should comply with PTransform style guide BEAM-1416 Write transform should comply with PTransform style guide BEAM-1426 SortValues should comply with PTransform style guide BEAM-1424 ToString should comply with PTransform style guide BEAM-1423 Sample should comply with PTransform style guide BEAM-1421 Latest should comply with PTransform style guide BEAM-1420 GroupByKey should comply with PTransform style guide Use UnsupportedSideInputReader in GroupAlsoByWindowEvaluatorFactory Include cython tests in presubmits for linux platform Update output stream cython declaration Make side inputs a map, rather than embedding the name in the message. Streaming tests, especially the ones using checkpoints, need a time buffer to finish. ...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Be sure to do all of the following to help us incorporate your contribution
quickly and easily:
[BEAM-<Jira issue #>] Description of pull requestmvn clean verify. (Even better, enableTravis-CI on your fork and ensure the whole test matrix passes).
<Jira issue #>in the title with the actual Jira issuenumber, if there is one.
Individual Contributor License Agreement.