BEAM-1151 Add failure handling to BigQueryIO.Write #1609

reuvenlax · 2016-12-14T01:12:03Z

Allow a pipeline to handle streaming insert failures when using BigQueryIO.Write in streaming mode. TableRow objects that fail to insert are returned to the user as a PCollection.

R: @jkff

asfbot · 2016-12-14T02:20:32Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/5873/

Build result: FAILURE

[...truncated 6653 lines...][JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/io/hdfs/pom.xml to org.apache.beam/beam-sdks-java-io-hdfs/0.5.0-incubating-SNAPSHOT/beam-sdks-java-io-hdfs-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/maven-archetypes/pom.xml to org.apache.beam/beam-sdks-java-maven-archetypes-parent/0.5.0-incubating-SNAPSHOT/beam-sdks-java-maven-archetypes-parent-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/google-cloud-dataflow-java/dependency-reduced-pom.xml to org.apache.beam/beam-runners-google-cloud-dataflow-java/0.5.0-incubating-SNAPSHOT/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/google-cloud-dataflow-java/target/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT.jar to org.apache.beam/beam-runners-google-cloud-dataflow-java/0.5.0-incubating-SNAPSHOT/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT.jar[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/google-cloud-dataflow-java/target/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT-javadoc.jar to org.apache.beam/beam-runners-google-cloud-dataflow-java/0.5.0-incubating-SNAPSHOT/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT-javadoc.jar[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/google-cloud-dataflow-java/target/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT-sources.jar to org.apache.beam/beam-runners-google-cloud-dataflow-java/0.5.0-incubating-SNAPSHOT/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT-sources.jar[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/google-cloud-dataflow-java/target/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT-test-sources.jar to org.apache.beam/beam-runners-google-cloud-dataflow-java/0.5.0-incubating-SNAPSHOT/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT-test-sources.jar[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/google-cloud-dataflow-java/target/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT-tests.jar to org.apache.beam/beam-runners-google-cloud-dataflow-java/0.5.0-incubating-SNAPSHOT/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT-tests.jar[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/apex/pom.xml to org.apache.beam/beam-runners-apex/0.5.0-incubating-SNAPSHOT/beam-runners-apex-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/io/jms/pom.xml to org.apache.beam/beam-sdks-java-io-jms/0.5.0-incubating-SNAPSHOT/beam-sdks-java-io-jms-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/flink/pom.xml to org.apache.beam/beam-runners-flink-parent/0.5.0-incubating-SNAPSHOT/beam-runners-flink-parent-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/io/mongodb/pom.xml to org.apache.beam/beam-sdks-java-io-mongodb/0.5.0-incubating-SNAPSHOT/beam-sdks-java-io-mongodb-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/extensions/sorter/pom.xml to org.apache.beam/beam-sdks-java-extensions-sorter/0.5.0-incubating-SNAPSHOT/beam-sdks-java-extensions-sorter-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/pom.xml to org.apache.beam/beam-sdks-parent/0.5.0-incubating-SNAPSHOT/beam-sdks-parent-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: SDKs #5875 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/extensions/join-library/pom.xml to org.apache.beam/beam-sdks-java-extensions-join-library/0.5.0-incubating-SNAPSHOT/beam-sdks-java-extensions-join-library-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/pom.xml to org.apache.beam/beam-runners-parent/0.5.0-incubating-SNAPSHOT/beam-runners-parent-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: Runners #5875 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/io/jdbc/pom.xml to org.apache.beam/beam-sdks-java-io-jdbc/0.5.0-incubating-SNAPSHOT/beam-sdks-java-io-jdbc-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/extensions/pom.xml to org.apache.beam/beam-sdks-java-extensions-parent/0.5.0-incubating-SNAPSHOT/beam-sdks-java-extensions-parent-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/examples/pom.xml to org.apache.beam/beam-examples-parent/0.5.0-incubating-SNAPSHOT/beam-examples-parent-0.5.0-incubating-SNAPSHOT.pomchannel stoppedSetting status of cc088fd to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/5873/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install
--none--

asfbot · 2016-12-14T03:17:00Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/5879/

Build result: FAILURE

[...truncated 6575 lines...][JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/maven-archetypes/examples/pom.xml to org.apache.beam/beam-sdks-java-maven-archetypes-examples/0.5.0-incubating-SNAPSHOT/beam-sdks-java-maven-archetypes-examples-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/spark/pom.xml to org.apache.beam/beam-runners-spark/0.5.0-incubating-SNAPSHOT/beam-runners-spark-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/io/hdfs/pom.xml to org.apache.beam/beam-sdks-java-io-hdfs/0.5.0-incubating-SNAPSHOT/beam-sdks-java-io-hdfs-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/maven-archetypes/pom.xml to org.apache.beam/beam-sdks-java-maven-archetypes-parent/0.5.0-incubating-SNAPSHOT/beam-sdks-java-maven-archetypes-parent-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/google-cloud-dataflow-java/dependency-reduced-pom.xml to org.apache.beam/beam-runners-google-cloud-dataflow-java/0.5.0-incubating-SNAPSHOT/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/google-cloud-dataflow-java/target/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT.jar to org.apache.beam/beam-runners-google-cloud-dataflow-java/0.5.0-incubating-SNAPSHOT/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT.jar[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/google-cloud-dataflow-java/target/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT-javadoc.jar to org.apache.beam/beam-runners-google-cloud-dataflow-java/0.5.0-incubating-SNAPSHOT/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT-javadoc.jar[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/google-cloud-dataflow-java/target/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT-sources.jar to org.apache.beam/beam-runners-google-cloud-dataflow-java/0.5.0-incubating-SNAPSHOT/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT-sources.jar[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/google-cloud-dataflow-java/target/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT-test-sources.jar to org.apache.beam/beam-runners-google-cloud-dataflow-java/0.5.0-incubating-SNAPSHOT/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT-test-sources.jar[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/google-cloud-dataflow-java/target/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT-tests.jar to org.apache.beam/beam-runners-google-cloud-dataflow-java/0.5.0-incubating-SNAPSHOT/beam-runners-google-cloud-dataflow-java-0.5.0-incubating-SNAPSHOT-tests.jar[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/apex/pom.xml to org.apache.beam/beam-runners-apex/0.5.0-incubating-SNAPSHOT/beam-runners-apex-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/io/jms/pom.xml to org.apache.beam/beam-sdks-java-io-jms/0.5.0-incubating-SNAPSHOT/beam-sdks-java-io-jms-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/flink/pom.xml to org.apache.beam/beam-runners-flink-parent/0.5.0-incubating-SNAPSHOT/beam-runners-flink-parent-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/io/mongodb/pom.xml to org.apache.beam/beam-sdks-java-io-mongodb/0.5.0-incubating-SNAPSHOT/beam-sdks-java-io-mongodb-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/extensions/sorter/pom.xml to org.apache.beam/beam-sdks-java-extensions-sorter/0.5.0-incubating-SNAPSHOT/beam-sdks-java-extensions-sorter-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/pom.xml to org.apache.beam/beam-sdks-parent/0.5.0-incubating-SNAPSHOT/beam-sdks-parent-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/extensions/join-library/pom.xml to org.apache.beam/beam-sdks-java-extensions-join-library/0.5.0-incubating-SNAPSHOT/beam-sdks-java-extensions-join-library-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/pom.xml to org.apache.beam/beam-runners-parent/0.5.0-incubating-SNAPSHOT/beam-runners-parent-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/io/jdbc/pom.xml to org.apache.beam/beam-sdks-java-io-jdbc/0.5.0-incubating-SNAPSHOT/beam-sdks-java-io-jdbc-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/extensions/pom.xml to org.apache.beam/beam-sdks-java-extensions-parent/0.5.0-incubating-SNAPSHOT/beam-sdks-java-extensions-parent-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/examples/pom.xml to org.apache.beam/beam-examples-parent/0.5.0-incubating-SNAPSHOT/beam-examples-parent-0.5.0-incubating-SNAPSHOT.pomchannel stoppedSetting status of 80c90c1 to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/5879/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install
--none--

asfbot · 2016-12-14T04:14:08Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/5882/

Build result: FAILURE

[...truncated 6623 lines...]No artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: Runners :: Google Cloud Dataflow #5881 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/apex/pom.xml to org.apache.beam/beam-runners-apex/0.5.0-incubating-SNAPSHOT/beam-runners-apex-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/io/jms/pom.xml to org.apache.beam/beam-sdks-java-io-jms/0.5.0-incubating-SNAPSHOT/beam-sdks-java-io-jms-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: SDKs :: Java :: IO :: JMS #5881 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/flink/pom.xml to org.apache.beam/beam-runners-flink-parent/0.5.0-incubating-SNAPSHOT/beam-runners-flink-parent-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: Runners :: Flink #5881 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/io/mongodb/pom.xml to org.apache.beam/beam-sdks-java-io-mongodb/0.5.0-incubating-SNAPSHOT/beam-sdks-java-io-mongodb-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: SDKs :: Java :: IO :: MongoDB #5881 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/extensions/sorter/pom.xml to org.apache.beam/beam-sdks-java-extensions-sorter/0.5.0-incubating-SNAPSHOT/beam-sdks-java-extensions-sorter-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: SDKs :: Java :: Extensions :: Sorter #5881 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/pom.xml to org.apache.beam/beam-sdks-parent/0.5.0-incubating-SNAPSHOT/beam-sdks-parent-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: SDKs #5881 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/extensions/join-library/pom.xml to org.apache.beam/beam-sdks-java-extensions-join-library/0.5.0-incubating-SNAPSHOT/beam-sdks-java-extensions-join-library-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: SDKs :: Java :: Extensions :: Join library #5881 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/runners/pom.xml to org.apache.beam/beam-runners-parent/0.5.0-incubating-SNAPSHOT/beam-runners-parent-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: Runners #5881 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/io/jdbc/pom.xml to org.apache.beam/beam-sdks-java-io-jdbc/0.5.0-incubating-SNAPSHOT/beam-sdks-java-io-jdbc-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: SDKs :: Java :: IO :: JDBC #5881 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/sdks/java/extensions/pom.xml to org.apache.beam/beam-sdks-java-extensions-parent/0.5.0-incubating-SNAPSHOT/beam-sdks-java-extensions-parent-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: SDKs :: Java :: Extensions #5881 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall/examples/pom.xml to org.apache.beam/beam-examples-parent/0.5.0-incubating-SNAPSHOT/beam-examples-parent-0.5.0-incubating-SNAPSHOT.pomchannel stoppedSetting status of 9445be7 to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/5882/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install
--none--

asfbot · 2016-12-14T05:27:01Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/5889/

Build result: FAILURE

[...truncated 6596 lines...]No artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: Runners :: Google Cloud Dataflow #5888 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/runners/apex/pom.xml to org.apache.beam/beam-runners-apex/0.5.0-incubating-SNAPSHOT/beam-runners-apex-0.5.0-incubating-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/sdks/java/io/jms/pom.xml to org.apache.beam/beam-sdks-java-io-jms/0.5.0-incubating-SNAPSHOT/beam-sdks-java-io-jms-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: SDKs :: Java :: IO :: JMS #5888 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/runners/flink/pom.xml to org.apache.beam/beam-runners-flink-parent/0.5.0-incubating-SNAPSHOT/beam-runners-flink-parent-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: Runners :: Flink #5886 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/sdks/java/io/mongodb/pom.xml to org.apache.beam/beam-sdks-java-io-mongodb/0.5.0-incubating-SNAPSHOT/beam-sdks-java-io-mongodb-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: SDKs :: Java :: IO :: MongoDB #5888 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/sdks/java/extensions/sorter/pom.xml to org.apache.beam/beam-sdks-java-extensions-sorter/0.5.0-incubating-SNAPSHOT/beam-sdks-java-extensions-sorter-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: SDKs :: Java :: Extensions :: Sorter #5886 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/sdks/pom.xml to org.apache.beam/beam-sdks-parent/0.5.0-incubating-SNAPSHOT/beam-sdks-parent-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: SDKs #5888 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/sdks/java/extensions/join-library/pom.xml to org.apache.beam/beam-sdks-java-extensions-join-library/0.5.0-incubating-SNAPSHOT/beam-sdks-java-extensions-join-library-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: SDKs :: Java :: Extensions :: Join library #5886 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/runners/pom.xml to org.apache.beam/beam-runners-parent/0.5.0-incubating-SNAPSHOT/beam-runners-parent-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: Runners #5888 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/sdks/java/io/jdbc/pom.xml to org.apache.beam/beam-sdks-java-io-jdbc/0.5.0-incubating-SNAPSHOT/beam-sdks-java-io-jdbc-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: SDKs :: Java :: IO :: JDBC #5888 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/sdks/java/extensions/pom.xml to org.apache.beam/beam-sdks-java-extensions-parent/0.5.0-incubating-SNAPSHOT/beam-sdks-java-extensions-parent-0.5.0-incubating-SNAPSHOT.pomNo artifacts from beam_PreCommit_Java_MavenInstall » Apache Beam :: SDKs :: Java :: Extensions #5886 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Java_MavenInstall@2/examples/pom.xml to org.apache.beam/beam-examples-parent/0.5.0-incubating-SNAPSHOT/beam-examples-parent-0.5.0-incubating-SNAPSHOT.pomchannel stoppedSetting status of a847248 to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/5889/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install
--none--

asfbot · 2016-12-14T06:30:10Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/5891/
--none--

kennknowles · 2016-12-19T04:09:24Z

Given that you'll need to rebase to resolve conflicts, may I suggest also squashing the commits a bit?

davorbonaci · 2017-02-27T18:06:54Z

This seems like a lot to take for the first stable release. Perhaps we should target this for the first release after that one. What we should do now is the refactoring of Bound with this in mind, so this can be done nicely and compatibly.

asfbot · 2017-02-27T19:04:18Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7901/

Build result: FAILURE

[...truncated 1.11 MB...] at hudson.remoting.UserRequest.perform(UserRequest.java:50) at hudson.remoting.Request$2.run(Request.java:336) at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Caused by: org.apache.maven.plugin.MojoExecutionException: Archetype IT 'basic' failed: Some content are not equals at org.apache.maven.archetype.mojos.IntegrationTestMojo.execute(IntegrationTestMojo.java:269) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) ... 31 more2017-02-27T19:04:09.905 [ERROR] 2017-02-27T19:04:09.905 [ERROR] Re-run Maven using the -X switch to enable full debug logging.2017-02-27T19:04:09.905 [ERROR] 2017-02-27T19:04:09.905 [ERROR] For more information about the errors and possible solutions, please read the following articles:2017-02-27T19:04:09.905 [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException2017-02-27T19:04:09.905 [ERROR] 2017-02-27T19:04:09.905 [ERROR] After correcting the problems, you can resume the build with the command2017-02-27T19:04:09.905 [ERROR] mvn -rf :beam-sdks-java-maven-archetypes-starterchannel stoppedSetting status of e3069d2 to FAILURE with url https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/7901/ and message: 'Build finished. 'Using context: Jenkins: Maven clean install
--none--

reuvenlax · 2017-03-12T03:29:32Z

This PR is technically backwards incompatible, as it changes the type of Bound to Bound<T>. I guess we could do this separately, however it would be strange to introduce an unused generic type.

…

On Mon, Feb 27, 2017 at 10:06 AM, Davor Bonaci ***@***.***> wrote: This seems like a lot to take for the first stable release. Perhaps we should target this for the first release after that one. What we should do now is the refactoring of Bound with this in mind, so this can be done nicely and compatibly. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1609 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AUGE1Rxk8pzCLkKPlJkT9FyexbRC5MK7ks5rgxDBgaJpZM4LMa96> .

davorbonaci · 2017-04-17T16:52:36Z

@reuvenlax, is this PR now obsolete?

reuvenlax · 2017-05-11T13:59:38Z

It needs tests, however I wanted to get comments on the approach before adding tests

…cy, and get failed inserts back.

jkff · 2017-05-18T22:13:08Z

The overall approach looks reasonable. Waiting for tests.

reuvenlax · 2017-05-26T02:23:20Z

@jkff added tests.

reuvenlax · 2017-05-26T02:31:59Z

retest this please.

reuvenlax · 2017-05-26T22:13:57Z

retest this please.

reuvenlax · 2017-05-26T23:32:25Z

retest this please.

jkff · 2017-05-30T22:36:07Z

The jenkins failure is not flaky, it's an issue spotted by findbugs.

reuvenlax · 2017-05-30T22:56:20Z

@jkff the most recent failure yes. Previously it was failing with strange jenkins errors.

…output.

reuvenlax · 2017-05-31T05:07:25Z

Fixed issues with windows.

asfbot · 2017-05-31T05:42:12Z

--none--

jkff

Thanks!

jkff · 2017-05-18T21:43:49Z

...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/InsertRetryPolicy.java

+    };
+  }
+
+  /**


Is this a useful policy, if it causes us to retry errors for which retrying never helps?

I believe yes, and it's the correct default policy that errs on the side of never losing data. Users have to explicitly write code to handle errors (extract the PCollection, write it somewhere, etc.). If they get stuck due to a persistent error they can always update. However if the default is to drop those records, updating with error-handling code won't help - the records are already gone.

jkff · 2017-05-18T21:45:51Z

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java

        streamingInserts.setTestServices(getBigQueryServices());
        return rowsWithDestination.apply(streamingInserts);
      } else {
+        checkArgument(getFailedInsertRetryPolicy() == null,
+            "Record-insert retry policies are not supported when using BigQuery load jobs.");


More like "when writing an unbounded collection" cause this is what the user needs to fix to get their pipeline working.

I didn't want to write that, because tying "unbounded collection" to BQ streaming inserts is an artifact of the current code, and arguably not a correct one. There are plans to fix this and have the insert method chosen independently of the input PCollection.

jkff · 2017-05-31T19:41:02Z

...e-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java

@@ -656,8 +657,11 @@ public void deleteDataset(String projectId, String datasetId)
    }

    @VisibleForTesting
-    long insertAll(TableReference ref, List<TableRow> rowList, @Nullable List<String> insertIdList,
-        BackOff backoff, final Sleeper sleeper) throws IOException, InterruptedException {
+    long insertAll(TableReference ref, List<ValueInSingleWindow<TableRow>> rowList,


ValueInSingleWindow seems out of place here (wrong level of abstraction), and I don't see the insertAll code using anything except the value itself.

I realize that this is being used to let caller code maintain the context from which every errored value originated. Perhaps maintain it by returning a list of indices of failed inserts in rowList? Or perhaps still better, return a list of tuples (failed index, error proto)?

I actually tried the list of indices first, but abandoned it because the code complexity grew too much (it's tricky for insertAll to keep track of the failed index today, and I'd prefer not to rewrite that function completely - there have been multiple subtle bugs in the past). Also I think this is a better abstraction - WindowedValues are used everywhere else, and we may well use them in the future in insertAll (e.g. for partitioned tables).

jkff · 2017-05-31T19:41:20Z

...e-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServicesImpl.java

-                retryRows.add(rowsToPublish.get(errorIndex));
-                if (retryIds != null) {
-                  retryIds.add(idsToPublish.get(errorIndex));
+                boolean skipRetry = !retryPolicy.shouldRetry(new InsertRetryPolicy.Context(error));


How about boolean shouldRetry and no inversion :)

jkff · 2017-05-31T19:43:26Z

...ogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/InsertRetryPolicy.java

+  /**
+   * Retry all failures except for known persistent errors.
+   */
+  public static InsertRetryPolicy dontRetryPersistentErrors() {


How about retryTransientErrors

jkff · 2017-05-31T19:44:21Z

...oogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StreamingInserts.java

-    return writes.apply(new StreamingWriteTables().withTestServices(bigQueryServices));
+    StreamingWriteTables streamingWriteTables =
+        new StreamingWriteTables().withTestServices(bigQueryServices);
+    streamingWriteTables.setInsertRetryPolicy(retryPolicy);


Can retry policy be a with method rather than a setter?

jkff · 2017-05-31T19:48:08Z

...e-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StreamingWriteTables.java

@@ -77,7 +86,9 @@ public WriteResult expand(PCollection<KV<TableDestination, TableRow>> input) {
    // different unique ids, this implementation relies on "checkpointing", which is
    // achieved as a side effect of having StreamingWriteFn immediately follow a GBK,
    // performed by Reshuffle.
-    tagged
+    TupleTag<Void> mainOutput = new TupleTag<Void>() {};


Same usual comments about tuple tag anonymous classes capturing their surrounding context. Instead, give the tags a name, and setCoder on the tagged output collections until it works.

jkff · 2017-05-31T19:56:20Z

.../io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/WriteResult.java

  private final Pipeline pipeline;
+  private final TupleTag<TableRow> failedInsertsTag = new TupleTag<>();
+  private final PCollection<TableRow> failedInserts;


Do you think it makes sense to return not just the rows, but also their errors, as in PCollection<KV<TableRow, List<ErrorProto>>> (or perhaps introduce a special type FailedInsertRow containing these)?

I thought about it, but I had a trouble imagining a good use case. If we did, I'd prefer FailedInsertRow over a KV.

jkff · 2017-05-31T22:52:05Z

...e-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StreamingWriteTables.java

@@ -86,8 +86,8 @@ public WriteResult expand(PCollection<KV<TableDestination, TableRow>> input) {
    // different unique ids, this implementation relies on "checkpointing", which is
    // achieved as a side effect of having StreamingWriteFn immediately follow a GBK,
    // performed by Reshuffle.
-    TupleTag<Void> mainOutput = new TupleTag<Void>() {};
-    TupleTag<TableRow> failedOutput = new TupleTag<TableRow>() {};
+    TupleTag<Void> mainOutput = new TupleTag<Void>("mainOutput") {};


These are still anonymous classes - please remove {}.

Done, and added setCoder

jkff · 2017-05-31T22:54:41Z

...oogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StreamingInserts.java

+  }
+
+  /** Constructor. */
+  public StreamingInserts(CreateDisposition createDisposition,


Are all three constructors necessary?

Do they need to be public?

Can you use constructor delegation - this(...);?

Done. The second constructor must be private, as our API surface tests aren't happy if BigQueryServices is exposed in a public method.

jkff · 2017-05-31T22:55:09Z

...oogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StreamingInserts.java

+    this.createDisposition = createDisposition;
+    this.dynamicDestinations = dynamicDestinations;
+    this.bigQueryServices = bigQueryServices;
+    this.retryPolicy = retryPolicy;
  }

  void setTestServices(BigQueryServices bigQueryServices) {


Maybe this should be a with method too, now that you have such convenient constructors.

jkff · 2017-05-31T22:57:15Z

...a/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java

@@ -861,6 +863,10 @@ static String getExtractDestinationUri(String extractDestinationDir) {
      return toBuilder().setTableDescription(tableDescription).build();
    }

+    public Write<T> withFailedInsertRetryPolicy(InsertRetryPolicy retryPolicy) {


Add a javadoc and explain that currently it works only when applying to an unbounded collection.
(the comment will need to be changed when insert method is selectable explicitly - but for now, it would feel icky to me to document that it works only when using "streaming inserts", without providing the user with a lever to use streaming inserts on a bounded collection in case they get this error)

jkff · 2017-05-31T22:57:30Z

...oogle-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryServices.java

@@ -161,9 +162,14 @@ void deleteDataset(String projectId, String datasetId)
    /**
     * Inserts {@link TableRow TableRows} with the specified insertIds if not null.
     *
+     * <p>If any insert fail permanently according to the retry policy, those indices are added


ah - leftover from a previous iteration of the code :)

coveralls · 2017-05-31T23:58:13Z

Coverage decreased (-0.05%) to 70.689% when pulling 15faa8e on reuvenlax:bq_dead_letter into d6cc850 on apache:master.

coveralls · 2017-06-01T02:04:19Z

Coverage decreased (-0.06%) to 70.679% when pulling 3768ec2 on reuvenlax:bq_dead_letter into d6cc850 on apache:master.

reuvenlax · 2017-06-01T07:33:20Z

retest this please.

jkff

LGTM, will merge now. Thanks!

reuvenlax force-pushed the bq_dead_letter branch from 2af0de1 to cc088fd Compare December 14, 2016 01:16

reuvenlax force-pushed the bq_dead_letter branch from e3069d2 to 5d0a2e9 Compare May 11, 2017 13:58

Dead-letter support for BigQuery. Allow users to specify a retry poli…

b9a027a

…cy, and get failed inserts back.

reuvenlax force-pushed the bq_dead_letter branch from 5d0a2e9 to b9a027a Compare May 17, 2017 05:50

Fix broken tests.

ada5ec7

reuvenlax added 3 commits May 24, 2017 09:25

Start adding tests.

1df8b62

More testing.

723bf46

Add tests for dead-letter support.

949ce88

Properly propagate window and timestamp information into dead-letter …

76c1b28

…output.

jkff requested changes May 31, 2017

View reviewed changes

Address review comments.

15faa8e

jkff requested changes May 31, 2017

View reviewed changes

Address comments.

3768ec2

jkff approved these changes Jun 2, 2017

View reviewed changes

asfgit closed this in 53c9bf4 Jun 2, 2017

reuvenlax deleted the bq_dead_letter branch December 9, 2018 23:01

+                  };
+                }
+                /**

BEAM-1151 Add failure handling to BigQueryIO.Write #1609

BEAM-1151 Add failure handling to BigQueryIO.Write #1609

Conversation

reuvenlax commented Dec 14, 2016 • edited

asfbot commented Dec 14, 2016

Build result: FAILURE

asfbot commented Dec 14, 2016

Build result: FAILURE

asfbot commented Dec 14, 2016

Build result: FAILURE

asfbot commented Dec 14, 2016

Build result: FAILURE

asfbot commented Dec 14, 2016

kennknowles commented Dec 19, 2016

davorbonaci commented Feb 27, 2017

asfbot commented Feb 27, 2017

Build result: FAILURE

reuvenlax commented Mar 12, 2017 via email

davorbonaci commented Apr 17, 2017

reuvenlax commented May 11, 2017

jkff commented May 18, 2017

reuvenlax commented May 26, 2017

reuvenlax commented May 26, 2017

reuvenlax commented May 26, 2017

reuvenlax commented May 26, 2017

jkff commented May 30, 2017

reuvenlax commented May 30, 2017

reuvenlax commented May 31, 2017

asfbot commented May 31, 2017

jkff left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

reuvenlax May 31, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented May 31, 2017

coveralls commented Jun 1, 2017

reuvenlax commented Jun 1, 2017

jkff left a comment

Choose a reason for hiding this comment

reuvenlax commented Dec 14, 2016 •

edited

reuvenlax May 31, 2017 •

edited