resolved conflicts

apache · Jun 1, 2022 · 3f8e8da · 3f8e8da
2 parents fc16a99 + 23aeca4
commit 3f8e8da
Show file tree

Hide file tree

Showing 246 changed files with 8,342 additions and 1,544 deletions.
diff --git a/.github/ISSUE_TEMPLATE/bug.yml b/.github/ISSUE_TEMPLATE/bug.yml
@@ -31,6 +31,10 @@ body:
     attributes:
       label: What happened?
       description: Please describe your issue, including the version of Beam and any other languages or tools being used.
+      placeholder: |
+          Please include steps to reproduce your issue.
+
+          If this isn't a bug and you have a question or support request, please email user@apache.beam.com with a description of the problem instead of opening this issue.
     validations:
       required: true
   - type: dropdown

diff --git a/.test-infra/jenkins/job_PostCommit_Java_Sickbay.groovy b/.test-infra/jenkins/job_PostCommit_Java_Sickbay.groovy
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import CommonJobProperties as commonJobProperties
+import PostcommitJobBuilder
+
+// This job runs the Java sickbay tests.
+PostcommitJobBuilder.postCommitJob('beam_PostCommit_Java_Sickbay',
+    'Run Java Sickbay', 'Java Sickbay Tests', this) {
+
+      description('Run Java Sickbay Tests')
+
+      // Set common parameters.
+      commonJobProperties.setTopLevelMainJobProperties(delegate, 'master', 120)
+
+      publishers {
+        archiveJunit('**/build/test-results/**/*.xml')
+      }
+
+      // Execute shell command to run sickbay tests.
+      steps {
+        gradle {
+          rootBuildScriptDir(commonJobProperties.checkoutDir)
+          tasks(':javaPostCommitSickbay')
+          commonJobProperties.setGradleSwitches(delegate)
+        }
+      }
+    }
diff --git a/CHANGES.md b/CHANGES.md
@@ -85,23 +85,41 @@
 
 * ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)).
 
-# [2.39.0] - Unreleased, Cut
+# [2.39.0] - 2022-05-25
 
 ## Highlights
 
-* Watermark estimation is now supported in the Go SDK ([BEAM-11105](https://issues.apache.org/jira/browse/BEAM-11105))
+* Watermark estimation is now supported in the Go SDK ([BEAM-11105](https://issues.apache.org/jira/browse/BEAM-11105)).
+* Support for impersonation credentials added to dataflow runner in the Java and Python SDK ([BEAM-14014](https://issues.apache.org/jira/browse/BEAM-14014)).
+* Implemented Apache PulsarIO ([BEAM-8218](https://issues.apache.org/jira/browse/BEAM-8218)).
 
 ## I/Os
 
 * JmsIO gains the ability to map any kind of input to any subclass of `javax.jms.Message` (Java) ([BEAM-16308](https://issues.apache.org/jira/browse/BEAM-16308)).
 * JmsIO introduces the ability to write to dynamic topics (Java) ([BEAM-16308](https://issues.apache.org/jira/browse/BEAM-16308)).
   * A `topicNameMapper` must be set to extract the topic name from the input value.
   * A `valueMapper` must be set to convert the input value to JMS message.
+* Reduce number of threads spawned by BigqueryIO StreamingInserts (
+  [BEAM-14283](https://issues.apache.org/jira/browse/BEAM-14283)).
+* Implemented Apache PulsarIO ([BEAM-8218](https://issues.apache.org/jira/browse/BEAM-8218)).
+
 
 ## New Features / Improvements
 
+* Support for flink scala 2.12, because most of the libraries support version 2.12 onwards. ([beam-14386](https://issues.apache.org/jira/browse/BEAM-14386))
 * 'Manage Clusters' JupyterLab extension added for users to configure usage of Dataproc clusters managed by Interactive Beam (Python) ([BEAM-14130](https://issues.apache.org/jira/browse/BEAM-14130)).
 * Pipeline drain support added for Go SDK ([BEAM-11106](https://issues.apache.org/jira/browse/BEAM-11106)). **Note: this feature is not yet fully validated and should be treated as experimental in this release.**
+* `DataFrame.unstack()`, `DataFrame.pivot() ` and  `Series.unstack()`
+  implemented for DataFrame API ([BEAM-13948](https://issues.apache.org/jira/browse/BEAM-13948), [BEAM-13966](https://issues.apache.org/jira/browse/BEAM-13966)).
+* Support for impersonation credentials added to dataflow runner in the Java and Python SDK ([BEAM-14014](https://issues.apache.org/jira/browse/BEAM-14014)).
+* Implemented Jupyterlab extension for managing Dataproc clusters ([BEAM-14130](https://issues.apache.org/jira/browse/BEAM-14130)).
+* ExternalPythonTransform API added for easily invoking Python transforms from
+  Java ([BEAM-14143](https://issues.apache.org/jira/browse/BEAM-14143)).
+* Added Add support for Elasticsearch 8.x ([BEAM-14003](https://issues.apache.org/jira/browse/BEAM-14003)).
+* Shard aware Kinesis record aggregation (AWS Sdk v2), ([BEAM-14104](https://issues.apache.org/jira/browse/BEAM-14104)).
+* Upgrade to ZetaSQL 2022.04.1 ([BEAM-14348](https://issues.apache.org/jira/browse/BEAM-14348)).
+* Fixed ReadFromBigQuery cannot be used with the interactive runner ([BEAM-14112](https://issues.apache.org/jira/browse/BEAM-14112)).
+
 
 ## Breaking Changes
 
@@ -113,20 +131,19 @@
         .withValueMapper(new TextMessageMapper());
 ```
 * Coders in Python are expected to inherit from Coder. ([BEAM-14351](https://issues.apache.org/jira/browse/BEAM-14351)).
+* New abstract method `metadata()` added to io.filesystem.FileSystem in the
+  Python SDK. ([BEAM-14314](https://issues.apache.org/jira/browse/BEAM-14314))
 
 ## Deprecations
 
-* Support for flink scala 2.12, because most of the libraries support version 2.12 onwards. ([beam-14386](https://issues.apache.org/jira/browse/BEAM-14386))
 * Flink 1.11 is no longer supported ([BEAM-14139](https://issues.apache.org/jira/browse/BEAM-14139)).
 * Python 3.6 is no longer supported ([BEAM-13657](https://issues.apache.org/jira/browse/BEAM-13657)).
 
 ## Bugfixes
 
-* Fixed X (Java/Python) ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)).
-
-## Known Issues
+* Fixed Java Spanner IO NPE when ProjectID not specified in template executions (Java) ([BEAM-14405](https://issues.apache.org/jira/browse/BEAM-14405)).
+* Fixed potential NPE in BigQueryServicesImpl.getErrorInfo (Java) ([BEAM-14133](https://issues.apache.org/jira/browse/BEAM-14133)).
 
-* ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)).
 
 # [2.38.0] - 2022-04-20
 

diff --git a/build.gradle.kts b/build.gradle.kts
@@ -207,6 +207,14 @@ tasks.register("javaPostCommit") {
   dependsOn(":sdks:java:io:neo4j:integrationTest")
 }
 
+tasks.register("javaPostCommitSickbay") {
+  dependsOn(":runners:samza:validatesRunnerSickbay")
+  dependsOn(":runners:flink:validatesRunnerSickbay")
+  dependsOn(":runners:spark:validatesRunnerSickbay")
+  dependsOn(":runners:direct-java:validatesRunnerSickbay")
+  dependsOn(":runners:portability:java:validatesRunnerSickbay")
+}
+
 tasks.register("javaHadoopVersionsTest") {
   dependsOn(":sdks:java:io:hadoop-common:hadoopVersionsTest")
   dependsOn(":sdks:java:io:hadoop-file-system:hadoopVersionsTest")

diff --git a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -356,7 +356,22 @@ class BeamModulePlugin implements Plugin<Project> {
   }
 
   def isRelease(Project project) {
-    return project.hasProperty('isRelease')
+    return parseBooleanProperty(project, 'isRelease');
+  }
+
+  /**
+   * Parses -Pprop as true for use as a flag, and otherwise uses Groovy's toBoolean
+   */
+  def parseBooleanProperty(Project project, String property) {
+    if (!project.hasProperty(property)) {
+      return false;
+    }
+
+    if (project.getProperty(property) == "") {
+      return true;
+    }
+
+    return project.getProperty(property).toBoolean();
   }
 
   def defaultArchivesBaseName(Project p) {
@@ -649,6 +664,7 @@ class BeamModulePlugin implements Plugin<Project> {
         kafka                                       : "org.apache.kafka:kafka_2.11:$kafka_version",
         kafka_clients                               : "org.apache.kafka:kafka-clients:$kafka_version",
         mockito_core                                : "org.mockito:mockito-core:3.7.7",
+        mockito_inline                              : "org.mockito:mockito-inline:4.5.1",
         mongo_java_driver                           : "org.mongodb:mongo-java-driver:3.12.10",
         nemo_compiler_frontend_beam                 : "org.apache.nemo:nemo-compiler-frontend-beam:$nemo_version",
         netty_all                                   : "io.netty:netty-all:$netty_version",
@@ -929,7 +945,7 @@ class BeamModulePlugin implements Plugin<Project> {
           'org.checkerframework.checker.nullness.NullnessChecker'
         ]
 
-        if (project.findProperty('enableCheckerFramework') || project.jenkins.isCIBuild) {
+        if (parseBooleanProperty(project, 'enableCheckerFramework') || project.jenkins.isCIBuild) {
           skipCheckerFramework = false
         } else {
           skipCheckerFramework = true

diff --git a/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/snippets/Snippets.kt b/examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/snippets/Snippets.kt
@@ -269,8 +269,8 @@ object Snippets {
                     BigQueryIO.write<WeatherData>()
                             .to(
                                     object : DynamicDestinations<WeatherData, Long>() {
-                                        override fun getDestination(elem: ValueInSingleWindow<WeatherData>): Long? {
-                                            return elem.value!!.year
+                                        override fun getDestination(elem: ValueInSingleWindow<WeatherData>?): Long {
+                                            return elem!!.value!!.year
                                         }
 
                                         override fun getTable(destination: Long?): TableDestination {

diff --git a/gradle.properties b/gradle.properties
@@ -37,5 +37,5 @@ javaVersion=1.8
 docker_image_default_repo_root=apache
 docker_image_default_repo_prefix=beam_
 
-flink_versions=1.12,1.13,1.14
+flink_versions=1.12,1.13,1.14,1.15
 
diff --git a/learning/katas/python/Examples/section-remote-info.yaml b/learning/katas/python/Examples/section-remote-info.yaml
@@ -1,2 +1,2 @@
 id: 85647
-update_date: Mon, 09 Mar 2020 14:34:14 UTC
+update_date: Thu, 12 May 2022 13:06:24 UTC
diff --git a/learning/katas/python/Introduction/Hello Beam/Hello Beam/tests.py b/learning/katas/python/Introduction/Hello Beam/Hello Beam/tests.py
@@ -20,6 +20,9 @@
 def test_output():
     output = get_file_output()
 
+    # Remove warning line about docker and Python versions
+    output = [x for x in output if not x.startswith("WARNING")]
+
     if len(output) == 1 and 'Hello Beam' in output:
         passed()
     else:

diff --git a/learning/katas/python/Triggers/Early Triggers/Early Triggers/task-remote-info.yaml b/learning/katas/python/Triggers/Early Triggers/Early Triggers/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 1315717712
+update_date: Thu, 01 Jan 1970 00:00:00 UTC
diff --git a/learning/katas/python/Triggers/Early Triggers/lesson-remote-info.yaml b/learning/katas/python/Triggers/Early Triggers/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 415852098
+update_date: Thu, 01 Jan 1970 00:00:00 UTC
+unit: 0
diff --git a/learning/katas/python/Triggers/Event Time Triggers/Event Time Triggers/task-remote-info.yaml b/learning/katas/python/Triggers/Event Time Triggers/Event Time Triggers/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 825593025
+update_date: Thu, 01 Jan 1970 00:00:00 UTC
diff --git a/learning/katas/python/Triggers/Event Time Triggers/lesson-remote-info.yaml b/learning/katas/python/Triggers/Event Time Triggers/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 1858884960
+update_date: Thu, 01 Jan 1970 00:00:00 UTC
+unit: 0
diff --git a/...s/python/Triggers/Window Accumulation Mode/Window Accumulation Mode/task-remote-info.yaml b/...s/python/Triggers/Window Accumulation Mode/Window Accumulation Mode/task-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 84386334
+update_date: Thu, 01 Jan 1970 00:00:00 UTC
diff --git a/learning/katas/python/Triggers/Window Accumulation Mode/lesson-remote-info.yaml b/learning/katas/python/Triggers/Window Accumulation Mode/lesson-remote-info.yaml
@@ -0,0 +1,3 @@
+id: 158109835
+update_date: Thu, 01 Jan 1970 00:00:00 UTC
+unit: 0
diff --git a/learning/katas/python/Triggers/section-remote-info.yaml b/learning/katas/python/Triggers/section-remote-info.yaml
@@ -0,0 +1,2 @@
+id: 334072318
+update_date: Thu, 01 Jan 1970 00:00:00 UTC
diff --git a/learning/katas/python/course-info.yaml b/learning/katas/python/course-info.yaml
@@ -29,5 +29,5 @@ content:
 - Common Transforms
 - IO
 - Windowing
+- Examples
 - Triggers
-- Examples
diff --git a/learning/katas/python/requirements.txt b/learning/katas/python/requirements.txt
@@ -14,7 +14,7 @@
 #   See the License for the specific language governing permissions and
 #   limitations under the License.
 
-apache-beam==2.36.0
-apache-beam[test]==2.36.0
+apache-beam==2.38.0
+apache-beam[test]==2.38.0
 
-pytz~=2021.3
+pytz~=2022.1
diff --git a/learning/katas/python/test_helper.py b/learning/katas/python/test_helper.py
@@ -34,8 +34,7 @@ def get_file_output(encoding="utf-8", path=sys.argv[-1], arg_string=""):
     """
     import subprocess
 
-    proc = subprocess.Popen([sys.executable, path], stdin=subprocess.PIPE, stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+    proc = subprocess.Popen([sys.executable, path], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
     if arg_string:
         for arg in arg_string.split("\n"):
             proc.stdin.write(bytearray(str(arg) + "\n", encoding))