preparing release 0.23.2

GoogleCloudDataproc · Jan 20, 2022 · 318184a · 318184a
1 parent a5e7585
commit 318184a
Show file tree

Hide file tree

Showing 21 changed files with 139 additions and 69 deletions.
diff --git a/CHANGES.md b/CHANGES.md
@@ -1,5 +1,15 @@
 # Release Notes
 
+## 0.23.2 - 2022-01-19
+* PR #521: Added Arrow compression options to the
+  spark-bigquery-with-dependencies_2.* connectors
+* PR #526: Added the option to use parent project for the metadata/jobs API as
+  well
+* BigQuery API has been upgraded to version 2.3.3
+* BigQuery Storage API has been upgraded to version 2.4.2
+* gRPC has been upgraded to version 1.42.1
+* Netty has been upgraded to version 4.1.70.Final
+
 ## 0.23.1 - 2021-12-08
 * Issue #501: Fixed using avro as an intermediate type for writing.
 

diff --git a/README.md b/README.md
@@ -68,9 +68,9 @@ The latest version of the connector is publicly available in the following links
 
 | version | Link |
 | --- | --- |
-| Scala 2.11 | `gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.23.1.jar` ([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.23.1.jar)) |
-| Scala 2.12 | `gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.23.1.jar` ([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.23.1.jar)) |
-| Spark 2.4  | `gs://spark-lib/bigquery/spark-2.4-bigquery-0.23.1-preview.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-2.4-bigquery-0.23.1-preview.jar)) |
+| Scala 2.11 | `gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.23.2.jar` ([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.23.2.jar)) |
+| Scala 2.12 | `gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.23.2.jar` ([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.23.2.jar)) |
+| Spark 2.4  | `gs://spark-lib/bigquery/spark-2.4-bigquery-0.23.2-preview.jar`([HTTP link](https://storage.googleapis.com/spark-lib/bigquery/spark-2.4-bigquery-0.23.2-preview.jar)) |
 
 **Note:** If you are using scala jars please use the jar relevant to your Spark installation. Starting from Spark 2.4 onwards there is an
 option to use the Java only jar.
@@ -82,9 +82,9 @@ repository. It can be used using the `--packages` option or the
 
 | version | Connector Artifact |
 | --- | --- |
-| Scala 2.11 | `com.google.cloud.spark:spark-bigquery-with-dependencies_2.11:0.23.1` |
-| Scala 2.12 | `com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.23.1` |
-| Spark 2.4  | `com.google.cloud.spark:spark-2.4-bigquery:0.23.1-preview` |
+| Scala 2.11 | `com.google.cloud.spark:spark-bigquery-with-dependencies_2.11:0.23.2` |
+| Scala 2.12 | `com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.23.2` |
+| Spark 2.4  | `com.google.cloud.spark:spark-2.4-bigquery:0.23.2-preview` |
 
 If you want to keep up with the latest version of the connector the following links can be used. Notice that for production
 environments where the connector version should be pinned, one of the above links should be used.
@@ -812,9 +812,9 @@ creating the job or added during runtime. See examples below:
 1) Adding python files while launching pyspark
 ```
 # use appropriate version for jar depending on the scala version
-pyspark --jars gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.23.1.jar
-  --py-files gs://spark-lib/bigquery/spark-bigquery-support-0.23.1.zip
-  --files gs://spark-lib/bigquery/spark-bigquery-support-0.23.1.zip
+pyspark --jars gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.23.2.jar
+  --py-files gs://spark-lib/bigquery/spark-bigquery-support-0.23.2.zip
+  --files gs://spark-lib/bigquery/spark-bigquery-support-0.23.2.zip
 ```
 
 2) Adding python files in Jupyter Notebook
@@ -823,9 +823,9 @@ from pyspark.sql import SparkSession
 # use appropriate version for jar depending on the scala version
 spark = SparkSession.builder\
   .appName('BigNumeric')\
-  .config('spark.jars', 'gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.23.1.jar')\
-  .config('spark.submit.pyFiles', 'gs://spark-lib/bigquery/spark-bigquery-support-0.23.1.zip')\
-  .config('spark.files', 'gs://spark-lib/bigquery/spark-bigquery-support-0.23.1.zip')\
+  .config('spark.jars', 'gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.23.2.jar')\
+  .config('spark.submit.pyFiles', 'gs://spark-lib/bigquery/spark-bigquery-support-0.23.2.zip')\
+  .config('spark.files', 'gs://spark-lib/bigquery/spark-bigquery-support-0.23.2.zip')\
   .getOrCreate()
 ```
 
@@ -834,10 +834,10 @@ spark = SparkSession.builder\
 # use appropriate version for jar depending on the scala version
 spark = SparkSession.builder\
   .appName('BigNumeric')\
-  .config('spark.jars', 'gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.23.1.jar')\
+  .config('spark.jars', 'gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.11-0.23.2.jar')\
   .getOrCreate()
 
-spark.sparkContext.addPyFile("gs://spark-lib/bigquery/spark-bigquery-support-0.23.1.zip")
+spark.sparkContext.addPyFile("gs://spark-lib/bigquery/spark-bigquery-support-0.23.2.zip")
 ```
 
 Usage Example:
@@ -920,7 +920,7 @@ using the following code:
 ```python
 from pyspark.sql import SparkSession
 spark = SparkSession.builder\
-  .config("spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.11:0.23.1")\
+  .config("spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.11:0.23.2")\
   .getOrCreate()
 df = spark.read.format("bigquery")\
   .load("dataset.table")
@@ -929,15 +929,15 @@ df = spark.read.format("bigquery")\
 **Scala:**
 ```python
 val spark = SparkSession.builder
-  .config("spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.11:0.23.1")
+  .config("spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.11:0.23.2")
   .getOrCreate()
 val df = spark.read.format("bigquery")
   .load("dataset.table")
 ```
 
 In case Spark cluster is using Scala 2.12 (it's optional for Spark 2.4.x,
 mandatory in 3.0.x), then the relevant package is
-com.google.cloud.spark:spark-bigquery-with-dependencies_**2.12**:0.23.1. In
+com.google.cloud.spark:spark-bigquery-with-dependencies_**2.12**:0.23.2. In
 order to know which Scala version is used, please run the following code:
 
 **Python:**
@@ -961,14 +961,14 @@ To include the connector in your project:
 <dependency>
   <groupId>com.google.cloud.spark</groupId>
   <artifactId>spark-bigquery-with-dependencies_${scala.version}</artifactId>
-  <version>0.23.1</version>
+  <version>0.23.2</version>
 </dependency>
 ```
 
 ### SBT
 
 ```sbt
-libraryDependencies += "com.google.cloud.spark" %% "spark-bigquery-with-dependencies" % "0.23.1"
+libraryDependencies += "com.google.cloud.spark" %% "spark-bigquery-with-dependencies" % "0.23.2"
 ```
 
 ## FAQ

diff --git a/bigquery-connector-common/pom.xml b/bigquery-connector-common/pom.xml
@@ -5,7 +5,7 @@
   <parent>
     <groupId>com.google.cloud.spark</groupId>
     <artifactId>spark-bigquery-parent</artifactId>
-    <version>0.23.2-SNAPSHOT</version>
+    <version>0.23.2</version>
     <relativePath>../spark-bigquery-parent</relativePath>
   </parent>
 

diff --git a/coverage/pom.xml b/coverage/pom.xml
@@ -5,7 +5,7 @@
   <parent>
     <groupId>com.google.cloud.spark</groupId>
     <artifactId>spark-bigquery-parent</artifactId>
-    <version>0.23.2-SNAPSHOT</version>
+    <version>0.23.2</version>
     <relativePath>../spark-bigquery-parent</relativePath>
   </parent>
 

diff --git a/pom.xml b/pom.xml
@@ -5,7 +5,7 @@
   <parent>
     <groupId>com.google.cloud.spark</groupId>
     <artifactId>spark-bigquery-parent</artifactId>
-    <version>0.23.2-SNAPSHOT</version>
+    <version>0.23.2</version>
     <relativePath>spark-bigquery-parent</relativePath>
   </parent>
 

diff --git a/spark-bigquery-connector-common/pom.xml b/spark-bigquery-connector-common/pom.xml
@@ -4,7 +4,7 @@
     <parent>
         <groupId>com.google.cloud.spark</groupId>
         <artifactId>spark-bigquery-parent</artifactId>
-        <version>0.23.2-SNAPSHOT</version>
+        <version>0.23.2</version>
         <relativePath>../spark-bigquery-parent</relativePath>
     </parent>
 

diff --git a/spark-bigquery-dsv1/pom.xml b/spark-bigquery-dsv1/pom.xml
@@ -5,7 +5,7 @@
   <parent>
     <groupId>com.google.cloud.spark</groupId>
     <artifactId>spark-bigquery-parent</artifactId>
-    <version>0.23.2-SNAPSHOT</version>
+    <version>0.23.2</version>
     <relativePath>../spark-bigquery-parent</relativePath>
   </parent>
 

diff --git a/spark-bigquery-dsv1/spark-bigquery-dsv1-parent/pom.xml b/spark-bigquery-dsv1/spark-bigquery-dsv1-parent/pom.xml
@@ -5,7 +5,7 @@
   <parent>
     <groupId>com.google.cloud.spark</groupId>
     <artifactId>spark-bigquery-parent</artifactId>
-    <version>0.23.2-SNAPSHOT</version>
+    <version>0.23.2</version>
     <relativePath>../../spark-bigquery-parent</relativePath>
   </parent>
 
@@ -44,6 +44,10 @@
           <groupId>io.netty</groupId>
           <artifactId>netty</artifactId>
         </exclusion>
+        <exclusion>
+          <groupId>io.netty</groupId>
+          <artifactId>netty-all</artifactId>
+        </exclusion>
       </exclusions>
     </dependency>
     <dependency>
@@ -56,6 +60,10 @@
           <groupId>io.netty</groupId>
           <artifactId>netty</artifactId>
         </exclusion>
+        <exclusion>
+          <groupId>io.netty</groupId>
+          <artifactId>netty-all</artifactId>
+        </exclusion>
       </exclusions>
     </dependency>
     <dependency>

diff --git a/spark-bigquery-dsv1/spark-bigquery-dsv1-spark3-support/pom.xml b/spark-bigquery-dsv1/spark-bigquery-dsv1-spark3-support/pom.xml
@@ -5,7 +5,7 @@
   <parent>
     <groupId>com.google.cloud.spark</groupId>
     <artifactId>spark-bigquery-parent</artifactId>
-    <version>0.23.2-SNAPSHOT</version>
+    <version>0.23.2</version>
     <relativePath>../../spark-bigquery-parent</relativePath>
   </parent>
 

diff --git a/spark-bigquery-dsv1/spark-bigquery-with-dependencies-parent/pom.xml b/spark-bigquery-dsv1/spark-bigquery-with-dependencies-parent/pom.xml
@@ -5,7 +5,7 @@
   <parent>
     <groupId>com.google.cloud.spark</groupId>
     <artifactId>spark-bigquery-parent</artifactId>
-    <version>0.23.2-SNAPSHOT</version>
+    <version>0.23.2</version>
     <relativePath>../../spark-bigquery-parent</relativePath>
   </parent>
   <artifactId>spark-bigquery-with-dependencies-parent</artifactId>

diff --git a/spark-bigquery-dsv1/spark-bigquery-with-dependencies_2.11/pom.xml b/spark-bigquery-dsv1/spark-bigquery-with-dependencies_2.11/pom.xml
@@ -5,7 +5,7 @@
   <parent>
     <groupId>com.google.cloud.spark</groupId>
     <artifactId>spark-bigquery-with-dependencies-parent</artifactId>
-    <version>0.23.2-SNAPSHOT</version>
+    <version>0.23.2</version>
     <relativePath>../spark-bigquery-with-dependencies-parent</relativePath>
   </parent>
 

diff --git a/spark-bigquery-dsv1/spark-bigquery-with-dependencies_2.12/pom.xml b/spark-bigquery-dsv1/spark-bigquery-with-dependencies_2.12/pom.xml
@@ -5,7 +5,7 @@
   <parent>
     <groupId>com.google.cloud.spark</groupId>
     <artifactId>spark-bigquery-with-dependencies-parent</artifactId>
-    <version>0.23.2-SNAPSHOT</version>
+    <version>0.23.2</version>
     <relativePath>../spark-bigquery-with-dependencies-parent</relativePath>
   </parent>
 

diff --git a/spark-bigquery-dsv1/spark-bigquery_2.11/pom.xml b/spark-bigquery-dsv1/spark-bigquery_2.11/pom.xml
@@ -4,7 +4,7 @@
   <parent>
     <groupId>com.google.cloud.spark</groupId>
     <artifactId>spark-bigquery-dsv1-parent</artifactId>
-    <version>0.23.2-SNAPSHOT</version>
+    <version>0.23.2</version>
     <relativePath>../spark-bigquery-dsv1-parent</relativePath>
   </parent>
 

diff --git a/spark-bigquery-dsv1/spark-bigquery_2.12/pom.xml b/spark-bigquery-dsv1/spark-bigquery_2.12/pom.xml
@@ -4,7 +4,7 @@
   <parent>
     <groupId>com.google.cloud.spark</groupId>
     <artifactId>spark-bigquery-dsv1-parent</artifactId>
-    <version>0.23.2-SNAPSHOT</version>
+    <version>0.23.2</version>
     <relativePath>../spark-bigquery-dsv1-parent</relativePath>
   </parent>
 

diff --git a/spark-bigquery-dsv2/pom.xml b/spark-bigquery-dsv2/pom.xml
@@ -5,7 +5,7 @@
   <parent>
     <groupId>com.google.cloud.spark</groupId>
     <artifactId>spark-bigquery-parent</artifactId>
-    <version>0.23.2-SNAPSHOT</version>
+    <version>0.23.2</version>
     <relativePath>../spark-bigquery-parent</relativePath>
   </parent>