[SPARK-39927][BUILD] Upgrade to Avro 1.11.1 #37352

iemejia · 2022-07-31T20:26:17Z

What changes were proposed in this pull request?

Update the Avro version to 1.11.1

Why are the changes needed?

To stay up to date with upstream

Does this PR introduce any user-facing change?

No

How was this patch tested?

Unit tests

HyukjinKwon · 2022-08-01T00:40:33Z

Seems like the test failure looks related:

AvroV1Suite.support user provided avro schema for writing non-nullable enum type
java.lang.AssertionError: assertion failed: Exception tree doesn't contain the expected exception of type org.apache.avro.AvroTypeException with message: Not an enum: null
org.apache.avro.AvroTypeException: value null is not a SuitEnumType
	at org.apache.avro.generic.GenericDatumWriter.writeEnum(GenericDatumWriter.java:269)
	at org.apache.avro.specific.SpecificDatumWriter.writeEnum(SpecificDatumWriter.java:66)
	at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:148)
	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:95)
	at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
	at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:245)
	at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:117)
	at org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:184)
	at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:234)
	at org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:92)
	at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:145)
	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:95)
	at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
	at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:82)
	at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:314)
	at org.apache.spark.sql.avro.SparkAvroKeyRecordWriter.write(SparkAvroKeyOutputFormat.java:86)
	at org.apache.spark.sql.avro.SparkAvroKeyRecordWriter.write(SparkAvroKeyOutputFormat.java:63)
	at org.apache.spark.sql.avro.AvroOutputWriter.write(AvroOutputWriter.scala:86)
	at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.write(FileFormatDataWriter.scala:175)
	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithMetrics(FileFormatDataWriter.scala:85)
	at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:92)
	at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:323)
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1524)

xkrogen · 2022-08-01T16:07:45Z

Is this an official release? The documentation links you updated (such as this one) give a 404 error and I don't see Avro 1.11.1 listed on the Avro Releases page.

dongjoon-hyun

Thank you for making a PR, @iemejia .
Unfortunately, as mentioned by Hyukjin, for the failures, it looks like that.
- It seems that Avro error message is changed. Could you adjust the error message assertion part in Spark test case according to Avro change, please, @iemejia ?
To @xkrogen . Yes, right, but @iemejia published it. Usually, it takes some time to sync.
- https://repo1.maven.org/maven2/org/apache/avro/avro/1.11.1/

AmplabJenkins · 2022-08-02T02:22:12Z

Can one of the admins verify this patch?

dongjoon-hyun · 2022-08-03T21:09:35Z

Gentle ping, @iemejia .

martin-g · 2022-08-04T09:05:37Z

I don't see Avro 1.11.1 listed on the Avro Releases page.

Also, we work on replacing the Avro website with a new one. Probably it will be done later this week!

LuciferYang · 2022-08-04T16:08:37Z

seems support user provided non-nullable avro schema for nullable catalyst schema without any null record *** FAILED *** (181 milliseconds) still failed

dongjoon-hyun · 2022-08-04T16:54:45Z

Yes, it does. @LuciferYang .

To @iemejia , as I mentioned here, please take a look at the test case. Otherwise, Avro 1.11.1 cannot pass the CIs.

dongjoon-hyun · 2022-08-11T21:52:38Z

connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala

          .option("avroSchema", avroSchema).save(s"$tempDir/${UUID.randomUUID()}")
      }
-      assertExceptionMsg[AvroTypeException](e1, "Not an enum: null")
+      assertExceptionMsg[AvroTypeException](e1, "value null is not a SuitEnumType")


Thank you for the fix.

dongjoon-hyun · 2022-08-11T21:52:45Z

connector/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala

      assert(message.contains("Caused by: java.lang.NullPointerException: "))
-      assert(message.contains(
-        "null of string in string in field Name of test_schema in test_schema"))
+      assert(message.contains("null in string in field Name"))


dongjoon-hyun

+1, LGTM. I verified Avro module.

$ build/sbt "avro/test"
...
[info] Test run org.apache.spark.sql.avro.JavaAvroFunctionsSuite started
[info] Test org.apache.spark.sql.avro.JavaAvroFunctionsSuite.testToAvroFromAvro started
[info] Test run org.apache.spark.sql.avro.JavaAvroFunctionsSuite finished: 0 failed, 0 ignored, 1 total, 0.633s
[info] ScalaTest
[info] Run completed in 2 minutes, 42 seconds.
[info] Total number of tests run: 288
[info] Suites: completed 13, aborted 0
[info] Tests: succeeded 288, failed 0, canceled 0, ignored 2, pending 0
[info] All tests passed.
[info] Passed: Total 289, Failed 0, Errors 0, Passed 289, Ignored 2
[success] Total time: 530 s (08:50), completed Aug 11, 2022 3:04:35 PM

dongjoon-hyun · 2022-08-11T22:05:22Z

Merged to master for Apache Spark 3.4.0.

dongjoon-hyun · 2022-08-11T22:06:30Z

Thank you, @iemejia , @HyukjinKwon , @xkrogen , @martin-g , @LuciferYang .

iemejia · 2022-08-11T22:17:17Z

Thanks as usual @dongjoon-hyun !

### What changes were proposed in this pull request? Update the Avro version to 1.11.1 ### Why are the changes needed? To stay up to date with upstream ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit tests Closes apache#37352 from iemejia/SPARK-39927-avro-1.11.1. Authored-by: Ismaël Mejía <iemejia@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> # Conflicts: # dev/deps/spark-deps-hadoop-2.7-hive-2.3 # dev/deps/spark-deps-hadoop-3-hive-2.3 # docs/sql-data-sources-avro.md # external/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala # external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala # pom.xml # project/SparkBuild.scala # sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala

### What changes were proposed in this pull request? Update the Avro version to 1.11.1 ### Why are the changes needed? To stay up to date with upstream ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit tests Closes apache#37352 from iemejia/SPARK-39927-avro-1.11.1. Authored-by: Ismaël Mejía <iemejia@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> # Conflicts: # dev/deps/spark-deps-hadoop-2.7-hive-2.3 # dev/deps/spark-deps-hadoop-3-hive-2.3 # docs/sql-data-sources-avro.md # external/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala # external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala # pom.xml # project/SparkBuild.scala # sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala Co-authored-by: Ismaël Mejía <iemejia@gmail.com>

### What changes were proposed in this pull request? Update the Avro version to 1.11.1 ### Why are the changes needed? To stay up to date with upstream ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit tests Closes apache#37352 from iemejia/SPARK-39927-avro-1.11.1. Authored-by: Ismaël Mejía <iemejia@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> # Conflicts: # dev/deps/spark-deps-hadoop-2.7-hive-2.3 # dev/deps/spark-deps-hadoop-3-hive-2.3 # docs/sql-data-sources-avro.md # external/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala # external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala # pom.xml # project/SparkBuild.scala # sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala

### What changes were proposed in this pull request? Update the Avro version to 1.11.1 ### Why are the changes needed? To stay up to date with upstream ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit tests Closes apache#37352 from iemejia/SPARK-39927-avro-1.11.1. Authored-by: Ismaël Mejía <iemejia@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> # Conflicts: # dev/deps/spark-deps-hadoop-2.7-hive-2.3 # dev/deps/spark-deps-hadoop-3-hive-2.3 # docs/sql-data-sources-avro.md # external/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala # external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala # pom.xml # project/SparkBuild.scala # sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala Co-authored-by: Ismaël Mejía <iemejia@gmail.com>

…herrypick ### What changes were proposed in this pull request? Update the Avro version to 1.11.1 ### Why are the changes needed? To stay up to date with upstream ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit tests Closes apache#37352 from iemejia/SPARK-39927-avro-1.11.1. Authored-by: Ismaël Mejía <iemejia@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> # Conflicts: # dev/deps/spark-deps-hadoop-2.7-hive-2.3 # dev/deps/spark-deps-hadoop-3-hive-2.3 # docs/sql-data-sources-avro.md # external/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala # external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala # pom.xml # project/SparkBuild.scala # sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala Co-authored-by: Ismaël Mejía <iemejia@gmail.com>

### What changes were proposed in this pull request? Update the Avro version to 1.11.1 ### Why are the changes needed? To stay up to date with upstream ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit tests Closes apache#37352 from iemejia/SPARK-39927-avro-1.11.1. Authored-by: Ismaël Mejía <iemejia@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> # Conflicts: # dev/deps/spark-deps-hadoop-2.7-hive-2.3 # dev/deps/spark-deps-hadoop-3-hive-2.3 # docs/sql-data-sources-avro.md # external/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala # external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala # pom.xml # project/SparkBuild.scala # sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala Co-authored-by: Ismaël Mejía <iemejia@gmail.com>

github-actions bot added BUILD DOCS SQL labels Jul 31, 2022

dongjoon-hyun reviewed Aug 1, 2022

View reviewed changes

iemejia force-pushed the SPARK-39927-avro-1.11.1 branch from 6da9901 to e919d1b Compare August 4, 2022 10:07

iemejia force-pushed the SPARK-39927-avro-1.11.1 branch from e919d1b to 48ba166 Compare August 11, 2022 08:05

[SPARK-39927][BUILD] Upgrade to Avro 1.11.1

5b8783a

iemejia force-pushed the SPARK-39927-avro-1.11.1 branch from 48ba166 to 5b8783a Compare August 11, 2022 21:32

dongjoon-hyun reviewed Aug 11, 2022

View reviewed changes

dongjoon-hyun approved these changes Aug 11, 2022

View reviewed changes

dongjoon-hyun closed this in 4394e24 Aug 11, 2022

iemejia deleted the SPARK-39927-avro-1.11.1 branch August 11, 2022 22:17

leejaywei mentioned this pull request Mar 9, 2023

[SPARK-39927][BUILD] Upgrade to Avro 1.11.1 Kyligence/spark#599

Merged

leejaywei mentioned this pull request Mar 16, 2023

[SPARK-39927][BUILD] Upgrade to Avro 1.11.1 Kyligence/spark#604

Merged

[SPARK-39927][BUILD] Upgrade to Avro 1.11.1 #37352

[SPARK-39927][BUILD] Upgrade to Avro 1.11.1 #37352

Uh oh!

Conversation

iemejia commented Jul 31, 2022

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

HyukjinKwon commented Aug 1, 2022

Uh oh!

xkrogen commented Aug 1, 2022

Uh oh!

dongjoon-hyun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented Aug 2, 2022

Uh oh!

dongjoon-hyun commented Aug 3, 2022

Uh oh!

martin-g commented Aug 4, 2022

Uh oh!

LuciferYang commented Aug 4, 2022

Uh oh!

dongjoon-hyun commented Aug 4, 2022

Uh oh!

dongjoon-hyun Aug 11, 2022

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Aug 11, 2022

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun commented Aug 11, 2022

Uh oh!

dongjoon-hyun commented Aug 11, 2022

Uh oh!

iemejia commented Aug 11, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

dongjoon-hyun left a comment •

edited

Loading