Skip to content

[SPARK-37206][BUILD][FOLLOWUP] Update avro to 1.11.0 in SparkBuild.scala and docs#36019

Closed
dongjoon-hyun wants to merge 3 commits intoapache:masterfrom
dongjoon-hyun:SPARK-37206
Closed

[SPARK-37206][BUILD][FOLLOWUP] Update avro to 1.11.0 in SparkBuild.scala and docs#36019
dongjoon-hyun wants to merge 3 commits intoapache:masterfrom
dongjoon-hyun:SPARK-37206

Conversation

@dongjoon-hyun
Copy link
Copy Markdown
Member

@dongjoon-hyun dongjoon-hyun commented Mar 31, 2022 •
edited
Loading

What changes were proposed in this pull request?

This is a follow-up of #34482 to update the Avro version consistently in both Maven and SBT.

In addition, this also reviews and updates the doc links and adds a note for the future.

Why are the changes needed?

Due to the mismatch, there occur some compilation failures in some systems.

$ build/mvn dependency:tree -pl core | grep avro
[INFO] +- org.apache.avro:avro:jar:1.11.0:compile
[INFO] +- org.apache.avro:avro-mapred:jar:1.11.0:compile
[INFO] |  \- org.apache.avro:avro-ipc:jar:1.11.0:compile

$ build/sbt "core/dependencyTree" | grep avro
[info]   +-org.apache.avro:avro-mapred:1.11.0
[info]   | +-org.apache.avro:avro-ipc:1.11.0
[info]   | | +-org.apache.avro:avro:1.10.2
[info]   +-org.apache.avro:avro:1.10.2

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manually verified.

@github-actions github-actions bot added the BUILD label Mar 31, 2022
@dongjoon-hyun
Copy link
Copy Markdown
Member Author

dongjoon-hyun commented Mar 31, 2022

cc @sarutak and @HyukjinKwon .

@sarutak
Copy link
Copy Markdown
Member

sarutak commented Mar 31, 2022

@dongjoon-hyun Nice catch! It LGTM, pending CIs.

@dongjoon-hyun
Copy link
Copy Markdown
Member Author

dongjoon-hyun commented Mar 31, 2022

Thank you, @sarutak .

@sarutak
Copy link
Copy Markdown
Member

sarutak commented Mar 31, 2022 •
edited
Loading

@dongjoon-hyun I found extra 2 occurrence of 1.10.2.

spark/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala

Line 80 in 26e93f9

* See https://avro.apache.org/docs/1.10.2/spec.html#schema_record .

spark/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala

Line 87 in 26e93f9

* See Avro spec for details: https://avro.apache.org/docs/1.10.2/spec.html#schema_record .

Should we fix them in this PR too?

@dongjoon-hyun
Copy link
Copy Markdown
Member Author

dongjoon-hyun commented Mar 31, 2022

Oh. Thank you! I'll include them too

@dongjoon-hyun
Copy link
Copy Markdown
Member Author

dongjoon-hyun commented Mar 31, 2022

It seems that the documentation is a little complicated. I found more.

$ git grep avro.apache.org
connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala:   * See https://avro.apache.org/docs/1.10.2/spec.html#schema_record .
connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala:   * See Avro spec for details: https://avro.apache.org/docs/1.10.2/spec.html#schema_record .
docs/sql-data-sources-avro.md:Currently Spark supports reading all [primitive types](https://avro.apache.org/docs/1.10.2/spec.html#schema_primitive) and [complex types](https://avro.apache.org/docs/1.10.2/spec.html#schema_complex) under records of Avro.
docs/sql-data-sources-avro.md:It also supports reading the following Avro [logical types](https://avro.apache.org/docs/1.10.2/spec.html#Logical+Types):
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala:    // https://avro.apache.org/docs/1.8.1/spec.html#Logical+Types

@dongjoon-hyun
Copy link
Copy Markdown
Member Author

dongjoon-hyun commented Mar 31, 2022

If the links are still valid in 1.11.0, I'll update all of them.

@sarutak
Copy link
Copy Markdown
Member

sarutak commented Mar 31, 2022

Oh, I didn't notice them in docs. Thank you!

@sarutak
Copy link
Copy Markdown
Member

sarutak commented Mar 31, 2022

BTW, it might be better to note that we need to modify other occurrences of the version when we upgrade Avro.

@dongjoon-hyun
Copy link
Copy Markdown
Member Author

dongjoon-hyun commented Mar 31, 2022

Update docs and added note.

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-37206][BUILD][FOLLOWUP] Update avro to 1.11.0 in SparkBuild.scala [SPARK-37206][BUILD][FOLLOWUP] Update avro to 1.11.0 in SparkBuild.scala and docs Mar 31, 2022
@HyukjinKwon
Copy link
Copy Markdown
Member

HyukjinKwon commented Mar 31, 2022

cc @gengliangwang too

gengliangwang
gengliangwang approved these changes Mar 31, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@gengliangwang gengliangwang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dongjoon-hyun
Copy link
Copy Markdown
Member Author

dongjoon-hyun commented Mar 31, 2022

Thank you, @HyukjinKwon , @gengliangwang , @sarutak (again)

@wangyum wangyum closed this in d790f3e Mar 31, 2022
wangyum pushed a commit that referenced this pull request Mar 31, 2022
@dongjoon-hyun @wangyum
[SPARK-37206][BUILD][FOLLOWUP] Update avro to 1.11.0 in `SparkBuild.s…
1ab7333 
…cala` and docs

### What changes were proposed in this pull request?

This is a follow-up of #34482 to update the Avro version consistently in both Maven and SBT.

In addition, this also reviews and updates the doc links and adds a note for the future.

### Why are the changes needed?

Due to the mismatch, there occur some compilation failures in some systems.
```
$ build/mvn dependency:tree -pl core | grep avro
[INFO] +- org.apache.avro:avro:jar:1.11.0:compile
[INFO] +- org.apache.avro:avro-mapred:jar:1.11.0:compile
[INFO] |  \- org.apache.avro:avro-ipc:jar:1.11.0:compile
```

```
$ build/sbt "core/dependencyTree" | grep avro
[info]   +-org.apache.avro:avro-mapred:1.11.0
[info]   | +-org.apache.avro:avro-ipc:1.11.0
[info]   | | +-org.apache.avro:avro:1.10.2
[info]   +-org.apache.avro:avro:1.10.2
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Manually verified.

Closes #36019 from dongjoon-hyun/SPARK-37206.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Yuming Wang <yumwang@ebay.com>
(cherry picked from commit d790f3e)
Signed-off-by: Yuming Wang <yumwang@ebay.com>
@wangyum
Copy link
Copy Markdown
Member

wangyum commented Mar 31, 2022

Merged to master and branch-3.3

@dongjoon-hyun
Copy link
Copy Markdown
Member Author

dongjoon-hyun commented Mar 31, 2022

Thank you, @wangyum !

@dongjoon-hyun dongjoon-hyun deleted the SPARK-37206 branch March 31, 2022 07:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gengliangwang gengliangwang gengliangwang approved these changes

@sarutak sarutak sarutak approved these changes

@wangyum wangyum wangyum approved these changes

@HyukjinKwon HyukjinKwon HyukjinKwon approved these changes

Assignees

No one assigned

Labels

BUILD DOCS SQL

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@dongjoon-hyun @sarutak @HyukjinKwon @wangyum @gengliangwang