Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade Avro to 1.9.1 #5938

Merged
merged 11 commits into from Jan 6, 2020
Merged

Upgrade Avro to 1.9.1 #5938

merged 11 commits into from Jan 6, 2020

Conversation

@massakam
Copy link
Contributor

massakam commented Dec 25, 2019

Motivation

Currently, Pulsar uses Avro 1.8.2, a version released two years ago. The latest version of Avro is 1.9.1, which uses FasterXML's Jackson 2.x instead of Codehaus's Jackson 1.x. Jackson is prone to security issues, so we should not keep using older versions.
https://blog.godatadriven.com/apache-avro-1-9-release

Modifications

Avro 1.9 has some major changes:

  • The library used to handle logical datetime values has changed from Joda-Time to JSR-310 (apache/avro#631)
  • Namespaces no longer include "$" when generating schemas containing inner classes using ReflectData (apache/avro#283)
  • Validation of default values has been enabled (apache/avro#288). This results in a validation error when parsing the following schema:
{
  "name": "fieldName",
  "type": [
    "null",
    "string"
  ],
  "default": "defaultValue"
}

The default value of a nullable field must be null (cf. https://issues.apache.org/jira/browse/AVRO-1803), and the default value of the field as above is actually null. However, this PR disables the validation in order to maintain the traditional behavior.


try {
// Disable validation of default values for compatibility
validateDefaults.set(false);

This comment has been minimized.

Copy link
@massakam

massakam Dec 25, 2019

Author Contributor

I think this is not a very good way. However, there seems to be no public method to disable validation. Another option is to enable validation of default values.

@massakam

This comment has been minimized.

Copy link
Contributor Author

massakam commented Dec 26, 2019

retest this please

@massakam massakam added this to the 2.6.0 milestone Dec 26, 2019
@massakam

This comment has been minimized.

Copy link
Contributor Author

massakam commented Dec 26, 2019

rerun cpp tests

@sijie
sijie approved these changes Jan 6, 2020
@sijie sijie merged commit d6f240e into apache:master Jan 6, 2020
10 of 18 checks passed
10 of 18 checks passed
cpp-tests cpp-tests
Details
backwards-compatibility backwards-compatibility
Details
process process
Details
sql sql
Details
thread thread
Details
License check License check
Details
unit-test-flaky unit-test-flaky
Details
unit-tests unit-tests
Details
cli
Details
function-state
Details
messaging
Details
schema
Details
standalone
Details
tiered-filesystem
Details
tiered-jcloud
Details
Jenkins: C++ / Python Tests SUCCESS
Details
Jenkins: Integration Tests SUCCESS
Details
Jenkins: Java 8 - Unit Tests SUCCESS
Details
@massakam massakam deleted the massakam:upgrade-avro branch Jan 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.