Avro 1.9 upgrade compatible change - upgraded jackson mapper to 2.x and kept jackson mapper 1.x for modules with hive, helix library dependency. #3368
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ith federated Yarn cluster enabled (apache#3345) * [GOBBLIN-1500]Support gobblin on yarn to be able to run on clusters with robin enabled (WIP) * address comments * address comments * Enable add dynamic property feature in azkabanGobblinYarnAppLauncher
apache#3347) * Add more comments in IcebergMetadataWriter while reading thru the code base * Fix unit tests post force the compile-time casting of gmce
…artition fails … (apache#3344) * dont add a partition if it already exists but the getPartition fails with TException, dont alter a partition if a partition does not exist but the add_partition fails with TException * add a unit test to verify hive client behavior
…nd kept jackson mapper 1.x for modules with hive, helix library dependency.
Contributor
|
Merging as there's no actual code change and CI passed. Thanks @abhisheknath2011 |
11 tasks
abhisheknath2011
added a commit
to abhisheknath2011/gobblin
that referenced
this pull request
Mar 28, 2022
…nd kept jackson mapper 1.x for modules with hive, helix library dependency. (apache#3368) * Avro 1.9 upgrade compatible change - upgraded jackson mapper to 2.x and kept jackson mapper 1.x for modules with hive, helix library dependency.
abhisheknath2011
added a commit
to abhisheknath2011/gobblin
that referenced
this pull request
Mar 28, 2022
…nd kept jackson mapper 1.x for modules with hive, helix library dependency. (apache#3368) * Avro 1.9 upgrade compatible change - upgraded jackson mapper to 2.x and kept jackson mapper 1.x for modules with hive, helix library dependency.
Will-Lo
pushed a commit
that referenced
this pull request
Oct 18, 2022
* [Branch avro_1_9] Avro 1.9 upgrade compatible change - replaced deprecated public APIs with the compatible APIs (#3349) * Avro 1.9 upgrade compatible change - replaced deprecated public APIs with the compatible APIs. * Avro 1.9 upgrade compatible change - Replaced guava library import from avro shaded with direct guava libraries * Applied Gobblin codestyle formatting. Co-authored-by: Lei <autumnust@users.noreply.github.com> * Avro 1.9 upgrade compatible change - upgraded jackson mapper to 2.x and kept jackson mapper 1.x for modules with hive, helix library dependency. (#3368) * Avro 1.9 upgrade compatible change - upgraded jackson mapper to 2.x and kept jackson mapper 1.x for modules with hive, helix library dependency. * Changes for upgrade Avro 1.9.2 and leverges hive with avro changes from https://linkedin.jfrog.io/artifactory/gobblin-hive (#3458) * Use helper-all v0.2.74 to solve issues around default values. (#3469) The latest version of helper-all fixes the issues seen before w.r.t. default values, so we can now revert the code and the *.avsc files back to how they used to be, with two minor exceptions: 1. Check Schema equality using their .toString() representations. Doing it the old way works for two out of the three instances, but one of them fails, for reasons I haven't figured out yet. 2. Add a `"default":null` piece to recursive_schema_1_converted.avsc. This is harmless, and is caused by the fact that the compatibility helper always adds it if it's a valid default for the schema. See the comments for FieldBuilder19.setDefault(): https://github.com/linkedin/avro-util/blob/b9e89c55980ea8e5fd3c8d8da362d7195dd2a99c/helper/impls/helper-impl-19/src/main/java/com/linkedin/avroutil1/compatibility/avro19/FieldBuilder19.java#L69 To verify that the files are otherwise the same as before: ``` $ for file in gobblin-core-base/src/test/resources/converter/*.avsc; do > git show 928e018:$file > /tmp/before > diff <(jq . </tmp/before) <(jq . <$file) > done ``` * [Branch avro_1_9] Avro 1.9 upgrade compatible change - replaced deprecated public APIs with the compatible APIs (#3349) * Avro 1.9 upgrade compatible change - replaced deprecated public APIs with the compatible APIs. * Avro 1.9 upgrade compatible change - Replaced guava library import from avro shaded with direct guava libraries * Applied Gobblin codestyle formatting. Co-authored-by: Lei <autumnust@users.noreply.github.com> * Avro 1.9 upgrade compatible change - upgraded jackson mapper to 2.x and kept jackson mapper 1.x for modules with hive, helix library dependency. (#3368) * Avro 1.9 upgrade compatible change - upgraded jackson mapper to 2.x and kept jackson mapper 1.x for modules with hive, helix library dependency. * Changes for upgrade Avro 1.9.2 and leverges hive with avro changes from https://linkedin.jfrog.io/artifactory/gobblin-hive (#3458) * Use helper-all v0.2.74 to solve issues around default values. (#3469) The latest version of helper-all fixes the issues seen before w.r.t. default values, so we can now revert the code and the *.avsc files back to how they used to be, with two minor exceptions: 1. Check Schema equality using their .toString() representations. Doing it the old way works for two out of the three instances, but one of them fails, for reasons I haven't figured out yet. 2. Add a `"default":null` piece to recursive_schema_1_converted.avsc. This is harmless, and is caused by the fact that the compatibility helper always adds it if it's a valid default for the schema. See the comments for FieldBuilder19.setDefault(): https://github.com/linkedin/avro-util/blob/b9e89c55980ea8e5fd3c8d8da362d7195dd2a99c/helper/impls/helper-impl-19/src/main/java/com/linkedin/avroutil1/compatibility/avro19/FieldBuilder19.java#L69 To verify that the files are otherwise the same as before: ``` $ for file in gobblin-core-base/src/test/resources/converter/*.avsc; do > git show 928e018:$file > /tmp/before > diff <(jq . </tmp/before) <(jq . <$file) > done ``` * [Branch avro_1_9] Avro 1.9 upgrade compatible change - replaced deprecated public APIs with the compatible APIs (#3349) * Avro 1.9 upgrade compatible change - replaced deprecated public APIs with the compatible APIs. * Avro 1.9 upgrade compatible change - Replaced guava library import from avro shaded with direct guava libraries * Applied Gobblin codestyle formatting. Co-authored-by: Lei <autumnust@users.noreply.github.com> * Avro 1.9 upgrade compatible change - upgraded jackson mapper to 2.x and kept jackson mapper 1.x for modules with hive, helix library dependency. (#3368) * Avro 1.9 upgrade compatible change - upgraded jackson mapper to 2.x and kept jackson mapper 1.x for modules with hive, helix library dependency. * Changes for upgrade Avro 1.9.2 and leverges hive with avro changes from https://linkedin.jfrog.io/artifactory/gobblin-hive (#3458) * Use helper-all v0.2.74 to solve issues around default values. (#3469) The latest version of helper-all fixes the issues seen before w.r.t. default values, so we can now revert the code and the *.avsc files back to how they used to be, with two minor exceptions: 1. Check Schema equality using their .toString() representations. Doing it the old way works for two out of the three instances, but one of them fails, for reasons I haven't figured out yet. 2. Add a `"default":null` piece to recursive_schema_1_converted.avsc. This is harmless, and is caused by the fact that the compatibility helper always adds it if it's a valid default for the schema. See the comments for FieldBuilder19.setDefault(): https://github.com/linkedin/avro-util/blob/b9e89c55980ea8e5fd3c8d8da362d7195dd2a99c/helper/impls/helper-impl-19/src/main/java/com/linkedin/avroutil1/compatibility/avro19/FieldBuilder19.java#L69 To verify that the files are otherwise the same as before: ``` $ for file in gobblin-core-base/src/test/resources/converter/*.avsc; do > git show 928e018:$file > /tmp/before > diff <(jq . </tmp/before) <(jq . <$file) > done ``` * Merging apache/gobblin master with avro_1_9 * Use helper-all v0.2.74 to solve issues around default values. (#3469) The latest version of helper-all fixes the issues seen before w.r.t. default values, so we can now revert the code and the *.avsc files back to how they used to be, with two minor exceptions: 1. Check Schema equality using their .toString() representations. Doing it the old way works for two out of the three instances, but one of them fails, for reasons I haven't figured out yet. 2. Add a `"default":null` piece to recursive_schema_1_converted.avsc. This is harmless, and is caused by the fact that the compatibility helper always adds it if it's a valid default for the schema. See the comments for FieldBuilder19.setDefault(): https://github.com/linkedin/avro-util/blob/b9e89c55980ea8e5fd3c8d8da362d7195dd2a99c/helper/impls/helper-impl-19/src/main/java/com/linkedin/avroutil1/compatibility/avro19/FieldBuilder19.java#L69 To verify that the files are otherwise the same as before: ``` $ for file in gobblin-core-base/src/test/resources/converter/*.avsc; do > git show 928e018:$file > /tmp/before > diff <(jq . </tmp/before) <(jq . <$file) > done ``` * Added deprecated json method using AvroCompatibilityHelper * Removed unused import and replaced Integer.valueOf with Integer.parseInt * Exclude com.linkedin.hive dependency from gradle build files similar to org.apache.hive * Repalce direct avro field creation with AvroCompatibilityHelper.createSchemaField * Removed extra dependency. Addressed review comment - removed jcenter() repository * Upgrade AvroCompatHelper version * Removed the code that are actually moved to AvroHiveTypeUtils.java in the master branch * Addresssed review comments: replaced getObjectProps/getObjectProp with AvroCompatibilityHelper methods * Fix for test failure Co-authored-by: Lei <autumnust@users.noreply.github.com> Co-authored-by: Sreeram Ramachandran <sramachandran@linkedin.com>
phet
pushed a commit
to phet/gobblin
that referenced
this pull request
Dec 3, 2022
* [Branch avro_1_9] Avro 1.9 upgrade compatible change - replaced deprecated public APIs with the compatible APIs (apache#3349) * Avro 1.9 upgrade compatible change - replaced deprecated public APIs with the compatible APIs. * Avro 1.9 upgrade compatible change - Replaced guava library import from avro shaded with direct guava libraries * Applied Gobblin codestyle formatting. Co-authored-by: Lei <autumnust@users.noreply.github.com> * Avro 1.9 upgrade compatible change - upgraded jackson mapper to 2.x and kept jackson mapper 1.x for modules with hive, helix library dependency. (apache#3368) * Avro 1.9 upgrade compatible change - upgraded jackson mapper to 2.x and kept jackson mapper 1.x for modules with hive, helix library dependency. * Changes for upgrade Avro 1.9.2 and leverges hive with avro changes from https://linkedin.jfrog.io/artifactory/gobblin-hive (apache#3458) * Use helper-all v0.2.74 to solve issues around default values. (apache#3469) The latest version of helper-all fixes the issues seen before w.r.t. default values, so we can now revert the code and the *.avsc files back to how they used to be, with two minor exceptions: 1. Check Schema equality using their .toString() representations. Doing it the old way works for two out of the three instances, but one of them fails, for reasons I haven't figured out yet. 2. Add a `"default":null` piece to recursive_schema_1_converted.avsc. This is harmless, and is caused by the fact that the compatibility helper always adds it if it's a valid default for the schema. See the comments for FieldBuilder19.setDefault(): https://github.com/linkedin/avro-util/blob/b9e89c55980ea8e5fd3c8d8da362d7195dd2a99c/helper/impls/helper-impl-19/src/main/java/com/linkedin/avroutil1/compatibility/avro19/FieldBuilder19.java#L69 To verify that the files are otherwise the same as before: ``` $ for file in gobblin-core-base/src/test/resources/converter/*.avsc; do > git show 928e018:$file > /tmp/before > diff <(jq . </tmp/before) <(jq . <$file) > done ``` * [Branch avro_1_9] Avro 1.9 upgrade compatible change - replaced deprecated public APIs with the compatible APIs (apache#3349) * Avro 1.9 upgrade compatible change - replaced deprecated public APIs with the compatible APIs. * Avro 1.9 upgrade compatible change - Replaced guava library import from avro shaded with direct guava libraries * Applied Gobblin codestyle formatting. Co-authored-by: Lei <autumnust@users.noreply.github.com> * Avro 1.9 upgrade compatible change - upgraded jackson mapper to 2.x and kept jackson mapper 1.x for modules with hive, helix library dependency. (apache#3368) * Avro 1.9 upgrade compatible change - upgraded jackson mapper to 2.x and kept jackson mapper 1.x for modules with hive, helix library dependency. * Changes for upgrade Avro 1.9.2 and leverges hive with avro changes from https://linkedin.jfrog.io/artifactory/gobblin-hive (apache#3458) * Use helper-all v0.2.74 to solve issues around default values. (apache#3469) The latest version of helper-all fixes the issues seen before w.r.t. default values, so we can now revert the code and the *.avsc files back to how they used to be, with two minor exceptions: 1. Check Schema equality using their .toString() representations. Doing it the old way works for two out of the three instances, but one of them fails, for reasons I haven't figured out yet. 2. Add a `"default":null` piece to recursive_schema_1_converted.avsc. This is harmless, and is caused by the fact that the compatibility helper always adds it if it's a valid default for the schema. See the comments for FieldBuilder19.setDefault(): https://github.com/linkedin/avro-util/blob/b9e89c55980ea8e5fd3c8d8da362d7195dd2a99c/helper/impls/helper-impl-19/src/main/java/com/linkedin/avroutil1/compatibility/avro19/FieldBuilder19.java#L69 To verify that the files are otherwise the same as before: ``` $ for file in gobblin-core-base/src/test/resources/converter/*.avsc; do > git show 928e018:$file > /tmp/before > diff <(jq . </tmp/before) <(jq . <$file) > done ``` * [Branch avro_1_9] Avro 1.9 upgrade compatible change - replaced deprecated public APIs with the compatible APIs (apache#3349) * Avro 1.9 upgrade compatible change - replaced deprecated public APIs with the compatible APIs. * Avro 1.9 upgrade compatible change - Replaced guava library import from avro shaded with direct guava libraries * Applied Gobblin codestyle formatting. Co-authored-by: Lei <autumnust@users.noreply.github.com> * Avro 1.9 upgrade compatible change - upgraded jackson mapper to 2.x and kept jackson mapper 1.x for modules with hive, helix library dependency. (apache#3368) * Avro 1.9 upgrade compatible change - upgraded jackson mapper to 2.x and kept jackson mapper 1.x for modules with hive, helix library dependency. * Changes for upgrade Avro 1.9.2 and leverges hive with avro changes from https://linkedin.jfrog.io/artifactory/gobblin-hive (apache#3458) * Use helper-all v0.2.74 to solve issues around default values. (apache#3469) The latest version of helper-all fixes the issues seen before w.r.t. default values, so we can now revert the code and the *.avsc files back to how they used to be, with two minor exceptions: 1. Check Schema equality using their .toString() representations. Doing it the old way works for two out of the three instances, but one of them fails, for reasons I haven't figured out yet. 2. Add a `"default":null` piece to recursive_schema_1_converted.avsc. This is harmless, and is caused by the fact that the compatibility helper always adds it if it's a valid default for the schema. See the comments for FieldBuilder19.setDefault(): https://github.com/linkedin/avro-util/blob/b9e89c55980ea8e5fd3c8d8da362d7195dd2a99c/helper/impls/helper-impl-19/src/main/java/com/linkedin/avroutil1/compatibility/avro19/FieldBuilder19.java#L69 To verify that the files are otherwise the same as before: ``` $ for file in gobblin-core-base/src/test/resources/converter/*.avsc; do > git show 928e018:$file > /tmp/before > diff <(jq . </tmp/before) <(jq . <$file) > done ``` * Merging apache/gobblin master with avro_1_9 * Use helper-all v0.2.74 to solve issues around default values. (apache#3469) The latest version of helper-all fixes the issues seen before w.r.t. default values, so we can now revert the code and the *.avsc files back to how they used to be, with two minor exceptions: 1. Check Schema equality using their .toString() representations. Doing it the old way works for two out of the three instances, but one of them fails, for reasons I haven't figured out yet. 2. Add a `"default":null` piece to recursive_schema_1_converted.avsc. This is harmless, and is caused by the fact that the compatibility helper always adds it if it's a valid default for the schema. See the comments for FieldBuilder19.setDefault(): https://github.com/linkedin/avro-util/blob/b9e89c55980ea8e5fd3c8d8da362d7195dd2a99c/helper/impls/helper-impl-19/src/main/java/com/linkedin/avroutil1/compatibility/avro19/FieldBuilder19.java#L69 To verify that the files are otherwise the same as before: ``` $ for file in gobblin-core-base/src/test/resources/converter/*.avsc; do > git show 928e018:$file > /tmp/before > diff <(jq . </tmp/before) <(jq . <$file) > done ``` * Added deprecated json method using AvroCompatibilityHelper * Removed unused import and replaced Integer.valueOf with Integer.parseInt * Exclude com.linkedin.hive dependency from gradle build files similar to org.apache.hive * Repalce direct avro field creation with AvroCompatibilityHelper.createSchemaField * Removed extra dependency. Addressed review comment - removed jcenter() repository * Upgrade AvroCompatHelper version * Removed the code that are actually moved to AvroHiveTypeUtils.java in the master branch * Addresssed review comments: replaced getObjectProps/getObjectProp with AvroCompatibilityHelper methods * Fix for test failure Co-authored-by: Lei <autumnust@users.noreply.github.com> Co-authored-by: Sreeram Ramachandran <sramachandran@linkedin.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA
Description
Tests
Commits