Skip to content

Comments

Adding the PropertyNamingStrategies from jackson for fixing hadoop ingestion#14671

Merged
abhishekagarwal87 merged 1 commit intoapache:masterfrom
pranavbhole:jacksonBackwardComp
Aug 1, 2023
Merged

Adding the PropertyNamingStrategies from jackson for fixing hadoop ingestion#14671
abhishekagarwal87 merged 1 commit intoapache:masterfrom
pranavbhole:jacksonBackwardComp

Conversation

@pranavbhole
Copy link
Contributor

…gestion

Fixes #XXXX.

Description

We found that aws jdk core is using the jackson version 2.12.7 and it is not compatible with older jackson esp inside hadoop container as it loads the classes from given extension. We found that s3 extension is loading S3 required jars and instantiating the PropertyNamingStrategies from newer jackson and failing.
Creating the PR to pull this class into druid repo itself to fix the hadoop ingestion and make it compatible with newer jackson.

Fixed the bug ...

Renamed the class ...

Added a forbidden-apis entry ...

Release note


Key changed/added classes in this PR
  • MyFoo
  • OurBar
  • TheirBaz

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@abhishekagarwal87
Copy link
Contributor

Just to add a bit more colour here.

Hadoop has support for classloader isolation that follows a child-first class loading mechanism. There are jackson classes present on Hadoop server and there are jackson classes present in the libs that druid uploads. The aws sdk that is part of druid lib, wants to load PropertyNamingStrategies but it is not available in the child classloader so that class gets loaded through the parent classloader. The parent classloader would also load the PropertyNamingStrategy referenced by the PropertyNamingStrategies. All is well so far. But then the child classloader needs to load PropertyNamingStrategy later which it is able to load. But because it has also been loaded through the parent classloader, we get a linkage error. Somewhat explained here - http://frankkieviet.blogspot.com/2009/03/javalanglinkageerror-loader-constraint.html

@pranavbhole pranavbhole force-pushed the jacksonBackwardComp branch from 893c60e to a86b692 Compare July 27, 2023 22:56
@abhishekagarwal87
Copy link
Contributor

@pranavbhole - code inspection jobs are failing.

@pranavbhole pranavbhole force-pushed the jacksonBackwardComp branch 4 times, most recently from 6ac3d23 to 922388a Compare August 1, 2023 01:20
@pranavbhole pranavbhole force-pushed the jacksonBackwardComp branch from 922388a to cf2d63f Compare August 1, 2023 04:04
@pranavbhole
Copy link
Contributor Author

@abhishekagarwal87 pushed the fix.

@abhishekagarwal87 abhishekagarwal87 merged commit 8a10b46 into apache:master Aug 1, 2023
@LakshSingla LakshSingla added this to the 28.0 milestone Oct 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants