Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HADOOP-18160 Avoid shading wildfly.openssl runtime dependency #4074

Conversation

andreAmorimF
Copy link
Contributor

Description of PR

org.wildfly.openssl is a runtime library and its references are being shaded on Hadoop, breaking the integration with other frameworks like Spark, whenever the "fs.s3a.ssl.channel.mode" is set to "openssl". The error produced in this situation is:

Suppressed: java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/org/wildfly/openssl/OpenSSLProvider

This PR aims to exclude org.wildfly.openssl from the shade, keeping the original namespace to be referenced whenever this runtime dependency is added to the classpath.

How was this patch tested?

Used the patched jar on a Spark 3.2.1 application which now correctly references the org.wildfly.openssl dependency added to the classpath.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

@steveloughran
Copy link
Contributor

thanks.

will have to raise this with on the hadoop common list to discuss what to do here.
i don't see any uses of the library other than hadoop azure and hadoop aws, so there isn't much else to worry about

@steveloughran
Copy link
Contributor

raised on the list...if nobody objects i will merge (remind me if I don't)

@steveloughran steveloughran self-requested a review March 22, 2022 17:12
@apache apache deleted a comment from hadoop-yetus Mar 22, 2022
Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1. nobody said no, and we do the same for other optional classes.

@steveloughran steveloughran merged commit 59d07bd into apache:trunk Mar 22, 2022
asfgit pushed a commit that referenced this pull request Mar 22, 2022
Contributed by André Fonseca.

Change-Id: I198c66b58c0e05c1752ad9d6a50fe2f2932f61d5
@steveloughran
Copy link
Contributor

ok, merged in to branch-3.3 and trunk. thank you!

@apache apache deleted a comment from hadoop-yetus Mar 22, 2022
@apache apache deleted a comment from hadoop-yetus Mar 22, 2022
@andreAmorimF andreAmorimF deleted the HADOOP-18160/dont-shade-wildfly-openssl branch March 22, 2022 21:46
asfgit pushed a commit that referenced this pull request Apr 18, 2022
Contributed by André Fonseca.

Change-Id: I198c66b58c0e05c1752ad9d6a50fe2f2932f61d5
HarshitGupta11 pushed a commit to HarshitGupta11/hadoop that referenced this pull request Nov 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants