Skip to content

HDDS-7440. Delete package-info class for org.apache.hadoop.io.erasurecode.rawcoder to avoid conflict with Hadoop#3925

Merged
adoroszlai merged 1 commit intoapache:masterfrom
adoroszlai:HDDS-7440
Nov 9, 2022
Merged

HDDS-7440. Delete package-info class for org.apache.hadoop.io.erasurecode.rawcoder to avoid conflict with Hadoop#3925
adoroszlai merged 1 commit intoapache:masterfrom
adoroszlai:HDDS-7440

Conversation

@adoroszlai
Copy link
Contributor

What changes were proposed in this pull request?

Trying to use Ozone FS jar in Spark points out conflict with one of Hadoop's packages:

scala> :require /opt/ozone/share/ozone/lib/ozone-filesystem-hadoop3-1.3.0-SNAPSHOT.jar
The path '/opt/ozone/share/ozone/lib/ozone-filesystem-hadoop3-1.3.0-SNAPSHOT.jar' cannot be loaded, it contains a classfile that already exists on the classpath: org.apache.hadoop.io.erasurecode.rawcoder.package-info

https://issues.apache.org/jira/browse/HDDS-7440

How was this patch tested?

Regular CI:
https://github.com/adoroszlai/hadoop-ozone/actions/runs/3368073737

After the update Spark still complains, but only about non-shaded third-party classes. Those will be addressed in separate task.

@adoroszlai adoroszlai self-assigned this Nov 1, 2022

<suppressions>
<suppress checks="JavadocPackage" files="[\\/]src[\\/]test[\\/].*"/>
<suppress checks="JavadocPackage" files="src/main/java/org/apache/hadoop/io/erasurecode/rawcoder"/>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW what about dropping JavadocPackage rule? I believe it's adopted from Hadoop code style, but it doesn't seem to make much sense.

Copy link
Contributor

@myskov myskov Nov 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropping the rule will allow to remove useless (in most cases) package-info files.

@kerneltime
Copy link
Contributor

@adoroszlai would shading be a better approach as we might not want EC package to be picked by the application but be based on the actual testing done in the released version?

@adoroszlai
Copy link
Contributor Author

would shading be a better approach?

I don't think shading helps here. We need this Hadoop-specific package name to access some protected members in some Hadoop classes in the same package. The only real code in this package is the utility to access the Hadoop code. If we shade the Ozone's classes in this package then I don't think it will be able to access Hadoop classes. If we shade both Hadoop and Ozone, we're back to duplicate package-info class.

The other possible solution is to copy all related code from Hadoop, but I think the goal in this case was to avoid that.

@adoroszlai adoroszlai merged commit 0af369e into apache:master Nov 9, 2022
@adoroszlai adoroszlai deleted the HDDS-7440 branch November 9, 2022 16:10
@adoroszlai
Copy link
Contributor Author

Thanks @kerneltime, @myskov, @sodonnel for the review.

captainzmc pushed a commit to captainzmc/hadoop-ozone that referenced this pull request Nov 10, 2022
captainzmc pushed a commit that referenced this pull request Nov 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants