Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HADOOP-18576. Lack of a JavaDoc comment before annotation causes Java 11 JavaDoc failure #5344

Merged
merged 1 commit into from
Feb 6, 2023

Conversation

snmvaughan
Copy link
Contributor

Description of PR

Add JavaDoc comments to package-info.java to avoid errors resulting from the use of Hadoop annotations.

How was this patch tested?

Running JavaDoc using Java 11 in an Hadoop development environment docker image.

mvn -Ptest-patch -Pdocs -DskipTests clean process-sources javadoc:javadoc-no-fork -DskipTests=true

These changes eliminate the JavaDoc errors.

For code changes:

  • Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

Add JavaDoc comments to package-info.java to avoid errors resulting from the use of Hadoop annotations.
@hadoop-yetus
Copy link

💔 -1 overall

Vote Subsystem Runtime Logfile Comment
+0 🆗 reexec 1m 28s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+0 🆗 codespell 0m 0s codespell was not available.
+0 🆗 detsecrets 0m 0s detect-secrets was not available.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ trunk Compile Tests _
+1 💚 mvninstall 46m 25s trunk passed
+1 💚 compile 25m 29s trunk passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚 compile 22m 3s trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 checkstyle 1m 8s trunk passed
+1 💚 mvnsite 1m 44s trunk passed
-1 ❌ javadoc 1m 8s /branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt hadoop-common in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.
+1 💚 javadoc 0m 42s trunk passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 spotbugs 2m 46s trunk passed
+1 💚 shadedclient 28m 23s branch has no errors when building and testing our client artifacts.
_ Patch Compile Tests _
+1 💚 mvninstall 1m 16s the patch passed
+1 💚 compile 24m 46s the patch passed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
+1 💚 javac 24m 46s the patch passed
+1 💚 compile 21m 57s the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 javac 21m 57s the patch passed
+1 💚 blanks 0m 0s The patch has no blanks issues.
+1 💚 checkstyle 1m 0s hadoop-common-project/hadoop-common: The patch generated 0 new + 0 unchanged - 15 fixed = 0 total (was 15)
+1 💚 mvnsite 1m 40s the patch passed
+1 💚 javadoc 1m 0s hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 0 new + 0 unchanged - 29 fixed = 0 total (was 29)
+1 💚 javadoc 0m 41s the patch passed with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08
+1 💚 spotbugs 2m 45s the patch passed
+1 💚 shadedclient 28m 19s patch has no errors when building and testing our client artifacts.
_ Other Tests _
+1 💚 unit 19m 15s hadoop-common in the patch passed.
+1 💚 asflicense 0m 54s The patch does not generate ASF License warnings.
234m 45s
Subsystem Report/Notes
Docker ClientAPI=1.42 ServerAPI=1.42 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5344/1/artifact/out/Dockerfile
GITHUB PR #5344
Optional Tests dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname Linux 0d99b933cb0a 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev-support/bin/hadoop.sh
git revision trunk / c40d976
Default Java Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08
Test Results https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5344/1/testReport/
Max. process+thread count 1264 (vs. ulimit of 5500)
modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
Console output https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5344/1/console
versions git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

@ayushtkn
Copy link
Member

ayushtkn commented Feb 3, 2023

Dupes #5226 / HADOOP-18576

@steveloughran
Copy link
Contributor

it does fix 29 of the complaints though, which #5226 didn't.

is it just the lack of a doc comment which broke things then, not the @interfaceAudience tag? if so, yes, let's merge -but use the original JIRA ID

@snmvaughan
Copy link
Contributor Author

This is different because it addresses the JavaDoc errors. I'd be happy with any fix that unblocks other pull requests.

@slfan1989
Copy link
Contributor

it does fix 29 of the complaints though, which #5226 didn't.
is it just the lack of a doc comment which broke things then, not the @interfaceAudience tag? if so, yes, let's merge -but use the original JIRA ID

I also don't think it's a tag issue, we can modify doc comment to solve this issue.

@snmvaughan Thank you for your contribution, but can you modify the title of the pr, as suggested by @steveloughran

Copy link
Member

@ayushtkn ayushtkn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure how it was tested actually by folks, atleast the Jenkins just checked hadoop-common, not the entire project.
I tried that and it fails further at yarn

[ERROR] /home/ayushsaxena/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/package-info.java:19: error: unknown tag: InterfaceAudience.Private
[ERROR] @InterfaceAudience.Private
[ERROR] ^
[ERROR] /home/ayushsaxena/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/package-info.java:20: error: unknown tag: InterfaceStability.Unstable
[ERROR] @InterfaceStability.Unstable
[ERROR] ^
[ERROR] 
[ERROR] Command line was: /usr/lib/jvm/java-11-openjdk-amd64/bin/javadoc @options @packages
[ERROR] 
[ERROR] Refer to the generated Javadoc files in '/home/ayushsaxena/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/target/site/apidocs' dir.
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :hadoop-yarn-server-resourcemanager

Now about the current fix, the JDK issue leads to issues for package-info.java files and can be workaround if we add Javadocs above these Interface Tags, this PR added them and made them happy, so hadoop-common passed.

  • Do we want to add these Javadocs for all package-info files?
  • We don't need them, Still?
  • Once Java issue is sorted will we revert these?
  • If we take this approach we are gonna do it for all modules with these errors? That looks like a brute-force approach to me.

I don't have objections around any approach, I am ok with whichever approach folks feel good , but I would have personally gone with this if the previous proposed fix had issues, just because it looked more clean to me and bothered less code and can be easily reverted in future....

Over to you folks, juzz my thoughts :-)

@snmvaughan snmvaughan changed the title HADOOP-18616. Java 11 JavaDoc fails due to missing package comments HADOOP-18616. Lack of a JavaDoc comment before annotation causes Java 11 JavaDoc failure Feb 6, 2023
@snmvaughan
Copy link
Contributor Author

I looked at the class JavaDoc comments and tried to come up with something useful for the package JavaDoc. I think a brute force approach (which does work) would be to add empty comments above the annotations to address the errors.

/**
 */
@InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"})
@InterfaceStability.Unstable

@snmvaughan
Copy link
Contributor Author

I tested this fix in the development environment you get from running start-build-env.sh, starting with trunk. Within that environment:

  1. I ran Javadoc using Java 8 without an issue
  2. Swtiched to Java 11 and got the errors
  3. Applied the patch and re-ran Java 11 JavaDoc without errors

Copy link
Contributor

@steveloughran steveloughran left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, merging. And least now we have a way to fix javadocs everywhere

@steveloughran steveloughran changed the title HADOOP-18616. Lack of a JavaDoc comment before annotation causes Java 11 JavaDoc failure HADOOP-18576. Lack of a JavaDoc comment before annotation causes Java 11 JavaDoc failure Feb 6, 2023
@steveloughran steveloughran merged commit aed6fce into apache:trunk Feb 6, 2023
@steveloughran
Copy link
Contributor

ok, in trunk.

@snmvaughan can you cherrypick the pr into branch-3.3 and resubmit it to see how yetus reacts there.

snmvaughan added a commit to snmvaughan/hadoop that referenced this pull request Feb 7, 2023
…pache#5344)


Add JavaDoc comments to package-info.java to avoid errors resulting from the use of Hadoop annotations.

Contributed by Steve Vaughan Jr
@snmvaughan
Copy link
Contributor Author

I've submitted PR #5362

@ayushtkn
Copy link
Member

ayushtkn commented Feb 9, 2023

This is one Jenkins report from yesterday:
#5356 (comment)
And the Javadoc build failed here:

Trunk:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5356/3/artifact/out/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt

Patch
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5356/3/artifact/out/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt

With the same error I posted above. Just posting as FYI., @steveloughran I hope that was intentional while merging this, like we just want to fix this much?

The branch had this commit as well(4th in the list):
https://github.com/virajjasani/hadoop/commits/HADOOP-18620-trunk

@steveloughran
Copy link
Contributor

the goal is to shut javadoc up...

@steveloughran
Copy link
Contributor

looks like something got left out; will do a quick followup

[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-5356/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/package-info.java:19: error: unknown tag: InterfaceAudience.Private
[ERROR] @InterfaceAudience.Private
[ERROR] ^
[ERROR] /home/jenkins/jenkins-home/workspace/hadoop-multibranch_PR-5356/ubuntu-focal/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/package-info.java:20: error: unknown tag: InterfaceStability.Unstable
[ERROR] @InterfaceStability.Unstable
[ERROR] ^

@steveloughran steveloughran mentioned this pull request Feb 9, 2023
4 tasks
@snmvaughan
Copy link
Contributor Author

I was focussing on those modules that were holding up other PRs I've submitted. The same technique could be applied to all modules.

ferdelyi pushed a commit to ferdelyi/hadoop that referenced this pull request May 26, 2023
…pache#5344)


Add JavaDoc comments to package-info.java to avoid errors resulting from the use of Hadoop annotations.

Contributed by Steve Vaughan Jr
snmvaughan added a commit to snmvaughan/hadoop that referenced this pull request Sep 17, 2023
…pache#5344)


Add JavaDoc comments to package-info.java to avoid errors resulting from the use of Hadoop annotations.

Contributed by Steve Vaughan Jr
steveloughran pushed a commit that referenced this pull request Sep 19, 2023
…5344)


Add JavaDoc comments to package-info.java to avoid errors resulting from the use of Hadoop annotations.

Contributed by Steve Vaughan Jr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants