8352249: Remove incidental whitespace in traditional doc comments

[hns]

Please review a patch to remove incidental indentation from traditional doc comments, comparable to doing a String.stripIndent() on the comment contents but without special treatment of a last blank line. This adds a stripIndent() method to the Tokens.Comment interface and a new JavadocTokenizer.StrippedComment nested class that implements indentation stripping while maintaining a map of position offsets to the original comment.

While the patch changes javadoc output by removing leading whitespace, the change is generally not visible in the browser except in <pre> elements, which is the point of this enhancement.

The change affects most tree positions in the AST checker tests in javac/doctree, but mostly does not affect the structure of the parsed trees, with the exception of BreakIterator tests in FirstSentenceTest.java.

BreakIterator does not recognize .\n immediately followed by a lower case letter as sentence break, while it recognizes the break if the letter is an upper-case letter or if there is a space between the line break and the letter. I find this rule a bit peculiar but AFAICT we can't influence the behavior of BreakIterator, so I have added tests that cover both lower and upper-case behavior.

The source position lookup in the stripped comment is implemented by creating a new OffsetMap that translates from the stripped to the original comment, then using the original comment's OffsetMap to translate the position to the source file. OffsetMap is relatively lightweight (usually 2 int[] elements per comment paragraph), so the added overhead is not too bad.

Inspired by JDK-8305688 I did various test builds with restricted jobs and memory settings, but didn't notice any change in processing or memory overhead to API docs builds.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8352249: Remove incidental whitespace in traditional doc comments (Enhancement - P3)

Reviewers

• Chen Liang (@liach - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/24032/head:pull/24032
$ git checkout pull/24032

Update a local copy of the PR:
$ git checkout pull/24032
$ git pull https://git.openjdk.org/jdk.git pull/24032/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 24032

View PR using the GUI difftool:
$ git pr show -t 24032

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/24032.diff

Using Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back hannesw! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@hns This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8352249: Remove incidental whitespace in traditional doc comments

Reviewed-by: liach

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 72 new commits pushed to the master branch:

• fcc2a24: 8350813: Rendering of bulky sound bank from MIDI sequence can cause OutOfMemoryError
• ac3ad03: 8350589: Investigate cleaner implementation of AArch64 ML-DSA intrinsic introduced in JDK-8348561
• 8a1c85e: 8350898: Shenandoah: Eliminate final roots safepoint
• 8e999b8: 8351969: Add Public Identifiers to the JDK built-in Catalog
• c7f3338: 8352109: java/awt/Desktop/MailTest.java fails in platforms where Action.MAIL is not supported
• fed34e4: 8351933: Inaccurate masking of TC subfield decrement in ForkJoinPool
• 8f64ccc: 8350485: C2: factor out common code in Node::grow() and Node::out_grow()
• c2be19c: 8351902: RISC-V: Several tests fail after JDK-8351145
• e57b272: 8350578: Refactor useless Parse and Template Assertion Predicate elimination code by using a PredicateVisitor
• 577ede7: 8352302: Test sun/security/tools/jarsigner/TimestampCheck.java is failing
• ... and 62 more: https://git.openjdk.org/jdk/compare/4c5956d7481e043c35f5dc78f095516288a00a2e...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[openjdk]

@hns The following labels will be automatically applied to this pull request:

• compiler
• javadoc

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing lists. If you would like to change these labels, use the /label pull request command.

[mlbridge]

Webrevs

• 01: Full - Incremental (1c21de77)
• 00: Full (eb9a090c)

[liach]

Looks good.

[hns]

Thanks!

/integrate

[openjdk]

Going to push as commit a5d06a1.
Since your change was applied there have been 73 commits pushed to the master branch:

• 74df384: 8352428: GenShen: Old-gen cycles are still looping
• fcc2a24: 8350813: Rendering of bulky sound bank from MIDI sequence can cause OutOfMemoryError
• ac3ad03: 8350589: Investigate cleaner implementation of AArch64 ML-DSA intrinsic introduced in JDK-8348561
• 8a1c85e: 8350898: Shenandoah: Eliminate final roots safepoint
• 8e999b8: 8351969: Add Public Identifiers to the JDK built-in Catalog
• c7f3338: 8352109: java/awt/Desktop/MailTest.java fails in platforms where Action.MAIL is not supported
• fed34e4: 8351933: Inaccurate masking of TC subfield decrement in ForkJoinPool
• 8f64ccc: 8350485: C2: factor out common code in Node::grow() and Node::out_grow()
• c2be19c: 8351902: RISC-V: Several tests fail after JDK-8351145
• e57b272: 8350578: Refactor useless Parse and Template Assertion Predicate elimination code by using a PredicateVisitor
• ... and 63 more: https://git.openjdk.org/jdk/compare/4c5956d7481e043c35f5dc78f095516288a00a2e...master

Your commit was automatically rebased without conflicts.

[openjdk]

@hns Pushed as commit a5d06a1.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.