Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

8318082: ConcurrentModificationException from IndexWriter #16271

Closed
wants to merge 3 commits into from

Conversation

pavelrappo
Copy link
Member

@pavelrappo pavelrappo commented Oct 19, 2023

Please review this fix to a bug in indexing.

The bug has been dormant for many releases and was noticed only because its likelihood was increased by JDK-8286470 in JDK 21.

Here's the bug: every time an item is indexed, it is indexed relative to the context that triggers the indexing. For example, if an item is indexed in the context of a class page, then the index will refer to that class page. However, if that same item is indexed in the context of the "Deprecated" page, then the index will refer to the "Deprecated" page.

Since an item is indexed every time it is seen, the fix is to never index an item if seen in the context of a derived page, such as hierarchy trees, indexes, and various lists (e.g. deprecated, new, preview). The fix includes a comprehensive test and updates an existing test to correct its bad assumptions.


Progress

  • Change must be properly reviewed (1 review required, with at least 1 Reviewer)
  • Change must not contain extraneous whitespace
  • Commit message must refer to an issue

Issue

  • JDK-8318082: ConcurrentModificationException from IndexWriter (Bug - P4)

Reviewers

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/16271/head:pull/16271
$ git checkout pull/16271

Update a local copy of the PR:
$ git checkout pull/16271
$ git pull https://git.openjdk.org/jdk.git pull/16271/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 16271

View PR using the GUI difftool:
$ git pr show -t 16271

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/16271.diff

Webrev

Link to Webrev Comment

@bridgekeeper
Copy link

bridgekeeper bot commented Oct 19, 2023

👋 Welcome back prappo! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

@openjdk
Copy link

openjdk bot commented Oct 19, 2023

@pavelrappo The following label will be automatically applied to this pull request:

  • javadoc

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

@openjdk openjdk bot added the javadoc javadoc-dev@openjdk.org label Oct 19, 2023
@pavelrappo pavelrappo changed the title 8318082: ConcurrentModificationException in javadoc tool 8318082: ConcurrentModificationException from IndexWriter Oct 19, 2023
@openjdk openjdk bot added the rfr Pull request is ready for review label Oct 19, 2023
@mlbridge
Copy link

mlbridge bot commented Oct 19, 2023

Webrevs

if (node.getName().toString().matches("(?i)h[1-6]")) {
if (node.getName().toString().matches("(?i)h[1-6]")
&& !(HtmlDocletWriter.this instanceof IndexWriter)
&& !(HtmlDocletWriter.this instanceof SummaryListWriter<?>)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add a Writer method to identify primary/derived/secondary writers such as isPrimaryWriter() or isDerivedWriter()? The instanceof check with these two classes looks a bit fragile and untidy.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be merged with a solution for
JDK-8318327 Merge Navigation.PageMode and HtmlConfiguration.ConditionalPage.

when we fix that.

case "index-all.html", "deprecated-list.html", "overview-tree.html",
"package-use.html", "package-tree.html", "preview-list.html",
"new-list.html", "allclasses-index.html", "allpackages-index.html",
"constant-values.html", "system-properties.html", "serialized-form.html"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The hard-coded list of derived HTML files could become outdated when new lists are added. For example, we just recently added restricted-list.html for restricted methods. Could we detect this from the presence of a dash in the file name, as this is our way to avoid conflict of derived files with Java types?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Upon second thought, feel free to ignore above comment. It's maybe not required to catch all derived files, and the explicit list makes the code much easier to understand.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While not every file with a hyphen in its name is derived, we might be able to future-proof and shorten this check by somewhat inverting it:

.filter(r -> {
    String f = r.group("file");
    if (!f.contains("-"))
        return false;
    return switch (f) {
        case "package-summary.html",
                "module-summary.html",
                "overview-summary.html",
                "help-doc.html" -> false;
        default -> {
            String p = r.group("path");
            yield !p.contains("/doc-files/") && !p.startsWith("doc-files/");
        }
    };
})

@hns?

Copy link
Contributor

@jonathan-gibbons jonathan-gibbons left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, but would be nice to eventually replace the use of instanceof perhaps by using the proposed PageKind and/or some new method on HtmlDocletWriter.

As always, coming up with a decent name (and documentation) for new methods is often the hardest part of a fix.

if (node.getName().toString().matches("(?i)h[1-6]")) {
if (node.getName().toString().matches("(?i)h[1-6]")
&& !(HtmlDocletWriter.this instanceof IndexWriter)
&& !(HtmlDocletWriter.this instanceof SummaryListWriter<?>)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be merged with a solution for
JDK-8318327 Merge Navigation.PageMode and HtmlConfiguration.ConditionalPage.

when we fix that.

@openjdk
Copy link

openjdk bot commented Oct 19, 2023

@pavelrappo This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8318082: ConcurrentModificationException from IndexWriter

Reviewed-by: jjg

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 39 new commits pushed to the master branch:

  • a045258: 8209595: MonitorVmStartTerminate.java timed out
  • 2121575: 8318421: AbstractPipeline.sourceStageSpliterator() chases pointers needlessly
  • 200b5a2: 8318420: AbstractPipeline invokes overridden method in constructor
  • 5281499: 8316563: test tools/jpackage/linux/LinuxResourceTest.java fails on CentOS Linux release 8.5.2111 and Fedora 27
  • 8065233: 8318598: FFM stylistic cleanups
  • b1228de: 8314588: gc/metaspace/TestMetaspaceInitialization.java failed "assert(capacity_until_gc >= committed_bytes) failed: capacity_until_gc: 3145728 < committed_bytes: 3211264"
  • 5a97411: 8317971: RISC-V: implement copySignF/D and signumF/D intrinsics
  • 9144287: 8318601: Remove javadoc text about restricted methods
  • 4010642: 8077371: Binary files in JAXP test should be removed
  • fe52917: 8318457: Use prefix-less prepend methods directly to reduce branches in String concat expressions
  • ... and 29 more: https://git.openjdk.org/jdk/compare/47bb1a1cefa242c39c22a8f2aa08d7d357c260b9...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

@openjdk openjdk bot added the ready Pull request is ready to be integrated label Oct 19, 2023
|| context.inTags.contains(DocTree.Kind.INDEX)
|| htmlWriter instanceof IndexWriter
|| htmlWriter instanceof SummaryListWriter<?>) {
if (context.isFirstSentence && context.inSummary || context.inTags.contains(DocTree.Kind.INDEX)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion for bonus points: how about introducing context.isIndexable to match htmlWriter.isIndexable

Copy link
Contributor

@jonathan-gibbons jonathan-gibbons left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the "opt-in" nature of isIndexable.

Approved, but added a bonus suggestion for your consideration.

@@ -370,7 +372,8 @@ public Content createAnchorAndSearchIndex(Element element, String tagText, Strin
@SuppressWarnings("preview")
Content createAnchorAndSearchIndex(Element element, String tagText, Content tagContent, String desc, DocTree tree) {
Content result;
if (context.isFirstSentence && context.inSummary || context.inTags.contains(DocTree.Kind.INDEX)) {
if (context.isFirstSentence && context.inSummary || context.inTags.contains(DocTree.Kind.INDEX)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you introduce context.isIndexable then the code may read better if you flip the then/else parts.

if (htmlWriter.isIndexable() && context.isIndexable()) {
    // index it
} else {
    // don't index it
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's leave it out for now. It's not because I don't like flipping the if and else arms around, it's because I'm not very comfortable with introducing isIndexable to Context at the moment.

@pavelrappo
Copy link
Member Author

/integrate

@openjdk
Copy link

openjdk bot commented Oct 23, 2023

Going to push as commit fc29a2e.
Since your change was applied there have been 52 commits pushed to the master branch:

  • 729f4c5: 8318507: G1: Improve remset clearing for humongous candidates
  • 4eab39d: 8318585: Rename CodeCache::UnloadingScope to UnlinkingScope
  • ffadd63: 8317868: Add @sealedGraph to MethodHandleDesc and descendants
  • ecd25e7: 8318484: Initial version of cdsConfig.hpp
  • a876beb: 8316741: BasicStroke.createStrokedShape miter-limits failing on small shapes
  • 4cf195f: 8318573: The nsk.share.jpda.SocketConnection should fail if socket was closed.
  • af2f4bf: 8318622: ProblemList gc/cslocker/TestCSLocker.java on linux-x64 in Xcomp mode
  • a1a62d9: 8306308: (ch) Writer created by Channels::newWriter may lose data
  • 77b2394: 8318482: problemlist compiler/codecache/CheckLargePages.java on Linux-x64 until JDK-8317831 is fixed
  • d3ebb4a: 8317373: Add Telia Root CA v2
  • ... and 42 more: https://git.openjdk.org/jdk/compare/47bb1a1cefa242c39c22a8f2aa08d7d357c260b9...master

Your commit was automatically rebased without conflicts.

@openjdk openjdk bot added the integrated Pull request has been integrated label Oct 23, 2023
@openjdk openjdk bot closed this Oct 23, 2023
@openjdk openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Oct 23, 2023
@openjdk
Copy link

openjdk bot commented Oct 23, 2023

@pavelrappo Pushed as commit fc29a2e.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
integrated Pull request has been integrated javadoc javadoc-dev@openjdk.org
3 participants