8334121: Anonymous class capturing two enclosing instances fails to compile

[mcimadamore]

This PR contains a fix for an issue when translating local/anon classes. Currently, javac assumes that a local/anon class C must have an enclosing instance whose type is the innermost enclosing class of C. But this is not always true - there are cases where the innermost enclosing instance is in early construction context, so we can't really refer to its members from the local class. Moreover, even if the innermost class was accessible, there could be cases where the enclosing instance of that innermost enclosing class is not accessible.

In other words, we cannot rely on the assumption that, given a local/anonymous class, there exist a chain of enclosing instances from the innermost to the outermost. Some chain exists, but it might not start from the innermost enclosing class, and it might have holes.

The core of the fix is the addition of a new function in Symbol, namely Symbol::innermostAccessibleEnclosingClass. As the name implies, the goal of this function is to return the class symbol corresponding to the innermost enclosing type that is accessible from the symbol's class. This is used by Lower to determine the type of this$0, which allows us to construct a chain of enclosing instances that skips over all the inaccessible types.

I've also updated the Symbol::hasOuterInstance method, so that it, too, will skip over inaccessible enclosing type (and only return true if some accessible enclosing type is found).

This cleanup brings a lot of simplifications to Lower.FreeVarCollector. That class is used to compute the set of captured variables inside a local/anon class. There is a workaround in this class so that if a local/anon class occurs in a pre-construction context (where the enclosing this is not accessible), we say that the local/anon class captures the enclosing constructor parameter of the place where the local/anon class occurs (e.g. the enclosing 'this's enclosing this). This adds complexity to the code, as now the set of captured variables depends on the state of Lower. With this PR, all that logic is now gone, and FreeVarCollector is effectively a stateless visitor.

---

Progress

• Change must be properly reviewed (1 review required, with at least 1 Reviewer)
• Change must not contain extraneous whitespace
• Commit message must refer to an issue

Issue

• JDK-8334121: Anonymous class capturing two enclosing instances fails to compile (Bug - P4)

Reviewers

• Jan Lahoda (@lahodaj - Reviewer)
• Vicente Romero (@vicente-romero-oracle - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk.git pull/19900/head:pull/19900
$ git checkout pull/19900

Update a local copy of the PR:
$ git checkout pull/19900
$ git pull https://git.openjdk.org/jdk.git pull/19900/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 19900

View PR using the GUI difftool:
$ git pr show -t 19900

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/19900.diff

Webrev

Link to Webrev Comment

[bridgekeeper]

👋 Welcome back mcimadamore! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

[openjdk]

@mcimadamore This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8334121: Anonymous class capturing two enclosing instances fails to compile

Reviewed-by: jlahoda, vromero

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 47 new commits pushed to the master branch:

• 45c4eaa: 8335274: SwitchBootstraps.ResolvedEnumLabels.resolvedEnum should be final
• 486aa11: 8335237: ubsan: vtableStubs.hpp is_vtable_stub exclude from ubsan checks
• f4d8c00: 8334562: Automate com/sun/security/auth/callback/TextCallbackHandler/Default.java test
• 49eb00d: 8299813: java/nio/channels/DatagramChannel/Disconnect.java fails with jtreg test timeout due to lost datagram
• 8ec378a: 8277949: (dc) java/nio/channels/DatagramChannel/AdaptorBasic.java failed in timeout
• c798316: 8269657: Test java/nio/channels/DatagramChannel/Loopback.java failed: Unexpected message
• 99d2bbf: 8334433: jshell.exe runs an executable test.exe on startup
• 6f4ddc2: 8335142: compiler/c1/TestTraceLinearScanLevel.java occasionally times out with -Xcomp
• 3b3a19e: 8335314: Problem list compiler/uncommontrap/DeoptReallocFailure.java
• d457609: 8319947: Recursive lightweight locking: s390x implementation
• ... and 37 more: https://git.openjdk.org/jdk/compare/741a0f39dd1fffc1caaa8d69bfe3662dad830452...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

[openjdk]

@mcimadamore The following label will be automatically applied to this pull request:

• compiler

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

[mcimadamore]

This change is caused by the fact that we no longer need to capture the constructor parameter of where the local class occurs, but we can simply point to a different enclosing instance. Now, I notice that we seem to set ACC_MANDATED even for enclosing instance parameters for local/anon classes. I'm not sure if this is correct, since these parameters are not, strictly speaking, mandated by the spec (which is silent when it comes to local/anon class translation). Perhaps we can rectify as a separate cleanup - but if deemed important I can also address here.

[mlbridge]

Webrevs

• 00: Full (d71313be)

[lahodaj]

Looks reasonable to me.

[vicente-romero-oracle]

lgtm

[mcimadamore]

/integrate

[openjdk]

Going to push as commit 9dfcd75.
Since your change was applied there have been 210 commits pushed to the master branch:

• 000de30: 8335269: [Graal] occasional timeout in java/lang/StringBuffer/TestSynchronization.java with loom
• 4635531: 8335159: Move method reference to lambda desugaring before Lower
• a253e0f: 8335642: Hide Transform implementation for Class-File API
• 2b0adfc: 8335817: javac AssertionError addLocalVar checkNull
• a96de6d: 8336256: memcpy short value to int local is incorrect in VtableStubs::unsafe_hash
• 3f2636d: 8335820: java/lang/invoke/LFCaching/LFSingleThreadCachingTest.java fails due to IllegalArgumentException: hash must be nonzero
• a9f5e76: 8335905: CompoundElement API cleanup
• 6f325db: 8310915: Typo in aarch64.ad: "envcodings"
• ae9f318: 8336301: test/jdk/java/nio/channels/AsyncCloseAndInterrupt.java leaves around a FIFO file upon test completion
• 4166e53: 8318106: Generated HTML for snippet does not always contain an id
• ... and 200 more: https://git.openjdk.org/jdk/compare/741a0f39dd1fffc1caaa8d69bfe3662dad830452...master

Your commit was automatically rebased without conflicts.

[openjdk]

@mcimadamore Pushed as commit 9dfcd75.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.