Parser changes for JEP 443 #1517

datho7561 · 2023-10-26T16:05:34Z

What it does

Support parsing unnamed variables and unnamed patterns.

How to test

Create a Java 21 project with preview features enabled. Ensure that the parser can parse unnamed variables and unnamed patterns as described in https://openjdk.org/jeps/443.

Author checklist

I have thoroughly tested my changes
The change is following the coding conventions
I have signed the Eclipse Contributor Agreement (ECA)

robstryker · 2023-10-26T19:52:47Z

@mpalat Hi Manoj: We were looking for feedback on this draft PR while we work to get all the tests green.

Thanks in advance.

datho7561 · 2023-10-27T20:38:23Z

waiting on #1523 to be merged for many of the tests to be fixed.

mickaelistria · 2023-10-30T15:59:30Z

This change looks pretty good to me.
I see a couple of tests are failing. Those tests may require an update to turn green and consider the PR for thorough review and merge. There are several ways to turn the test green; which one to pick is up to you to chose the one that work best.

Do not change the behavior for those tests: new rules and code don't introduce deviation in current behavior so the same rules remain used (it's often very hard as far as I can tell)
change the test case: sometimes, it's possible to ensure we go through the same rules as before by tweaking a bit the test case; for example adding 1 import, 1 statement, 1 parameter....)
change the test assertion: if the rule cannot be avoided, then unless the message becomes more confusing, it is fine to change the assertion to mention the new message/rule. As those messages are presented to users, I think you can play a bit with the readableName to improve those when the default rule name is hard to understand.

HTH

akurtakov · 2023-11-01T07:30:34Z

@mpalat Would you please review this one?

mpalat · 2023-11-01T07:40:46Z

@mpalat Would you please review this one?

sure.will take a look

First of all, the grammar change - java.g. I think there are too many changes here to accommodate the underscore. An alternate approach to try out is to accept the underscore in the parser, and in some way use the mechanism similar to var type. Could you please try out this alternate? - like how @mickaelistria is doing in his patch - accept a larger set for unnammed and then flag errors at a later stage rather than having many changes in th java.g

Also, I thinks its better to cut down into incremental patches similar to those in #1106

datho7561 · 2023-11-02T13:07:56Z

Hi Manoj. Thanks for the review. I think I found a way to reduce the number of changes necessary in java.g, but it still requires modifying the Scanner to add a new UNDERSCORE token for _ in order for the grammar to remain LALR(1). Are you okay with this?

mpalat · 2023-11-03T06:55:28Z

Hi Manoj. Thanks for the review. I think I found a way to reduce the number of changes necessary in java.g, but it still requires modifying the Scanner to add a new UNDERSCORE token for _ in order for the grammar to remain LALR(1). Are you okay with this?

@datho7561 - That should be fine -

I was experimenting with something similar here as shown below:
" BeginRecordPattern
UnderScore
".

However, I would need to see those changes to comment further- please provide the changes for me to take a look

datho7561 · 2023-11-08T19:28:14Z

@mpalat Alright, I've pushed the changes so that less work is done in Parser.java. The tests should be green now.

mpalat · 2023-11-09T05:21:24Z

@mpalat Alright, I've pushed the changes so that less work is done in Parser.java. The tests should be green now.

Noting that I am away from work (or supposed to) for the last couple of days not being in the best of health, am just looking at this as health permits. Also my time is limited in the next few weeks - in short the reviews may not happen fast.

That said, to have more number of (very good) eyes look at it and to continue the review, have discussed with @srikanth-sankaran and he has agreed to help out with the new patches once he is finished reviewing String templates this is being implemented by Jay with inputs from Srikanth.

And ideally it would be good to take it one by one since both Unnammed Classes and this have java.g changes and that we need to really concentrate on one problem of grammar change at a time to evolve a minimal, maintainable change. I would suggest to wait for @mickaelistria unnamed class and then take this patch unnammed patterns . Both of us (@srikanth-sankaran and I ) felt that the grammar changes should be very minimal in the unnammed patterns - much less than the java.g changes listed in the patch, but we will have to scrutinize/try it out to be sure.

datho7561 · 2023-11-13T16:46:01Z

@mpalat Hello Manoj, is it possible for you to review this PR sometime soon? Mickael and I are still working on getting the tests for the Unnamed Classes grammar change PR to pass, whereas on this PR they are passing.

mickaelistria · 2023-11-13T17:44:01Z

@mpalat Thanks for considering the unnamed class as higher priority, but at the moment, I think the patch is not yet ready for review (as long as tests are stuck or some are failing), so I would recommend for the best interest of the project that you remove the unnamed class patch for your current monitoring and instead consider the unnamed variable patch which is assumed complete already.

datho7561 · 2023-11-14T14:22:48Z

@srikanth-sankaran when you have time, could you please review this PR?

srikanth-sankaran · 2023-11-15T04:59:17Z

@srikanth-sankaran when you have time, could you please review this PR?

Yes, I will, thanks.

mpalat · 2023-11-16T00:40:11Z

@srikanth-sankaran when you have time, could you please review this PR?

Yes, I will, thanks.

Thanks @srikanth-sankaran. @datho7561 as informed earlier, am tied up next few days, however will work with Srikanth as time permits

jarthana · 2024-01-19T12:28:51Z

I am Okay with the changes in the specific set of files I looked at. I still see some requested from Srikanth pending. Once they are resolved, we are good to go.

srikanth-sankaran · 2024-01-19T12:51:00Z

I am Okay with the changes in the specific set of files I looked at. I still see some requested from Srikanth pending. Once they are resolved, we are good to go.

@datho7561, please go over all the past comments to make sure everything is addressed. Once you confirm, I'll make a sanity check and help close this

datho7561 · 2024-01-19T14:18:46Z

The main thing that isn't addressed is this bug in code generation that I created a separate PR for: #1816

Other than that it should be good.

srikanth-sankaran · 2024-01-21T07:54:32Z

A heads up that fundamental restructuring of pattern matching support is happening via #1885 which should get merged in a day or two. I'll see what if any changes would be needed to harmonize this work with that PR and help align them.

srikanth-sankaran · 2024-01-23T12:42:54Z

A heads up that fundamental restructuring of pattern matching support is happening via #1885 which should get merged in a day or two. I'll see what if any changes would be needed to harmonize this work with that PR and help align them.

This is done and if you are update your changes, I can take it forward. Thanks

srikanth-sankaran · 2024-01-24T10:30:32Z

OK, I am starting a quick pass over this and will try to dedicate the next 2-3 days to this and see if this can be merged.
My plan is to raise follow up tickets for any issues discovered and not block this PR unless something absolutely in the nature of a show stopper is uncovered.

srikanth-sankaran · 2024-01-26T03:51:26Z

I am having a tough time rebasing to master - I know this has been a recurring exercise and I wanted to take it off of your hands. But having a tough time, hitting various parser generator bugs ... I will continue to look into this, in the meantime if you see this message and can spare the cycles to update against master and force-push, that would be super

srikanth-sankaran · 2024-01-26T04:39:42Z

I am having a tough time rebasing to master - I know this has been a recurring exercise and I wanted to take it off of your hands. But having a tough time, hitting various parser generator bugs ... I will continue to look into this, in the meantime if you see this message and can spare the cycles to update against master and force-push, that would be super

OK, I have figured out the parser generator breakage. But merge is still proving to be unintutive. I think I'll squash all commit into one and then rebase. This may result in a simpler workflow

srikanth-sankaran · 2024-01-26T04:55:02Z

I am having a tough time rebasing to master - I know this has been a recurring exercise and I wanted to take it off of your hands. But having a tough time, hitting various parser generator bugs ... I will continue to look into this, in the meantime if you see this message and can spare the cycles to update against master and force-push, that would be super

OK, I have figured out the parser generator breakage. But merge is still proving to be unintutive. I think I'll squash all commit into one and then rebase. This may result in a simpler workflow

No, spoke too soon it seems. It was working for a while but now I get various errors while running the parser generator scripts:

  [java] parser6.rsc creation complete
     [java] parser7.rsc creation complete
     [java] parser8.rsc creation complete
     [java] parser9.rsc creation complete
     [java] java.lang.ArrayIndexOutOfBoundsException: Index 61296 out of bounds for length 61296
     [java] 	at org.apache.tools.ant.taskdefs.ExecuteJava.execute(ExecuteJava.java:194)
     [java] 	at org.apache.tools.ant.taskdefs.Java.run(Java.java:892)
     [java] 	at org.apache.tools.ant.taskdefs.Java.executeJava(Java.java:232)
     [java] 	at org.apache.tools.ant.taskdefs.Java.executeJava(Java.java:136)
     [java] 	at org.apache.tools.ant.taskdefs.Java.execute(Java.java:109)
     [java] 	at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:299)
     [java] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [java] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
     [java] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     [java] 	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
     [java] 	at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:99)
     [java] 	at org.apache.tools.ant.Task.perform(Task.java:350)
     [java] 	at org.apache.tools.ant.Target.execute(Target.java:449)
     [java] 	at org.apache.tools.ant.Target.performTasks(Target.java:470)
     [java] 	at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1401)
     [java] 	at org.apache.tools.ant.Project.executeTarget(Project.java:1374)
     [java] 	at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
     [java] 	at org.eclipse.ant.internal.launching.remote.EclipseDefaultExecutor.executeTargets(EclipseDefaultExecutor.java:34)
     [java] 	at org.apache.tools.ant.Project.executeTargets(Project.java:1264)

srikanth-sankaran · 2024-01-26T05:05:39Z

OK, I am giving up on the rebase/merge conflicts/ approach. I will review the PR as is. Before merging it needs to be brought to uptodate state anyway. This you can handle.

srikanth-sankaran · 2024-01-26T05:32:26Z

OK, I am giving up on the rebase/merge conflicts/ approach. I will review the PR as is. Before merging it needs to be brought to uptodate state anyway. This you can handle.

Summary: I am not blocked. I am reviewing by comparing your branch HEAD against master as opposed to rebase and then compare with previous version. It should all wash out

srikanth-sankaran · 2024-01-26T08:11:03Z

...re.compiler.batch/src/org/eclipse/jdt/internal/compiler/ast/AbstractVariableDeclaration.java

+	public boolean isUnnamed(BlockScope scope) {
+		return ((this.name.length == 1 && this.name[0] == '_') || ("\\u005F".equals(this.name.toString()))) //$NON-NLS-1$
+				&& scope.compilerOptions().sourceLevel >= ClassFileConstants.JDK21
+				&& scope.compilerOptions().enablePreviewFeatures;


Curious - is the ("\\u005F".equals(this.name.toString()))) really required ? Won't the scanner internalize the unicode and return an '_' ??

srikanth-sankaran · 2024-01-26T08:16:20Z

org.eclipse.jdt.core.compiler.batch/src/org/eclipse/jdt/internal/compiler/parser/Scanner.java

+		if ((length = this.withoutUnicodePtr) == 1) {
+			if (this.withoutUnicodeBuffer[0] == '_') {
+				return TokenNameUNDERSCORE;
+			}


This is what I meant. When an underscore occurs as unicode, the scanner converts that to '_' - No ??

srikanth-sankaran

Ok, I have made a full pass over the changes and exhausted all questions and comments. Please proceed to

Rebase against master - there are some more pattern related changes on master that need to be carefully picked up.
I have provided a comment about unicode - it is more a nit and academic curiosity actually. Please take a look.
Do you swear under penalty of perjury that you have tested every snippet in org.eclipse.jdt.core.tests.compiler.regression.UnnamedPatternsAndVariablesTest and in general every modified test to compare behavior with JDK21 ?? :)

If all this is addressed, we are ready to merge this into master.

srikanth-sankaran · 2024-01-26T08:53:13Z

FYI, I have been drastically overhauling our patterns implementation to make it simpler to understand and maintain:

See:

#1884 - [Patterns] Alternate, simpler pattern binding management implementation
#1897 - [Patterns] Cleanup/refactor patterns implementation
#1887 - [Patterns] ECJ tolerates erroneous redeclaration of pattern bindings in some cases.
#1856 - [switch][record patterns] NPE: Cannot invoke "org.eclipse.jdt.internal.compiler.lookup.MethodBinding.isStatic()
#1853 - [switch][pattern] Scope of pattern binding extends illegally resulting in wrong diagnostic
#1835 - [Switch][patterns][record] AssertionError at org.eclipse.jdt.internal.compiler.ast.YieldStatement.addSecretYieldResultValue(YieldStatement.java
#1788 - Inference issue between the diamond syntax and pattern matching (switch on objects)
#1759 - Pattern variable is not recognized in AND_AND_Expression
#1726 - [21] Pattern-matching in instanceof doesn't work
#1485 ECJ hangs when pattern matching code is used in a nested conditional expression
#1364 - SelectionParser behavior erratic wrt to live pattern variables upon loop exit
#1288 - Open Declaration (F3) sometimes not working for "Pattern Matching for instanceof
#1263 - CCE: LocalDeclaration cannot be cast to class ForeachStatement
#1278 Wrong method redirect
#1197 - Get rid of separate stacks for managing patterns during parsing
#1195 Open declaration results in ClassCastException: LocalDeclaration cannot be cast to LambdaExpression
#1076 ECJ accepts invalid Java code instanceof final Type
#968 - JavaElementHyperlinkDetector: Parser runs into NegativeArraySizeException in some cases
#769 Open Declaration(F3) broken in pattern instanceof
#567 - [Patterns] Report unused variable for variables declared in instanceof pattern
#1907 - [Patterns] Withdraw support for record pattern element variable in enhanced for loop
#456 - Syntax error on token "RestrictedIdentifierWhen"

There is plenty more particularly in the area of code generation that will change.

An alert that your work will also come under this comprehensive overhaul. If you wish to be notified, I can CC you on the various tickets.

@iloveeclipse - FYI

srikanth-sankaran · 2024-01-26T08:56:56Z

This one is ready to merge - let us move forward please. I am aware that you have had to repeatedly rebase and resolve conflicts - what can look like a thanksless exercise sometimes. Really appreciate your patience @datho7561 !

srikanth-sankaran · 2024-01-26T09:13:49Z

So everybody is on same page: I am listing the next steps for the 4 PRs @datho7561 has been working on:

Parser changes for JEP 443 #1517 - this one - is ready for merge subjecting to Parser changes for JEP 443 #1517 (review)
List of patterns in case statements #1742 - I have proposed that David and I will jointly own and close this as part of cleanup work going on patterns.
Fix nested pattern code generation #1816 - I have proposed that David and I will jointly own and close this as part of cleanup work going on patterns.
Jep 445 grammar #1521 - I can review the parser/grammar changes next week. Other areas we need reviewers. @mpalat please help with delegating so we can divide and conquer

iloveeclipse · 2024-01-26T09:17:11Z

Please rebase and squash all intermediate commits to one before merging.

Contains support for unnamed patterns and variables (JEP443), including parsing, analysis and code generation, but omits support for lists of patterns in case statements, which will be handled in a future PR. Signed-off-by: David Thompson <davthomp@redhat.com>

datho7561 · 2024-01-26T19:22:43Z

Alright, I've rebased the branch, squashed the commits, addressed the comment about Unicode support (you're right, I didn't need that check), and double checked I have all the snippets from the JEP (excluding those dealing with lists of patterns).

datho7561 force-pushed the 893-jep-443 branch from baa7215 to b366cad Compare October 30, 2023 13:00

datho7561 force-pushed the 893-jep-443 branch 2 times, most recently from 0a69259 to 267f76d Compare October 30, 2023 20:48

rgrunber linked an issue Nov 1, 2023 that may be closed by this pull request

[21] JEP 443: Unnamed Patterns and Variables (Preview) #893

Closed

datho7561 force-pushed the 893-jep-443 branch 8 times, most recently from 6c07fc4 to 5274544 Compare November 8, 2023 19:24

datho7561 marked this pull request as ready for review November 8, 2023 19:24

datho7561 mentioned this pull request Nov 13, 2023

Maven tests execution for org.eclipse.core.tests.compiler is often stuck #1597

Closed

datho7561 force-pushed the 893-jep-443 branch from 5274544 to bbaa65d Compare November 14, 2023 20:07

akurtakov mentioned this pull request Nov 15, 2023

Jep 445 grammar #1521

Closed

3 tasks

datho7561 mentioned this pull request Jan 18, 2024

List of patterns in case statements #1742

Closed

3 tasks

jarthana approved these changes Jan 19, 2024

View reviewed changes

datho7561 force-pushed the 893-jep-443 branch from 83d9d1d to 8af9b30 Compare January 19, 2024 14:13

datho7561 force-pushed the 893-jep-443 branch from 94b354d to 0e58669 Compare January 23, 2024 16:10

srikanth-sankaran reviewed Jan 26, 2024

View reviewed changes

srikanth-sankaran approved these changes Jan 26, 2024

View reviewed changes

srikanth-sankaran mentioned this pull request Jan 26, 2024

Fix nested pattern code generation #1816

Closed

3 tasks

datho7561 force-pushed the 893-jep-443 branch from 0e58669 to 22fcccc Compare January 26, 2024 16:30

jarthana merged commit b4875fc into eclipse-jdt:master Jan 29, 2024
9 checks passed

datho7561 deleted the 893-jep-443 branch January 29, 2024 14:10

iloveeclipse mentioned this pull request Jan 30, 2024

Build I20240129-1800 is unstable eclipse-platform/eclipse.platform.releng.aggregator#1757

Closed

This was referenced Feb 15, 2024

[Patterns][Unnamed] VerifyError with unnamed pattern variable in instanceof #2024

Closed

VerifyError with unnamed pattern variable in instanceof #2025

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parser changes for JEP 443 #1517

Parser changes for JEP 443 #1517

datho7561 commented Oct 26, 2023 •

edited

robstryker commented Oct 26, 2023

datho7561 commented Oct 27, 2023

mickaelistria commented Oct 30, 2023

akurtakov commented Nov 1, 2023

mpalat commented Nov 1, 2023 •

edited

datho7561 commented Nov 2, 2023

mpalat commented Nov 3, 2023 •

edited

datho7561 commented Nov 8, 2023

mpalat commented Nov 9, 2023

datho7561 commented Nov 13, 2023

mickaelistria commented Nov 13, 2023

datho7561 commented Nov 14, 2023

srikanth-sankaran commented Nov 15, 2023

mpalat commented Nov 16, 2023

jarthana commented Jan 19, 2024

srikanth-sankaran commented Jan 19, 2024

datho7561 commented Jan 19, 2024

srikanth-sankaran commented Jan 21, 2024

srikanth-sankaran commented Jan 23, 2024

srikanth-sankaran commented Jan 24, 2024

srikanth-sankaran commented Jan 26, 2024

srikanth-sankaran commented Jan 26, 2024

srikanth-sankaran commented Jan 26, 2024 •

edited

srikanth-sankaran commented Jan 26, 2024

srikanth-sankaran commented Jan 26, 2024

srikanth-sankaran Jan 26, 2024

srikanth-sankaran Jan 26, 2024 •

edited

srikanth-sankaran left a comment

srikanth-sankaran commented Jan 26, 2024 •

edited

srikanth-sankaran commented Jan 26, 2024

srikanth-sankaran commented Jan 26, 2024

iloveeclipse commented Jan 26, 2024

datho7561 commented Jan 26, 2024

Parser changes for JEP 443 #1517

Parser changes for JEP 443 #1517

Conversation

datho7561 commented Oct 26, 2023 • edited

What it does

How to test

Author checklist

robstryker commented Oct 26, 2023

datho7561 commented Oct 27, 2023

mickaelistria commented Oct 30, 2023

akurtakov commented Nov 1, 2023

mpalat commented Nov 1, 2023 • edited

datho7561 commented Nov 2, 2023

mpalat commented Nov 3, 2023 • edited

datho7561 commented Nov 8, 2023

mpalat commented Nov 9, 2023

datho7561 commented Nov 13, 2023

mickaelistria commented Nov 13, 2023

datho7561 commented Nov 14, 2023

srikanth-sankaran commented Nov 15, 2023

mpalat commented Nov 16, 2023

jarthana commented Jan 19, 2024

srikanth-sankaran commented Jan 19, 2024

datho7561 commented Jan 19, 2024

srikanth-sankaran commented Jan 21, 2024

srikanth-sankaran commented Jan 23, 2024

srikanth-sankaran commented Jan 24, 2024

srikanth-sankaran commented Jan 26, 2024

srikanth-sankaran commented Jan 26, 2024

srikanth-sankaran commented Jan 26, 2024 • edited

srikanth-sankaran commented Jan 26, 2024

srikanth-sankaran commented Jan 26, 2024

srikanth-sankaran Jan 26, 2024

Choose a reason for hiding this comment

srikanth-sankaran Jan 26, 2024 • edited

Choose a reason for hiding this comment

srikanth-sankaran left a comment

Choose a reason for hiding this comment

srikanth-sankaran commented Jan 26, 2024 • edited

srikanth-sankaran commented Jan 26, 2024

srikanth-sankaran commented Jan 26, 2024

iloveeclipse commented Jan 26, 2024

datho7561 commented Jan 26, 2024

datho7561 commented Oct 26, 2023 •

edited

mpalat commented Nov 1, 2023 •

edited

mpalat commented Nov 3, 2023 •

edited

srikanth-sankaran commented Jan 26, 2024 •

edited

srikanth-sankaran Jan 26, 2024 •

edited

srikanth-sankaran commented Jan 26, 2024 •

edited