Support Java JLS 13 and JLS14 preview in M3 models and Java ASTs; also make ASTs jump through the correctness specification in lang::analysis::AST #1936

jurgenvinju · 2024-04-16T11:54:39Z

This branch starts from the main branch where Java support for M3 and ASTs was done up to level 8 of the Java Language Standard (JLS8).

However several key concepts were missing from the syntax trees namely type parameters and modifiers and annotations here and there. Some of these concepts were represented only in the M3 database, and at the time it was decided that double representation was not good for memory consumption. Today and with experience from the C++/C M3 models, PHP and Ada, this design decision has shifted: every AST must be fully informative, and the AST model must be aligned with the M3 model by sharing src (physical location), decl (logical location) and typ resolved static types keyword parameters on representative AST nodes. The full specification of a correct AST nodes is a boolean predicate in lang::analysis::M3.

added AST nodes for all new Java features up to JLS13, including the Java 9 module system
added AST nodes for the new Java features in the preview for JLS14
added M3 support for source and binary models of the Java 9 module system
added back modifiers AST nodes everywhere
made sure annotations and modifers that can be typed in different orders for methods and variabele and parameter declarations end up in the AST in the order of appearance in the source code
added back type parameter AST nodes everywhere
added AST nodes for simple and qualified names everywhere (for concrete syntax support)
tested new AST nodes and M3 on vallang, rascal and jsoup
validated all AST nodes on the AST correctness specification in analysis::m3::AST and fixed all the bugs
documented all M3 relations
documented the logical location scheme's of Java M3 and Java AST.
resolve builtin array .length field accesses to java+arrayLength:///TypeName[] to fix a number of inconsistencies around these hidden fields.

…port

…tem, in particular the AST nodes that may occur in module-info.java files. Still to be tested

…ng as only set and relation fields are being used

…parameters to a node without having to repeat the existing ones

…are still str that should be Expression in my mind

…rmat. Also documented analysis::m3::AST better

…on and more uniformity

…Java\'s AST format, and fixed the mapping to accomodate

…nd regenerated the reference files

jurgenvinju · 2024-05-08T10:02:30Z

I think this is ready for review, but I should want that the number of changes is quite enormous.

I have compared the new ASTs to the old ASTs and the new M3 models to the old M3 models meticulously (tuple by tuple, node by node), until I had fixed all new bugs in the AST and M3 builders, and confirmed that existing bugs have been solved.
The changes in the definition of ASTs reflect what users have to do to get up-to-date, M3 models have hardly changed.
Some TODO's are left, such as adding implicitDeclarations to the M3 model, and including lambda's in a way in the M3 model such that a complete call graph construction algorithm is possible with only M3 as source data. However, that would add more noise to the current PR so I'm pushing that to the future.

jurgenvinju · 2024-05-08T10:11:02Z

Am still working to run the astNodeSpecification checker over all the ASTs. Errors keep popping up. Those are going to be minor changes of list.insert to list.append and such.

…ist to make debugging unresolved declarations easier

jurgenvinju · 2024-05-08T11:13:24Z

Ok all the Java ASTs now pass all of the AST specification. The M3 specification (made based on the C++ support) is not robust enough yet to be able to be used as a quality gate.

DavyLandman

This was a huge refactoring, the java was unreviewable so I:

reviewed the rascal change, they look much improved, with 1 note about defaulting to java 13
looked at a few things in java that would worry me, but all is good there.
ran it on my machine to test it would still work on windows (as there is some path stuff in here as well).

Looks good to me, great improvement.

DavyLandman · 2024-05-08T12:02:22Z

src/org/rascalmpl/interpreter/TestEvaluator.java

@@ -87,6 +87,10 @@ public static int readIntTag(AbstractFunction test, String key, int defaultVal)

    private void runTests(ModuleEnvironment env, List<AbstractFunction> tests) {
        testResultListener.start(env.getName(), tests.size());
+        if (tests.size() <= 0) {
+            return;


should this also have a testResultListener.done(); call?

definitely! good catch @DavyLandman . I removed the superfluous call to testResultListener.start instead. This will help a lot with the reporting in the IDE and github actions.

DavyLandman · 2024-05-08T12:11:44Z

src/org/rascalmpl/library/analysis/m3/AST.rsc

+| `src`      | the exact source location of the declaration in a source file |
+| `decl`     | the resolved fully qualified name of the artefact that is being declared here |
+| `typ`      | a symbolic representation of the static type of the declared artefact here (not the syntax of the type) |
+} 


👍 thanks for also writing all this documentation.

DavyLandman · 2024-05-08T12:15:21Z

src/org/rascalmpl/library/analysis/m3/Core.rsc

 @memo
 M3 composeM3(loc id, set[M3] models) {
 	M3 comp = m3(id);
+  map[M3, map[str,value]] fields = (m:getKeywordParameters(m) | m <- models);
+  set[str] keys                  = {*domain(getKeywordParameters(m)) | m <- models };


nit: this could be written as a comprehension over fields to avoid extra calls to getKeywordParameters.

The current solution is faster, contra-intuitively. getKeywordParameters does not require a lookup in the hash trie, since this map is attached directly to the constructor. Especially for large sets of models this choice can save a few instructions per model.

DavyLandman · 2024-05-08T12:23:06Z

src/org/rascalmpl/library/lang/java/m3/AST.rsc

@@ -199,7 +361,7 @@ set[loc] findRoots(set[loc] folders) {
 @description{
 Wrapper around ((createAstsFromFiles)) to call it on a single file.
 }
-public Declaration createAstFromFile(loc file, bool collectBindings, bool errorRecovery = false, list[loc] sourcePath = [], list[loc] classPath = [], str javaVersion = "1.7") {
+public Declaration createAstFromFile(loc file, bool collectBindings, bool errorRecovery = false, list[loc] sourcePath = [], list[loc] classPath = [], Language javaVersion = JLS13()) {


I like this enum 👍 but why default to java 13? I would expect one of the LTS versions like 8/11/17/21. And what happens if in the future we also add support for newer versions, will the default change?

(same for the others in this file)

To make sure the porting to newer versions is always as small as possible, I would advice users to always go with the highest support we currently have, if possible. That way when we release support for newer versions, upgrading will always be between the lowest number of versions.

So yes, the default will always be the highest we have.

If there are other considerations, I'd be happy to listen.

I'm thinking of backwards compatibility, if I have code that's not setting this language level (because of the default, I'm not forced to commit to a specific version). Hypothetical say we release this as 11. And I'm analyzing code with it that has some variables named 'yield' in there. Now we do some new work, and add Java 17 supported, and we switch the default to java 17. For that new version of rascal, my analysis failed, without me changing my code. (Since yield is now a keyword)

So either we freeze the default, or we make it an required parameter, and have a special 'highestSupported' kind of ADT member, for people that just want the lastest and greatest.

I understand the argument but I don't agree setting an old version and fix it is a good solution for most cases. It is for the yield example; but it would mean we now have to lock on Java 8, as this was the version we have supported up until now (and partially at that). Most students would have to set the version parameter to 12 or 13.

if people need backwards compatibility for their actual compiler, they must set the version level explicitly, otherwise all compilers always default to their current/latest version. So to get this 100% right, we probably should add some code to the Maven support to extract that configuration parameter from the XML. But otherwise I think it is safe enough to follow the normal Java compiler behavior.

highestSupported constructor is not right since M3 models need to be self-descriptive. The goal of the version parameter is not only to configure the compiler the right way, but also to log the chosen version as it is. The Language ADT is a specific ADT to encode language names and their versions from analysis::m3::Core.

How do you feel about making it a positional argument?

And my 'latestVersion' would just be meant as a parameter, not as a valid value in an m3 model.

DavyLandman · 2024-05-08T12:41:05Z

src/org/rascalmpl/library/lang/java/m3/internal/M3LocationUtil.java

-        catch (URISyntaxException e) {
-            throw new RuntimeException("Extending path error with URIUtil", e);
-        }
+        return URIUtil.getChildLocation(uri, path);


DavyLandman · 2024-05-08T12:42:31Z

src/org/rascalmpl/library/lang/rascal/tests/library/lang/java/m3/BasicM3Tests.rsc

-
-public loc unpackExampleProject(str name, loc projectZip) {
+@synopsis{Unpack a test source zip in a temporary folder for downstream analysis.}
+private loc unpackExampleProject(str name, loc projectZip) {
    targetRoot = |tmp:///<name>|;
    sourceRoot = projectZip[scheme = "jar+<projectZip.scheme>"][path = projectZip.path + "!/"];


I think that we can actually use zip+ now instead of jar+? At the time zip might not have been available?

Yes I was also thinking that :-) But this private function is used for both unpacking jars and zips, and the jar unzipper does something special with the META-INF folder.

…machines

codecov · 2024-05-08T17:31:30Z

Codecov Report

Attention: Patch coverage is 56.46186% with 411 lines in your changes are missing coverage. Please review.

Project coverage is 49%. Comparing base (6bb94a2) to head (efa6401).

Files	Patch %	Lines
...pl/library/lang/java/m3/internal/ASTConverter.java	58%	198 Missing and 25 partials ⚠️
...pl/library/lang/java/m3/internal/JarConverter.java	19%	52 Missing and 3 partials ⚠️
...library/lang/java/m3/internal/SourceConverter.java	32%	53 Missing and 2 partials ⚠️
...ary/lang/java/m3/internal/EclipseJavaCompiler.java	54%	24 Missing and 5 partials ⚠️
...ibrary/lang/java/m3/internal/BindingsResolver.java	66%	9 Missing and 8 partials ⚠️
...y/lang/java/m3/internal/JavaToRascalConverter.java	67%	8 Missing and 6 partials ⚠️
...library/lang/java/m3/internal/ASMNodeResolver.java	89%	4 Missing and 3 partials ⚠️
...mpl/library/lang/java/m3/internal/M3Converter.java	75%	6 Missing ⚠️
src/org/rascalmpl/library/Prelude.java	0%	3 Missing ⚠️
...org/rascalmpl/repl/TerminalProgressBarMonitor.java	0%	1 Missing ⚠️
... and 1 more

Additional details and impacted files

@@           Coverage Diff            @@
##              main   #1936    +/-   ##
========================================
- Coverage       49%     49%    -1%     
- Complexity    6193    6215    +22     
========================================
  Files          663     662     -1     
  Lines        58946   59432   +486     
  Branches      8575    8637    +62     
========================================
+ Hits         29026   29178   +152     
- Misses       27724   28018   +294     
- Partials      2196    2236    +40

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

jurgenvinju · 2024-05-08T17:40:38Z

Thanks @DavyLandman As soon as the tests run on the server, I'll merge.

jurgenvinju · 2024-05-08T17:52:26Z

Aha! The type-checker is now forcing us to update JavaToObjectFlow :-) good catch.

…oObjectFlow

… all new Java features

…er classes, with and without type qualifiers

jurgenvinju added 30 commits April 15, 2024 16:40

added switch expressions to Java M3 AST support

d9dcfac

starting to weed out all the different language levels

f42c97e

revamped JLS level selection parameter and added preview features sup…

a30fa23

…port

fixed switchCaseRule

98c5795

added method reference expression

7ebcb9c

added expression method reference

bff9a3d

added intersectiontype

fd08a80

added missing @OverRide annotations to visit methods

2302099

added super.MethodReference to AST

575a5d3

minor

ab8944b

refactored IValueList usages to IListWriter and removed IValueList

c74ebb6

added initial implementation of the AST nodes for the Java module sys…

8225b23

…tem, in particular the AST nodes that may occur in module-info.java files. Still to be tested

added exports directive

2c0f68b

radically improved M3 documentation

c20ad42

typos

a26aad6

composeM3 is now _generic_ for all M3 models for all languages, as lo…

cdf7dd6

…ng as only set and relation fields are being used

added missing mergeKeywordParameters function for adding new keyword …

665051e

…parameters to a node without having to repeat the existing ones

documented TypeSymbol better

ce40c5f

using the Language concept from M3 to encode language name and version

6879945

fixed initial problems in module spec parser, but some of the fields …

35b5340

…are still str that should be Expression in my mind

fixed all name references in ASTs of module declarations

fb138b0

fixed names in method references

14ca354

layout

69c90da

added module as a type for Java and fixed naming issues in the AST fo…

61cd581

…rmat. Also documented analysis::m3::AST better

fixed field access

4a11698

rationalized all shortcuts of str to Expression for better resoluti…

92f104d

…on and more uniformity

removed illegal bool value from AST definition

06236ec

added all missing AST nodes for modifiers and annotations aback into …

e72f1b6

…Java\'s AST format, and fixed the mapping to accomodate

minor cleanup

0babe3e

jar converter now records java version for each class file

f00d64a

jurgenvinju added 6 commits May 8, 2024 10:58

regenerated ASTs after superfluous merge from main

11f546c

checked differences in snakes and ladders regression tests manually a…

860cd30

…nd regenerated the reference files

regenerated M3 binaries after minor update

6f9bac7

Merge branch 'main' into m3-jls-13-and-higher

3637eb5

Merge branch 'main' into m3-jls-13-and-higher

5f8b840

added .length array field back to fieldAccess

0e3b120

jurgenvinju requested a review from DavyLandman May 8, 2024 09:58

jurgenvinju added 5 commits May 8, 2024 12:50

java M3 and AST extraction functions now check if path elements do ex…

a46a36e

…ist to make debugging unresolved declarations easier

JUnit ASTs now pass the full AST specification

3664287

Snakes and ladders ASTs satisfy the full AST spec

bc7e238

java root package bug fixed

915c3be

Java root package bug also fixed for junit4 reference model

8fa72c1

improved the M3 correctness specification

f0c5d3e

DavyLandman approved these changes May 8, 2024

View reviewed changes

jurgenvinju added 2 commits May 8, 2024 19:25

organize imports

ba936da

jar extraction must be from a location that is the same on all build …

0d32f78

…machines

jurgenvinju added 2 commits May 8, 2024 19:32

removed superfluous testResultlistener.start and refactored a bit

8361385

fixed a typo

2fd49c8

updated newObject matches and constructions with type arguments

467b914

jurgenvinju added 3 commits May 9, 2024 13:15

fixed a big number of downstream updates to the AST patterns in JavaT…

3943a9d

…oObjectFlow

JavaToObjectFlow is correct again, statically, but not yet supporting…

c6ee41c

… all new Java features

added new missing cases for calling constructors and methods from sup…

efa6401

…er classes, with and without type qualifiers

jurgenvinju merged commit 2690e1f into main May 9, 2024
7 checks passed

jurgenvinju deleted the m3-jls-13-and-higher branch May 9, 2024 13:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Java JLS 13 and JLS14 preview in M3 models and Java ASTs; also make ASTs jump through the correctness specification in lang::analysis::AST #1936

Support Java JLS 13 and JLS14 preview in M3 models and Java ASTs; also make ASTs jump through the correctness specification in lang::analysis::AST #1936

jurgenvinju commented Apr 16, 2024 •

edited

jurgenvinju commented May 8, 2024

jurgenvinju commented May 8, 2024

jurgenvinju commented May 8, 2024

DavyLandman left a comment

DavyLandman May 8, 2024

jurgenvinju May 8, 2024

DavyLandman May 8, 2024

DavyLandman May 8, 2024

jurgenvinju May 8, 2024

DavyLandman May 8, 2024

jurgenvinju May 8, 2024

DavyLandman May 8, 2024

jurgenvinju May 9, 2024

DavyLandman May 9, 2024

DavyLandman May 8, 2024

DavyLandman May 8, 2024

jurgenvinju May 8, 2024

codecov bot commented May 8, 2024 •

edited

jurgenvinju commented May 8, 2024

jurgenvinju commented May 8, 2024

Support Java JLS 13 and JLS14 preview in M3 models and Java ASTs; also make ASTs jump through the correctness specification in lang::analysis::AST #1936

Support Java JLS 13 and JLS14 preview in M3 models and Java ASTs; also make ASTs jump through the correctness specification in lang::analysis::AST #1936

Conversation

jurgenvinju commented Apr 16, 2024 • edited

jurgenvinju commented May 8, 2024

jurgenvinju commented May 8, 2024

jurgenvinju commented May 8, 2024

DavyLandman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented May 8, 2024 • edited

Codecov Report

jurgenvinju commented May 8, 2024

jurgenvinju commented May 8, 2024

jurgenvinju commented Apr 16, 2024 •

edited

codecov bot commented May 8, 2024 •

edited