Make "PrimitiveParameterizedClass.default" a poly expression. #369

jespersm · 2021-03-19T00:01:50Z

Make .default a separate node type in the parser.

Issue

JDK-8211914: [lworld] Javac should support type inference for default value creation

Note: The Linux x86 builds in GitHub actions seem to fail with something completely unrelated to these changes.

Progress

Change must not contain extraneous whitespace

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.java.net/valhalla pull/369/head:pull/369
$ git checkout pull/369

Update a local copy of the PR:
$ git checkout pull/369
$ git pull https://git.openjdk.java.net/valhalla pull/369/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 369

View PR using the GUI difftool:
$ git pr show -t 369

Using diff file

Download this PR as a diff file:
https://git.openjdk.java.net/valhalla/pull/369.diff

bridgekeeper · 2021-03-19T00:02:26Z

👋 Welcome back jespersm! A progress list of the required criteria for merging this PR into lworld will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2021-03-19T07:36:03Z

@jespersm This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

Make "PrimitiveParameterizedClass.default" a poly expression.

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 20 new commits pushed to the lworld branch:

ad0491e: 8271017: [lworld] Implement withfield changes from JDK-8269408 on AArch64
8c11290: 8271154: [lworld] CDS LoaderConstraintsTest fails after injection of IdentityObject
a4066d7: 8244231: [lworld] Add support for ref-default and val-default inline classes.
02584ab: 8269756: [lworld] Add tests for invalid withfield operands
01cab82: 8271141: [lworld] Remove unused jvaluetype q member in jvalue
64ad8d0: 8271113: [lworld] java/lang/constant/MethodHandleDescTest.java fails after JDK-8247376
6bc5ea9: 8247376: [lworld] Constant API support for primitive classes
4d42fe1: 8271025: [lworld] vmTestbase/jit/t/* tests fail after JDK-8237073
31d3bb0: 8269408: [lworld] Withfield and field resolution update
b45994a: 8268946: [lworld] Re-implement JDK-8266015 after the merge
... and 10 more: https://git.openjdk.java.net/valhalla/compare/b3b7fb2bf96b40fd4636a699e150b8ab5c2d8610...lworld

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

As you do not have Committer status in this project an existing Committer must agree to sponsor your change. Possible candidates are the reviewers of this PR (@mcimadamore) but any other Committer may sponsor as well.

➡️ To flag this PR as ready for integration with the above commit message, type /integrate in a new comment. (Afterwards, your sponsor types /sponsor in a new comment to perform the integration).

mlbridge · 2021-03-19T07:39:29Z

Webrevs

mcimadamore

Nice work - it's a very nice cleanup of the code base! I like how the code that was previously scattered in several places now can just move where it belongs.

Functionality-wise, I'm not sure this supports poly expressions yet - e.g. I see that you set the polyKind tag correctly, but I'm afraid that this along is not sufficient to support stuff like:

List<String> ls = List.default;

or:

void m(List<String> ls) { ... }

m(List.default)

I think you might be led to believe that the current impl is doing the right thing - but my feeling is that javac might be tricking you: if the expected type is e.g. List<String> but the qualifier of the default expression is List, the compiler will check (as per checkId) that List <: List - which is true by subtyping conversion. I don't think there is any inference going on, in other words. You might try debugging and looking at the type of the expression that comes out of checkId.

Anyway, even if this works and I did miss something, I don't see how this approach can scale to default expression passed as method arguments - given that in that case there is no pt(). To add support for that you need to add another case/node in ArgumentAttr (see what's been done for anonymous inner classes - this should be simpler as there's no constructor call).

In other words, you want Attr.visitApply to treat your default expression method argument in a special way - by creating what we call a "deferred type" - that is a type that is not fully inferred until after overload resolution (when you DO know the pt()).

One possibility would be to push this change as is - as the refactoring part is really good - and maybe issue raw type warning when generic types are involved, or when default expressions are used in method context. And then in another, separate PR we can refine what happens in truly poly cases.

mcimadamore · 2021-03-19T12:25:36Z

So, I debugged what happens in case where we have

List<String> ls = List.default

What I missed is that we create a variable symbol for default which has the right type (from the expected type) - so the expression is correctly typed as List<String>. But this seems to happen "just because" the compiler detects that the expected type has same base class as the default expression type. Then, the check performed by checkId is always vacuously satisfied, given that we're passing in a synthetic variable whose type is the same as the expected one.

Counter example:

interface Foo<X extends Number> extends List<X> { }
...
List<String> ls = Foo.default; // still ok, type is `Foo`

This example is more complex, for two reasons:

in this case, Foo is a sub-interface, so the trick of saying "if base class is the same just use the expected type" doesn't work. In fact the compiler types this with Foo - which is a raw type
in this particular instance, the code should not even be allowed - because the constraint on the Foo type parameter are incompatible with those of the expected type List (X must extend Number).

All this stuff needs to work correctly if we want to claim that default expressions are true poly expressions.

mlbridge · 2021-03-19T15:36:39Z

Mailing list message from Jesper Steen M��ller on valhalla-dev:

On 19 Mar 2021, at 13.28, Maurizio Cimadamore <mcimadamore at openjdk.java.net> wrote:

On Fri, 19 Mar 2021 10:57:22 GMT, Maurizio Cimadamore <mcimadamore at openjdk.org <mailto:mcimadamore at openjdk.org>> wrote:

One possibility would be to push this change as is - as the refactoring part is really good - and maybe issue raw type warning when generic types are involved, or when default expressions are used in method context. And then in another, separate PR we can refine what happens in truly poly cases.

Yes, that might make sense. I?ll split the PR up into the two cases.

So, I debugged what happens in case where we have

`List<String> ls = List.default`

What I missed is that we create a variable symbol for `default` which has the right type (from the expected type) - so the expression is correctly typed as `List<String>`. But this seems to happen "just because" the compiler detects that the expected type has same base class as the default expression type. Then, the check performed by `checkId` is always vacuously satisfied, given that we're passing in a synthetic variable whose type is the same as the expected one.

Counter example:

interface Foo<X extends Number> extends List<X> { }
...
List<String> ls = Foo.default; // still ok, type is `Foo`

This example is more complex, for two reasons:

* in this case, Foo is a sub-interface, so the trick of saying "if base class is the same just use the expected type" doesn't work. In fact the compiler types this with `Foo` - which is a raw type
* in this particular instance, the code should not even be allowed - because the constraint on the `Foo` type parameter are incompatible with those of the expected type List<String> (X must extend Number).

All this stuff needs to work correctly if we want to claim that default expressions are true poly expressions.

I get it; this needs to be checked before blindly promoting to the expected type.
Thank you for the good counter example.

As for method overload, I?ve come up with this example, which seems to demonstrate the need for the "full poly treatment" using deferred types.

class Whatever {

static int foo\(Consumer\<Integer> c\) \{ return 1\; \}
static String foo\(BiConsumer\<Integer\, Integer> c\) \{ return \"x\"\; \}

static primitive class Quux\<T extends Number> implements Consumer\<Integer>\, BiConsumer\<T\, T> \{
	public void accept\(T t\, T u\) \{ \}
	public void accept\(Integer t\) \{ \}
\}

public static void ambiguous\(\) \{
	var result \= foo\(Quux\.default\)\; \/\/ foo here is ambiguous\, just like non\-primitive new Quux\<>\(\) is
\}

}

-Jesper

jespersm · 2021-03-20T11:56:53Z

@mcimadamore : I've opened https://bugs.openjdk.java.net/browse/JDK-8263900 for the refactoring alone. I'm not totally confident that all the checks in Attr are syntactically possible, and should likely be turned into assertions.

So this PR is temporarily on hold, awaiting merge of #370

… synthetic method.

jespersm · 2021-07-04T19:52:16Z

/test tier1

openjdk · 2021-07-04T19:53:44Z

@jespersm you need to get approval to run the tests in tier1 for commits up until e9289cf

sadayapalam · 2021-07-19T08:22:56Z

Hello Jesper, this latest patch is problematic. it does not properly treat Foo.default as a poly expression. As Maurizio's earlier review comments points out:

I don't see how this approach can scale to default expression passed as method arguments - given that in that case there is no pt(). To add support for that you need to add another case/node in ArgumentAttr (see what's been done for anonymous inner classes - this should be simpler as there's no constructor call).

In other words, you want Attr.visitApply to treat your default expression method argument in a special way - by creating what we call a "deferred type" - that is a type that is not fully inferred until after overload resolution (when you DO know the pt()).

I don't see the relevant code changes in your change set - i.e there is no changes in ArgumentAttr {} - I would have expected to see a visitDefaultValue method there.

As Maurizio further pointed out earlier, in Attr.visitApply, right after arguments are attributed, the type of Foo.default as a method argument should be a deferred type . If I debug the uploaded patch, the type of Foo.default is actually Foo<T>

The problem is masked by there being only negative tests in your patch.

If I compile the following:

public primitive class X<T> {

    
    static void m(X<String> ls) { }

    public static void main(String [] args) {
        m(new X<>()); // OK
        m(X<String>.default); // OK.
        m(X.default);   // OK, BUT get an error here!
    }

}

at the line with the .default usage I get:

 error: incompatible types: X<T> cannot be converted to X<String>
        m(X.default);
           ^
  where T is a type-variable:
    T extends Object declared in class X
Note: Some messages have been simplified; recompile with -Xdiags:verbose to get full output
1 error

This shows that there is no real inference happening there.

(when a poly type argument is attributed without a target type in context, it should be typed to be DeferredType, when considered against overload candidates the positional parameter type becomes the pt() and the deferred type can now be typed to a concrete type)

sadayapalam · 2021-07-19T08:26:24Z

A way to see this in more detail is to debug through the Attr.visitApply call corresponding to the method invocation

m(new X<>(), X.default);

in this test case:

public primitive class X<T> {

    
    static void m(X<String> ls, X<String> ls2) { }

    public static void main(String [] args) {
        m(new X<>(), X.default);
    }

}

You will see that first argument is typed to be DeferredType and second one incorrectly to be X<T>

jespersm · 2021-07-25T21:52:14Z

Hey @sadayapalam and @mcimadamore, I think this patch is ready for review again.

I've not merged lworld (to keep a clearer history), but it still applies cleanly to the current lworld tip, so it should merge OK.

sadayapalam · 2021-08-02T11:41:34Z

src/jdk.compiler/share/classes/com/sun/tools/javac/tree/TreeInfo.java

+        JCTypeApply applyTree = TreeInfo.getTypeApplication(tree.clazz);
+        if (applyTree != null) {
+            return applyTree.arguments.isEmpty();
+        } else {


The return statement is not reachable and so the block is not useful. This is because, Foo<>.default is a syntax error. Given a primitive class Foo we will only support Foo.default involving implicit diamond like inference and will not support Foo<>.default which is what this code is handling.

(I think I meant to say return applyTree.arguments.isEmpty(); will never be true.)

sadayapalam · 2021-08-02T11:41:34Z

src/jdk.compiler/share/classes/com/sun/tools/javac/tree/TreeInfo.java

+        } else {
+            // No type arguments before .default - Consider if the type is generic or not
+            return clazztype == null || clazztype.tsym.type.isParameterized();
+        }


This method is problematic - This will always treat Foo.default as a poly expression in an invocation context even when the concerned primitive type is not a generic class. For instance the Foo.default expression in the following snippet becomes a poly expression when it is a standalone expression in reality:

primitive class X { static void foo(Object o) { } public static void main(String [] args) { foo(X.default); } }

sadayapalam · 2021-08-02T11:41:34Z

src/jdk.compiler/share/classes/com/sun/tools/javac/tree/TreeInfo.java

+        switch(tree.getTag()) {
+            case TYPEAPPLY: return (JCTypeApply)tree;
+            case NEWCLASS: return getTypeApplication(((JCNewClass)tree).clazz);
+            case ANNOTATED_TYPE: return getTypeApplication(((JCAnnotatedType)tree).underlyingType);


I think there is a copy + paste problem. We can never see a NEWCLASS here. The LHS of .default is just a type node and cannot be an rvalue expression ==> new X().default is illegal.

I think perhaps what we really need to do here is to attribute the clazz node of JCDefaultValue first - perhaps by first making a copy of the node and then see if it is a generic class but lacks type arguments. Such as this:

@Override public void visitDefaultValue(JCDefaultValue that) { JCExpression copy = new TreeCopier<Void>(attr.make).copy(that.clazz); attr.attribType(copy, env); if (copy.type.getTypeArguments().isEmpty() && copy.type.tsym.type.isParameterized()) { processArg(that, speculativeTree -> new ResolvedDefaultType(that, env, speculativeTree)); } else { //not a poly expression, just call Attr setResult(that, attr.attribTree(that, env, attr.unknownExprInfo)); } }

sadayapalam · 2021-08-03T09:20:50Z

src/jdk.compiler/share/classes/com/sun/tools/javac/comp/ArgumentAttr.java

+     * Argument type for default values.
+     */
+    class ResolvedDefaultType extends ResolvedMemberType<JCDefaultValue> {
+


I am surprised that ResolvedDefaultType extends ResolvedMemberType rather than ArgumentType directly. While the Inference behavior could be indirectly modelled it as if the class has a generic factory method named 'default':

<T> static Foo<T> default() { ... }

this implementation detail could be pushed deep down and not manifest here and in elsewhere (setPolyKind())

sadayapalam · 2021-08-03T09:24:43Z

src/jdk.compiler/share/classes/com/sun/tools/javac/comp/Attr.java

+                enclosingContext.report(tree.clazz,
+                        diags.fragment(Fragments.CantApplyDiamond1(Fragments.Diamond(tsym), details)));
+            }
+        };


I don't see a test that triggers this diagnostic.

sadayapalam · 2021-08-03T09:29:49Z

src/jdk.compiler/share/classes/com/sun/tools/javac/tree/TreeInfo.java

@@ -294,6 +315,9 @@ public static void setPolyKind(JCTree tree, PolyKind pkind) {
            case NEWCLASS:
                ((JCNewClass)tree).polyKind = pkind;
                break;
+            case DEFAULT_VALUE:
+                ((JCDefaultValue)tree).polyKind = pkind;
+                break;
            case REFERENCE:


This is surprising. I would expected polyKind to be set in a similar fashion to switch and conditional expressions.

sadayapalam · 2021-08-03T09:34:48Z

Hi Jesper, I have left some comments - but I will need to make a couple of more passes to fully understand the implementation as this is a fairly involved and complex task. I will actually have to tinker with the version you have posted to comprehend it - Could you give me some time - I will play around with it and propose a version that addresses the problems I have outlined and also includes further tweaks that could simplify the implementation more ? Then we can study it, finalize it and then ask Maurizio for a final review once both of us are satisfied.

jespersm · 2021-08-17T23:51:16Z

@sadayapalam : I can take another go at this one to adress the review comments above, which may make a second review pass easier.

sadayapalam · 2021-08-23T08:48:28Z

@sadayapalam : I can take another go at this one to adress the review comments above, which may make a second review pass easier.

Sure, this is recognized to be a tricky area, so it is normal to have iterate over it multiple times incrementally improving it. As long as this is had in mind, by all means it helps to have the comments made so far addressed. Thanks!

openjdk bot added ready rfr labels Mar 19, 2021

mcimadamore reviewed Mar 19, 2021

View changes

jespersm closed this Mar 20, 2021

jespersm reopened this Mar 20, 2021

[lworld] Handling poly-inference of missing arguments by simulating a…

e9289cf

… synthetic method.

jespersm force-pushed the default-polytype-8211914 branch from 0f969f6 to e9289cf Compare Jul 4, 2021

openjdk bot added the test-request label Jul 4, 2021

Handle default values as polytype method arguments.

0c08826

openjdk bot removed ready rfr labels Jul 23, 2021

jespersm added 3 commits Jul 25, 2021

Properly propagate reference projections for defaults, and add tests.

8ce4dce

Whitespace fixes

131b9c7

Whitespace

f4073c4

openjdk bot added ready rfr labels Jul 25, 2021

sadayapalam suggested changes Aug 2, 2021

View changes

sadayapalam reviewed Aug 3, 2021

View changes

Make "PrimitiveParameterizedClass.default" a poly expression. #369

Make "PrimitiveParameterizedClass.default" a poly expression. #369

jespersm commented Mar 19, 2021 •

edited by openjdk[bot] bot

bridgekeeper bot commented Mar 19, 2021

openjdk bot commented Mar 19, 2021 •

edited

mlbridge bot commented Mar 19, 2021 •

edited

mcimadamore left a comment •

edited

mcimadamore commented Mar 19, 2021

mlbridge bot commented Mar 19, 2021

jespersm commented Mar 20, 2021

jespersm commented Jul 4, 2021

openjdk bot commented Jul 4, 2021

sadayapalam commented Jul 19, 2021 •

edited

sadayapalam commented Jul 19, 2021 •

edited

jespersm commented Jul 25, 2021

sadayapalam Aug 2, 2021

sadayapalam Aug 2, 2021

sadayapalam Aug 2, 2021

sadayapalam Aug 2, 2021

sadayapalam Aug 2, 2021

sadayapalam Aug 3, 2021

sadayapalam Aug 3, 2021

sadayapalam Aug 3, 2021

sadayapalam commented Aug 3, 2021

jespersm commented Aug 17, 2021

sadayapalam commented Aug 23, 2021

Make "PrimitiveParameterizedClass.default" a poly expression. #369

Are you sure you want to change the base?

Make "PrimitiveParameterizedClass.default" a poly expression. #369

Conversation

jespersm commented Mar 19, 2021 • edited by openjdk[bot] bot

Issue

Progress

Reviewing

bridgekeeper bot commented Mar 19, 2021

openjdk bot commented Mar 19, 2021 • edited

mlbridge bot commented Mar 19, 2021 • edited

Webrevs

mcimadamore left a comment • edited

mcimadamore commented Mar 19, 2021

mlbridge bot commented Mar 19, 2021

jespersm commented Mar 20, 2021

jespersm commented Jul 4, 2021

openjdk bot commented Jul 4, 2021

sadayapalam commented Jul 19, 2021 • edited

sadayapalam commented Jul 19, 2021 • edited

jespersm commented Jul 25, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sadayapalam commented Aug 3, 2021

jespersm commented Aug 17, 2021

sadayapalam commented Aug 23, 2021

jespersm commented Mar 19, 2021 •

edited by openjdk[bot] bot

openjdk bot commented Mar 19, 2021 •

edited

mlbridge bot commented Mar 19, 2021 •

edited

mcimadamore left a comment •

edited

sadayapalam commented Jul 19, 2021 •

edited

sadayapalam commented Jul 19, 2021 •

edited