Method reference support

[rmuir]

This adds support for method references (Class::virtualMethod, Class::staticMethod, Class::new).

The syntax for this was added in #18578, as a stub that throws UnsupportedOperationException. This adds the logic. It is useful because these are the simplest lambda types (non-capturing, no desugaring needed), but can make the functional apis usable to users in many ways, based on functions that already exist. For example plugging in different functions for comparators, using streams apis, etc.

There are two cases: static and dynamic. In the static case we call LambdaMetaFactory from bytecode, since we know everything we need at compile time. This is similar to javac's case, just infer interface from the method's argument type.

In the dynamic case, we don't yet know enough to do anything interesting, so we just push the reference on the stack as a string, and mark the argument as a function in a bitset passed as a static bootstrap param. Once types are known at runtime, then we do the same stuff as the static case, just with java code, and replace the "placeholders" using MethodHandles.filterArguments.

Array constructor references (foo[]::new) are not yet supported. We should do that as a followup if we want to support them, as they are a little more complicated (require us to write bridge methods). For cases like this we will need to "expand" our recipe param to be something better than a bitset, so that pointers to our own-written functions can work too.

There are also some random cleanups, and additional strictness to the whitelist: enforcing that we always whitelist covariant and generic overrides. Some of this is kinda unrelated, but happens to be here.

The PR is not yet ready. In particular it is super-messy, and there needs to be a lot of cleanup and refactoring before we should push it. For example, FunctionRef class should maybe work with MethodType rather than ASM types, and Definition should do the functional interface computation stuff (no reflection api needs to be used). The two different apis for the two ways we call this thing (Handle/MethodHandle and Type/MethodType) make things a mess at the moment. Needs help :)

@uschindler i know you are at buzzwords, but if you have some time to look, would be great to get your thoughts on some of this.

[uschindler]

I will check and commit changes when I come back from Berlin buzzwords. 😉

Am 6. Juni 2016 15:49:33 MESZ, schrieb Robert Muir notifications@github.com:

This adds support for method references (Class::virtualMethod,
Class::staticMethod, Class::new).

The syntax for this was added in #18578, as a stub that throws
UnsupportedOperationException. This adds the logic. It is useful
because these are the simplest lambda types (non-capturing, no
desugaring needed), but can make the functional apis usable to users in
many ways, based on functions that already exist. For example plugging
in different functions for comparators, using streams apis, etc.

There are two cases: static and dynamic. In the static case we call
LambdaMetaFactory from bytecode, since we know everything we need at
compile time. This is similar to javac's case, just infer interface
from the method's argument type.

In the dynamic case, we don't yet know enough to do anything
interesting, so we just push the reference on the stack as a string,
and mark the argument as a function in a bitset passed as a static
bootstrap param. Once types are known at runtime, then we do the same
stuff as the static case, just with java code, and replace the
"placeholders" using MethodHandles.filterArguments.

Array constructor references (foo[]::new) are not yet supported. We
should do that as a followup if we want to support them, as they are a
little more complicated (require us to write bridge methods). For cases
like this we will need to "expand" our recipe param to be something
better than a bitset, so that pointers to our own-written functions can
work too.

There are also some random cleanups, and additional strictness to the
whitelist: enforcing that we always whitelist covariant and generic
overrides. Some of this is kinda unrelated, but happens to be here.

The PR is not yet ready. In particular it is super-messy, and there
needs to be a lot of cleanup and refactoring before we should push it.
For example, FunctionRef class should maybe work with MethodType rather
than ASM types, and Definition should do the functional interface
computation stuff (no reflection api needs to be used). The two
different apis for the two ways we call this thing (Handle/MethodHandle
and Type/MethodType) make things a mess at the moment. Needs help :)

@uschindler i know you are at buzzwords, but if you have some time to
look, would be great to get your thoughts on some of this.
You can view, comment on, or merge this pull request online at:

#18748

-- Commit Summary --

• initial messy impl of painless method references
• don't do a no-op filter, that was just for testing

-- File Changes --

M modules/lang-painless/src/main/antlr/PainlessParser.g4 (2)
M
modules/lang-painless/src/main/java/org/elasticsearch/painless/Def.java
(137)
M
modules/lang-painless/src/main/java/org/elasticsearch/painless/DefBootstrap.java
(16)
M
modules/lang-painless/src/main/java/org/elasticsearch/painless/Definition.java
(79)
A
modules/lang-painless/src/main/java/org/elasticsearch/painless/FunctionRef.java
(126)
M
modules/lang-painless/src/main/java/org/elasticsearch/painless/PainlessScriptEngineService.java
(1)
M
modules/lang-painless/src/main/java/org/elasticsearch/painless/ScriptImpl.java
(2)
M
modules/lang-painless/src/main/java/org/elasticsearch/painless/WriterConstants.java
(14)
M
modules/lang-painless/src/main/java/org/elasticsearch/painless/antlr/PainlessParser.java
(221)
M
modules/lang-painless/src/main/java/org/elasticsearch/painless/antlr/Walker.java
(10)
M
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/EFunctionRef.java
(41)
M
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/LCall.java
(2)
M
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/LDefArray.java
(4)
M
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/LDefCall.java
(14)
M
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/LDefField.java
(4)
M
modules/lang-painless/src/main/java/org/elasticsearch/painless/node/LNewObj.java
(6)
M
modules/lang-painless/src/main/resources/org/elasticsearch/painless/java.lang.txt
(14)
M
modules/lang-painless/src/main/resources/org/elasticsearch/painless/java.math.txt
(2)
M
modules/lang-painless/src/main/resources/org/elasticsearch/painless/java.text.txt
(1)
M
modules/lang-painless/src/main/resources/org/elasticsearch/painless/java.time.chrono.txt
(53)
M
modules/lang-painless/src/main/resources/org/elasticsearch/painless/java.time.txt
(32)
M
modules/lang-painless/src/main/resources/org/elasticsearch/painless/java.time.zone.txt
(1)
M
modules/lang-painless/src/main/resources/org/elasticsearch/painless/java.util.txt
(43)
M
modules/lang-painless/src/main/resources/org/elasticsearch/painless/org.elasticsearch.txt
(4)
M
modules/lang-painless/src/test/java/org/elasticsearch/painless/BasicAPITests.java
(11)
M
modules/lang-painless/src/test/java/org/elasticsearch/painless/DefBootstrapTests.java
(6)
M
modules/lang-painless/src/test/java/org/elasticsearch/painless/FunctionRefTests.java
(59)
M
modules/lang-painless/src/test/java/org/elasticsearch/painless/OverloadTests.java
(2)

-- Patch Links --

https://github.com/elastic/elasticsearch/pull/18748.patch
https://github.com/elastic/elasticsearch/pull/18748.diff

---

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#18748

Uwe Schindler
H.-H.-Meier-Allee 63, 28213 Bremen
http://www.thetaphi.de

[uschindler]

Hi Robert: I just fixed the methodhandles lookup object passed to LambdaMetaFactory. For real lambdas we need to pass the lookup object of the original invokedyanmic, otherwise it wont find the lambda method impl. This fixes one todo

[uschindler]

Hi Robert,
to mee this looks like a good start. We should only fix the issues with the fromMethodDescriptorString() because I have the feeling this is using wrong class loader. In my opinion, we should (as you say in your XXX comment), fix the whole lookup stuff in the FunctionRef / Def class.

[uschindler]

BTW: filterArguments should be fine!

One question: How do we handle method calls using functional references that are statically compiled (e.g., calls on non-Def methods that pass a functional ref)

[rmuir]

As i mentioned above: we dont need to be doing any lookuping or classloader passing. This is just because of shitty ASM apis. We can fix that.

[rmuir]

One question: How do we handle method calls using functional references that are statically compiled (e.g., calls on non-Def methods that pass a functional ref)

This is the easy case: https://github.com/rmuir/elasticsearch/blob/3238868cc43346c34fe4068c0d7e60eff4e4c387/modules/lang-painless/src/main/java/org/elasticsearch/painless/node/EFunctionRef.java#L67-L79

We just invoke LambdaMetaFactory right there, and push the result (which is e.g. a Comparator or whatever the functional interface is) on the stack.

[uschindler]

Ah, missed that one!

Uwe Schindler
H.-H.-Meier-Allee 63, 28213 Bremen
http://www.thetaphi.de

[rmuir]

Yeah, the thing done there is the same as javac, you know the type of the expected interface, so you know everything :)

[rmuir]

I cleaned this up, it is not so horrible anymore. I want to iterate with further stuff as a followup.

[jdconrad]

LGTM. Thanks for getting method refs working.

[rmuir]

Thanks for the help here @uschindler @jdconrad . The ugly parts are fixed, we can try to extend it in followup issues with some of the deferred stuff, e.g. type[]::new or object::virtual. I am working on changes for some of those already.