Support a low-level PolyglotEngine execution mode

[pniederw]

Use of PolyglotEngine is required to make certain parts of Truffle work (e.g. retrieval of ExecutionContext). From what I've heard, PolyglotEngine may even become mandatory in the future.

However, I found that when writing tools tailored to a specific language, PolyglotEngine isn't a good fit: Values have to be unwrapped, exceptions have to be unwrapped, only Sources (but not ASTs) can be evaluated, etc. After a lot of fighting, I gave up and decided to bypass PolyglotEngine for the time being.

To solve this problem, I propose that PolyglotEngine should provide an additional lower-level execution mode that allows to run any callback within an engine context, without the baggage of Value, Source, etc. This would greatly simplify writing of language-specific tools.

Thanks to Chris Seaton for bringing up the idea of a lower-level execution mode.

[pniederw]

@jtulach Any thoughts?

Most of the language implementations I've seen to date work around these issues with bootstrapping tricks. It would be nice to have a better solution.

[jtulach]

I haven't responded as I didn't know what to say. The text in this issue is so vague that it cannot be treated as a real bug report. Unless I see the real problems, I can't discuss solutions.

[pniederw]

@jtulach I've tried my best to give an initial problem statement, but I'll try again:

When language-specific code (e.g. a language-specific REPL implementation) wants to interface with its language, it's natural that it wants to work directly with the value types, AST nodes, and exceptions used by the language implementation. In such a case, PolyglotEngine's abstractions such as Value and wrapping every exception with IOException get in the way, and the fact that only a Source (but not a Node) can be executed is limiting.

Here is how SOMns and JRuby/Truffle work around this:

https://github.com/smarr/SOMns/blob/master/src/som/interpreter/SomLanguage.java#L150

https://github.com/jruby/jruby/blob/3cabc55bb8e4ca3f3133f508ebbefce9e0768869/truffle/src/main/java/org/jruby/truffle/JRubyTruffleImpl.java#L54

I believe these constructions are only necessary because there is no supported way to execute anything other than a Source in a PolyglotEngine managed context.

Has @chrisseaton talked to you about this? We had a discussion in the Gitter channel, and he wanted to propose the "lower-level execution mode" idea to you.

[chrisseaton]

My idea was

vm.executeInLanguageContext('application/x-my-language', () -> {
    whatever you want here
})

It would just be syntactic sugar for the current workarounds in the other languages.

That was my proposed solution to what you wanted, but really I'm happy with the current system - I only have the workaround in a couple of places so I don't see it as a huge burden.

[pniederw]

@chrisseaton Keep in mind that your understanding is way above that of the average Truffle user, and that other language implementations may have additional constraints that make such workarounds more complicated.

[jtulach]

When language-specific code (e.g. a language-specific REPL implementation) wants to interface
with its language, it's natural that ...

...they want to take shortcuts. Yes, in such system it may seem that the overhead of PolyglotEngine is too big and unnecessary. However it isn't true, as soon as you find yourself talking with other language or using some language agnostic instrument - you'll benefit from using the PolyglotEngine concepts.

it wants to work directly with the value types, AST nodes, and exceptions used by the language
implementation. In such a case, PolyglotEngine's abstractions such as Value and wrapping every > exception with IOException get in the way, and the fact that only a Source (but not a Node) can be > executed is limiting.

• I'll start working on removing the IOException soon.
• Value has method as which you can use to extract your internal type value
• implementing your own parsing function and invoking it via Value.execute is also possible - with any parameters

Using these tricks is dangerous - unless you are careful, you may find out things like debugger don't work - to guarantee that I would need to tighten things up, not relax them.

[chrisseaton]

@pniederw I think we should be able to implement the method I proposed, using the existing Truffle API stuff. We could that in your code, so you don't need a modification to the API. Let me know if you want to try that.

[pniederw]

...they want to take shortcuts. Yes, in such system it may seem that the overhead of PolyglotEngine is too big and unnecessary. However it isn't true, as soon as you find yourself talking with other language or using some language agnostic instrument - you'll benefit from using the PolyglotEngine concepts.

The point I'm trying to make is that something seems to be missing from PolyglotEngine's concepts. It shouldn't be necessary for every other language implementation, even those by highly skilled Truffle experts, to funnel some bootstrapping code through PolyglotEngine. Or if that's really the way to go, it's critical to have this pattern documented.

To give one example from my own language, my tooling sometimes needs to programmatically construct an AST and execute that. It's not clear to me how to achieve this with PolyglotEngine, at least not without significant workarounds.

By now I've implemented a complete Truffle interpreter for my language (which is roughly as complex as SOMns) and some APIs/tooling around it. How to fit in PolyglotEngine has been the hardest part of this (!), the only part I haven't been able to figure out yet. That's the feedback I have for you. Oh, and thanks for this great technology!

[pniederw]

@chrisseaton I'd love to try that. The only constraint I have is that (unfortunately) I can't share any of my code at this time.

[chrisseaton]

I thought I could do this cleanly, because I thought that one of the public PolyglotEngine eval methods accepted arguments, but it looks like that isn't the case. If you don't care about races you can do it like below, or maybe you could allocate some kind of unique name to recognise a particular callback. You'd have to GC those somehow.

diff --git a/truffle/com.oracle.truffle.sl/src/com/oracle/truffle/sl/SLLanguage.java b/truffle/com.oracle.truffle.sl/src/com/oracle/truffle/sl/SLLanguage.java
index 1f52d88..f6e86ca 100644
--- a/truffle/com.oracle.truffle.sl/src/com/oracle/truffle/sl/SLLanguage.java
+++ b/truffle/com.oracle.truffle.sl/src/com/oracle/truffle/sl/SLLanguage.java
@@ -52,9 +52,11 @@ import com.oracle.truffle.api.Truffle;
 import com.oracle.truffle.api.TruffleLanguage;
 import com.oracle.truffle.api.debug.DebuggerTags;
 import com.oracle.truffle.api.frame.MaterializedFrame;
+import com.oracle.truffle.api.frame.VirtualFrame;
 import com.oracle.truffle.api.instrumentation.ProvidedTags;
 import com.oracle.truffle.api.instrumentation.StandardTags;
 import com.oracle.truffle.api.nodes.Node;
+import com.oracle.truffle.api.nodes.RootNode;
 import com.oracle.truffle.api.source.Source;
 import com.oracle.truffle.sl.nodes.SLEvalRootNode;
 import com.oracle.truffle.sl.nodes.SLRootNode;
@@ -89,6 +91,20 @@ public final class SLLanguage extends TruffleLanguage<SLContext> {

     @Override
     protected CallTarget parse(Source source, Node node, String... argumentNames) throws IOException {
+        if (source.getCode().equals("@callback")) {
+            final Runnable runnableWhenCreated = SLMain.runnable;
+
+            return Truffle.getRuntime().createCallTarget(new RootNode(SLLanguage.class, null, null) {
+
+                @Override
+                public Object execute(VirtualFrame frame) {
+                    runnableWhenCreated.run();
+                    return null;
+                }
+
+            });
+        }
+
         Map<String, SLRootNode> functions;
         try {
             /*
diff --git a/truffle/com.oracle.truffle.sl/src/com/oracle/truffle/sl/SLMain.java b/truffle/com.oracle.truffle.sl/src/com/oracle/truffle/sl/SLMain.java
index ab994d5..9cff745 100644
--- a/truffle/com.oracle.truffle.sl/src/com/oracle/truffle/sl/SLMain.java
+++ b/truffle/com.oracle.truffle.sl/src/com/oracle/truffle/sl/SLMain.java
@@ -208,6 +208,14 @@ public final class SLMain {
         PolyglotEngine engine = PolyglotEngine.newBuilder().setIn(in).setOut(out).build();
         assert engine.getLanguages().containsKey(SLLanguage.MIME_TYPE);

+        runInSLContext(engine, new Runnable() {
+
+            public void run() {
+                System.err.println("hello");
+            }
+
+        });
+
         try {
             Value result = engine.eval(source);

@@ -290,4 +298,20 @@ public final class SLMain {
         }
         return result.toString();
     }
+
+    public static Runnable runnable;
+
+    private static void runInSLContext(PolyglotEngine engine, Runnable runnable) {
+        SLMain.runnable = runnable;
+
+        final Source source = Source.newBuilder("@callback").name("(callback)").mimeType(SLLanguage.MIME_TYPE).build();
+
+        try {
+            engine.eval(source);
+        } catch (IOException e) {
+            // TODO Auto-generated catch block
+            e.printStackTrace();
+        }
+    }
+
 }

[jtulach]

Hello Chris,
can't we do it like this (speaking in JavaScript):

Runnable r = // your own code
enter = engine.eval("(function(f) { f(); })");
TruffleObject callback = JavaInterop.asTruffleFunction(r);

and then any time later:

enter.execute(null, callback); // null is JavaScript specifiec thing

I believe I am using this style in my own work at
http://source.apidesign.org/hg/html~html4j/rev/47cd2110ed8d
and it seems to work somehow.

-jt

Chris Seaton : 10. 7. 2016 @ 05:52

I thought I could do this cleanly, because I thought that one of the public
PolyglotEngine eval methods accepted arguments, but it looks like that
isn't the case. If you don't care about races you can do it like below, or
maybe you could allocate some kind of unique name to recognise a particular
callback. You'd have to GC those somehow.

diff --git
a/truffle/com.oracle.truffle.sl/src/com/oracle/truffle/sl/SLLanguage.java
b/truffle/com.oracle.truffle.sl/src/com/oracle/truffle/sl/SLLanguage.java
index 1f52d88..f6e86ca 100644
---
a/truffle/com.oracle.truffle.sl/src/com/oracle/truffle/sl/SLLanguage.java
+++
b/truffle/com.oracle.truffle.sl/src/com/oracle/truffle/sl/SLLanguage.java
@@ -52,9 +52,11 @@ import com.oracle.truffle.api.Truffle;
 import com.oracle.truffle.api.TruffleLanguage;
 import com.oracle.truffle.api.debug.DebuggerTags;
 import com.oracle.truffle.api.frame.MaterializedFrame;
+import com.oracle.truffle.api.frame.VirtualFrame;
 import com.oracle.truffle.api.instrumentation.ProvidedTags;
 import com.oracle.truffle.api.instrumentation.StandardTags;
 import com.oracle.truffle.api.nodes.Node;
+import com.oracle.truffle.api.nodes.RootNode;
 import com.oracle.truffle.api.source.Source;
 import com.oracle.truffle.sl.nodes.SLEvalRootNode;
 import com.oracle.truffle.sl.nodes.SLRootNode;
@@ -89,6 +91,20 @@ public final class SLLanguage extends
TruffleLanguage<SLContext> {

     @Override
     protected CallTarget parse(Source source, Node node, String...
argumentNames) throws IOException { +        if
(source.getCode().equals("@callback")) {
+            final Runnable runnableWhenCreated = SLMain.runnable;
+
+            return Truffle.getRuntime().createCallTarget(new
RootNode(SLLanguage.class, null, null) { +
+                @Override
+                public Object execute(VirtualFrame frame) {
+                    runnableWhenCreated.run();
+                    return null;
+                }
+
+            });
+        }
+
         Map<String, SLRootNode> functions;
         try {
             /*
diff --git
a/truffle/com.oracle.truffle.sl/src/com/oracle/truffle/sl/SLMain.java
b/truffle/com.oracle.truffle.sl/src/com/oracle/truffle/sl/SLMain.java index
ab994d5..9cff745 100644
--- a/truffle/com.oracle.truffle.sl/src/com/oracle/truffle/sl/SLMain.java
+++ b/truffle/com.oracle.truffle.sl/src/com/oracle/truffle/sl/SLMain.java
@@ -208,6 +208,14 @@ public final class SLMain {
         PolyglotEngine engine =
PolyglotEngine.newBuilder().setIn(in).setOut(out).build(); assert
engine.getLanguages().containsKey(SLLanguage.MIME_TYPE);

+        runInSLContext(engine, new Runnable() {
+
+            public void run() {
+                System.err.println("hello");
+            }
+
+        });
+
         try {
             Value result = engine.eval(source);

@@ -290,4 +298,20 @@ public final class SLMain {
         }
         return result.toString();
     }
+
+    public static Runnable runnable;
+
+    private static void runInSLContext(PolyglotEngine engine, Runnable
runnable) { +        SLMain.runnable = runnable;
+
+        final Source source =
Source.newBuilder("@callback").name("(callback)").mimeType(SLLanguage.MIME_
TYPE).build(); +
+        try {
+            engine.eval(source);
+        } catch (IOException e) {
+            // TODO Auto-generated catch block
+            e.printStackTrace();
+        }
+    }
+
 }

---

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#183 (comment)

[chrisseaton]

Yes that works. And I guess if you had a language where you couldn't natively write that, you could introduce a special form for it.

[chumer]

We are going to continue to support to use the Truffle core APIs without PolyglotEngine. However the instrumentation and multi-language interop will only work embedded in PolyglotEngine.

@pniederw : @jtulach has explained how PolyglotEngine can be used to achieve your goals. Can you work with that? Or do you want to make other concrete suggestions on how to improve? Please close this issue if all your questions are answered.

[smarr]

ok, so, I should report NullPointerExceptions for stuff that breaks when not initializing the PolyglotEngine?

[pniederw]

We are going to continue to support to use the Truffle core APIs without PolyglotEngine.

Without PolyglotEngine, lookup of the execution context (a concept exposed by TruffleLanguage) doesn't work. Hence I'd argue that use of PolyglotEngine is already mandatory.

@jtulach has explained how PolyglotEngine can be used to achieve your goals. Can you work with that?

My language doesn't currently support language interop, and supporting it is not a priority. The "special form" solution suggested by @chrisseaton should work, but feels like an obscure workaround for a common need. However, it does demonstrate that running non-program code within a context is already possible, which means that adding a public API for this wouldn't lead to any loss of control (which was used as an argument for not adding such an API).

Or do you want to make other concrete suggestions on how to improve?

@chrisseaton's suggestion would solve my problem immediately: #183 (comment)

[pniederw]

Is there any consensus on whether something like @chrisseaton's vm.executeInLanguageContext() idea should be added?

[chrisseaton]

It's up to @jtulach whether the new method is added, and from his previous comments it looks like he's not keen on it.

All I can suggest is helping you to implement the tiny amount of interop needed to make the eval pattern work for you.

[jtulach]

I believe that common things should be easy, complex possible. Here is a working code that gets anyone into the language context:

public final class Main {
    public void callMeInContext() {
        Thread.dumpStack();
    }

    public static void main(String[] args) throws IOException {
        PolyglotEngine engine = PolyglotEngine.newBuilder().build();

        final Source source = Source.
            newBuilder("function() { return this.callMeInContext(); }").
            name("callback.js").
            mimeType("text/javascript").
            build();

        final PolyglotEngine.Value invoke = engine.eval(source);
        final TruffleObject callback = JavaInterop.asTruffleObject(new Main());

        invoke.execute(callback);
        invoke.execute(callback);
    }
}

I don't think the solution is that complex and the request that common to justify addition of new method into the API.

[chumer]

@pniederw PolyglotEngine is not mandatory if you are not using TruffleLanguage. Only TruffleRuntime, CallTarget, RootNode, Node can be used as well. TruffleLanguage is the SPI side of PolyglotEngine.

To execute your language you would parse on your own and then call into a CallTarget. Thats what PolyglotEngine is currently doing for you, but if you, as you say, don't care about instrumentation/interop feel free skip PolyglotEngine and TruffleLanguage.

[pniederw]

@jtulach There's no OSS Truffle JS implementation at this point in time, and shipping a JS implementation with my language just to solve this problem doesn't make sense. From my perspective, language interop is a complex solution to this common problem.

@chrisseaton I'm keen on giving this a go. Can you give me some pointers on how to implement the necessary amount of language interop? Unfortunately I can't share my code at this point.

@chumer Good point. Ditching all ofPolyglotEngine, TruffleLanguage, and ExecutionContext seems like an option.

[chrisseaton]

You need to subclass ObjectType and override getForeignAccessFactory, returning a ForeignAccess. You can create one of those using a DSL. The Ruby one is only a couple of hundred lines, and it's only complicated because it needs to be fast.

https://github.com/jruby/jruby/blob/1fdc6454a6f9c63e5dc0f400f836b34e8bf7efe7/truffle/src/main/java/org/jruby/truffle/interop/RubyMessageResolution.java

If you are just using this for Jaroslav's pattern, you could put simple call logic here:

https://github.com/jruby/jruby/blob/1fdc6454a6f9c63e5dc0f400f836b34e8bf7efe7/truffle/src/main/java/org/jruby/truffle/interop/RubyMessageResolution.java#L57-L57

Catch me on Gitter and I'll walk you through it.

[chumer]

Seems like the issue is resolved. Low-level execution without any polyglot support is possible for a custom language. I would however not recommend it as it eliminates many benefits, like interop/instruments/debugging.