Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is there a grammatically way to execute Node.js scripts via the Context API #2

Closed
swaechter opened this issue Apr 30, 2018 · 25 comments
Closed
Assignees

Comments

@swaechter
Copy link

@swaechter swaechter commented Apr 30, 2018

Hey there

Is it possible to execute/eval a Node.js script from the Context API? I can execute a hello world script via the GraalVM node binary:

const http = require('http');

const name = 'node-hello-world';
const port = '8888';

const app = new http.Server();

app.on('request', (req, res) => {
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.write('Hello World');
    res.end('\n');
});

app.listen(port, () => {
    console.log(`${name} is listening on port ${port}`);
});

But that doesn't work for the Java API:

import org.apache.commons.io.IOUtils;
import org.graalvm.polyglot.Context;

import java.io.InputStream;
import java.nio.charset.StandardCharsets;

public class NodeJsLauncher {

    public static void main(String[] args) throws Exception {
        new NodeJsLauncher();
    }

    public NodeJsLauncher() throws Exception {
        InputStream inputStream = getClass().getResourceAsStream("/HelloWorld.js");
        String helloWorldScript = IOUtils.toString(inputStream, StandardCharsets.UTF_8);

        Context context = Context.create();
        context.eval("js", helloWorldScript);
    }
}

Output:

Exception in thread "main" ReferenceError: require is not defined
	at <js> :program(Unnamed:1:13-19)
	at org.graalvm.polyglot.Context.eval(Context.java:336)
	at NodeJsLauncher.<init>(NodeJsLauncher.java:18)
	at NodeJsLauncher.main(NodeJsLauncher.java:10)

Process finished with exit code 1

Does the js language even support JavaScript with the Node.js module loading system? Or is it just not enabled:

Graal.js can execute Node.js applications. It provides high compatibility with existing npm packages, with high likelyhood that your application will run out of the box. This includes npm packages with native implementations. Note that you will need to re-compile from source with Graal.js if you want to run binaries that have beeen compiled for Node.js based on V8, or any other compatible engine.

Background: I am in the process of developing a Java solution that is able to server side render Angular SPA applications. The server side rendering requires some Node.js modules, so solutions like J2V8 (or maybe GraalVM) are required. For more information see https://github.com/swaechter/angularj-universal

@wirthi
Copy link
Member

@wirthi wirthi commented May 1, 2018

Hi Simon,

thanks for your interested in our engine.

Yes, what you ask for makes sense of course. However, consider that Node.js is a native application. Our approach is to base on Node.js sources directly - with only the JavaScript engine replaced - without major changes in the Node.js source code or its build process. This implies that, for Node.js support, you currently need to start Node.js as main (native) application. From there, you are free to call into Java and start whatever you want. Starting from Java and only then calling into Node.js would mean to deprive Node.js of all native access to e.g. signals, and it's unclear how the event loop would be started this way - we would have to significantly interfere the way it is started and managed.

This limitation is the reason why the Polyglot feature gives you a "pure" JavaScript (ECMAScript) engine, and not Node.js. Note that you can use load to eval external files, although that will only work for code not depending on the event loop and other core Node.js features.

We are currently working on APIs and code examples to do this nicely without running into threading problems. We could already show in experiments how to submit entries back into the Node.js event loop from Java. You want to avoid a scenario where your interop call (e.g., your Java application) blocks the event loop. We will address this in one of our next blog posts on Graal.js.

Best,
Christian

dougxc pushed a commit that referenced this issue May 25, 2018
Remove a pointless adapter frame  by fixing up the function's formal
parameter count.  Before:

    frame #0: 0x000033257ea446d5 onParserExecute(...)
    frame #1: 0x000033257ea3b93f <adaptor>
    frame #2: 0x000033257ea41959 <internal>
    frame #3: 0x000033257e9840ff <entry>

After:

    frame #0: 0x00000956287446d5 onParserExecute(...)
    frame #1: 0x0000095628741959 <internal>
    frame #2: 0x00000956286840ff <entry>

PR-URL: nodejs/node#17693
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com>
Reviewed-By: Khaidi Chu <i@2333.moe>
@swaechter
Copy link
Author

@swaechter swaechter commented May 28, 2018

Hey Christian

Thank you very much for the detailed explanation. This of course makes sense, so it's easier to keep up to date with the latest Node.js/V8 engine and profit from the work done there (Also you don't have to reinvent the wheel).

Best,
Simon

@edopalmieri
Copy link

@edopalmieri edopalmieri commented Sep 7, 2018

Hi,
Does enabling the v8-compat option when initialising a Context for JS execution bring any of the Node functionality (obviously not the core stuff like event loop, etc)?

I am wondering specifically about the require module. I have done a quick test:

Context ctx = Context.newBuilder().option("js.v8-compat", "true").build();
String jsCode = "const someLib = require('/someLib.js');";
ctx.eval("js", jsCode);
ctx.close();

But I get the same error that @swaechter was getting: require is not defined. Perhaps the require module is part of the native Node package and not the V8 engine?

Thanks
Edoardo.

@wirthi
Copy link
Member

@wirthi wirthi commented Sep 7, 2018

Hi Edoardo,

exactly, we don't implement require, that's provided by Node.js.

As an alternative, we offer load (https://github.com/graalvm/graaljs/blob/master/docs/user/JavaScriptCompatibility.md#loadsource), which might or might not work on npm modules.

Best,
Christian

@StephenOTT
Copy link

@StephenOTT StephenOTT commented Oct 15, 2018

@edopalmieri is there a way to run a command from within the context? such as node myJsFile.js --jvm ? So that we can pass the result of the command back into the Java application just as if it was normally run with context.eval("js", "....");

@edopalmieri
Copy link

@edopalmieri edopalmieri commented Oct 16, 2018

Hi @StephenOTT
From my understanding so far, what you are proposing is not possible. I think only two alternatives exist:

  1. Use load() inside the JS code to execute your myJsFile.js
  2. Execute myJsFile.js using ctx.eval() and then inject then result back into the ctx when running the code that needs it using the bindings.

Perhaps the Graal guys can suggest more approaches. I would be interested to know myself as I am trying to figure out a way to provide some sort of dependency importer/manager to my guest code.

Cheers
Edoardo.
PS: note that it is not possible to execute "node" code via the Polyglot API (ie. context) , only via the native node executable.

@StephenOTT
Copy link

@StephenOTT StephenOTT commented Oct 25, 2018

@edopalmieri, I had been playing around for using something like Process Builder to execute a node command as a CLI command and execute the specific .js file and then read back the result into a stream.
Had mixed results depending on argument usage, but generally it worked.

would be nice if this was built into the gralljs usage rather than having to jump through the hoops of processbuilder and streams. There is definitely the startup time factor of the node command, but if overall speed is not a concern and you are more concerned about specific usage of node modules, then this works well.

@edopalmieri
Copy link

@edopalmieri edopalmieri commented Oct 26, 2018

Hi @StephenOTT, I am guessing you mean java.lang.ProcessBuilder right?
That makes sense! It sounds like a nice work around to the problem, but I agree: it would be nice if it was offered as a feature of Graaljs. Perhaps it is in their roadmap ...

Thanks for sharing your solution
Edoardo.

@StephenOTT
Copy link

@StephenOTT StephenOTT commented Oct 26, 2018

@edopalmieri yes Java lang processbuilder. 👍🏻

@mahesh-kharat
Copy link

@mahesh-kharat mahesh-kharat commented Mar 18, 2019

Hi @wirthi @edopalmieri I am trying to load index.js from my java code

message.js

function prepareMessage(message) { var str = "Hello "; str += message; return str; } exports.prepareMessage = prepareMessage;


index.js

var index = load('./message.js'); console.log("Display message:"+index.prepareMessage('World!!!!'));


IndexTest.java

Context context = Context.newBuilder("js").allowIO(true).build(); context.eval("js", "load('index.js');"); context.close();

I am getting Exception in thread "main" TypeError: Cannot load script: index.js at <js> :program(Unnamed:1:0-15) at org.graalvm.polyglot.Context.eval(Context.java:361) at com.ge.hac.test.main.IndexTest.callIndex(IndexTest.java:29) at com.ge.hac.test.main.IndexTest.main(IndexTest.java:18)

Does this work? Or I am missing something?

Basically I want to evaluate that If I can use Node JS script (containing require feature...excluding event loop etc) from java 11 or not?

Any suggestion will help me to understand use of graaljs.

Thanks
Mahesh

@edopalmieri
Copy link

@edopalmieri edopalmieri commented Mar 19, 2019

Hi @mahesh-kharat

As @wirthi explained above, node features are not implemented in the Polyglot API. As a result you will not be able to use "require".
Looking at your code, I get the impression that you are trying to use Graal's "load" function as if it was Node's require. Careful with this as the two functions do not do the same thing.
Graal's load() simply loads a JS file for execution (with no concept of Node, modules, etc.).

Finally, it looks to me like your code should not throw an exception, and while it may not behave as you expect (by using load() that way), it should still run. Perhaps, the code simply cannot find the index.js file?
In any case, @wirth will be able to help you more than I can.

Cheers
Edoardo.

@wirthi
Copy link
Member

@wirthi wirthi commented Mar 19, 2019

Hi @mahesh-kharat

(thanks @edopalmieri for your answer)

It is working for me. As Edoardo wrote, most likely, the index.js file might not be found, it needs to be in the root of your java app (i.e. in the same directory as your java file, if this uses the default package).

But then, as Edoardo wrote, you need to use pure JavaScript syntax and cannot use Node.js concepts (like export). load will parse and evaluate the file; the prepareMessage function will then be in the global scope, you cannot put it in exports (which does not exist - of course you could provide it from index.js).

index.js (with the variant of providing exports from there):

var exports={};
load('./message.js');
console.log("Display message:"+exports.prepareMessage('World!!!!'));

Cleaner would be if you created the exports object in message.js and returned that. But you have to manually do that, this is not magically happening like in Node.js.

Best,
Christian

@mahesh-kharat
Copy link

@mahesh-kharat mahesh-kharat commented Mar 19, 2019

Thank you for replying @edopalmieri @wirthi

@wirthi I was able to run above example after putting those index.js and message.js in project's default location. Thanks for pointing out.

I wanted to know if we can run "Node modules from jdk 11 using graalvm". In above example I have changed 'require' to 'load' in javascript file to fetch dependency (as it's custom implementation). I have a requirement where I have to use third party dependencies like d3, d3-node and many other where I can't change 'require' to 'load'. In that case will it be possible to execute those node modules from jdk 11 using graalvm? Is there any way of doing it?

@wirthi
Copy link
Member

@wirthi wirthi commented Mar 19, 2019

Hi @mahesh-kharat

short answer: no, not possible.

long answer: Node.js is a binary application. It cannot trivially be started from within a running JVM. Longer-term, we are working towards enabling such a setup, but it will require some modifications in core Node.js code that we typically want to avoid in order not to break compatibility. The easiest way forward, if that is possible for you, is to use <GraalVM>/bin/node --jvm and start you Java code from there. https://medium.com/graalvm/multi-threaded-java-javascript-language-interoperability-in-graalvm-2f19c1f9c37b can give you some hints how to be do that.

Best,
Christian

@ralleman
Copy link

@ralleman ralleman commented Apr 23, 2019

IMO, being able to run Node modules from within a Java-based Graal application is likely to be a common request. I'm looking into it too. I'm wondering if a project is needed to create Javaised polyfills for the built-in Node modules. If they existed, anyone could run Node modules in their Java apps.

@psanders
Copy link

@psanders psanders commented May 13, 2019

I had this issue before with Nashorn, and endup using Nodyn's package loader. I created a repo with an example of my approach. You can also see this working in a larger project.

I hope this helps someone!

@jfrantzius
Copy link

@jfrantzius jfrantzius commented May 14, 2019

Hi @wirthi ,
is there per chance any publicly accessible information about this longer-term work you mentioned? It would be great if it was tracked as a Github issue, so people can get notified on eventual progress :)

On a sidenote, in our particular case we'd just like to be able to require any npm modules from Java, which in turn should be able to use Node API like fs. If this was possible without spinning up a Node.js event loop, that would be even better!

I guess that in the case of a Java web server that wants to make use of npm modules, there probably is no need for a Node.js event loop?

@hitsmaxft
Copy link

@hitsmaxft hitsmaxft commented Jul 27, 2019

hey @wirthi .

We are currently working on APIs and code examples to do this nicely without running into threading problems. We could already show in experiments how to submit entries back into the Node.js event loop from Java. You want to avoid a scenario where your interop call (e.g., your Java application) blocks the event loop. We will address this in one of our next blog posts on Graal.js.

Is there any code example for message passing from java back to node under node --jvm mode ?

As you mentioned , submit entries back into the Node.js event loop from Java . node thread may receive message from java call by a blocking worker , but it's no an elegant way. ince i've found this in the source code https://github.com/graalvm/graaljs/blob/a6c93a837805b8041fabd59d9f822b3887bc4d66/graal-nodejs/test/graal/unit/javaMessages.js

it('Java can schedule back to the main Node.js event loop using a (blocking) worker', function(done) {

I still look for a postMessage style api for java side to pass new message into node event pool.

for example.

// node side 

javaPort.on("message", handler)

 
//java side 

nodePort.postMesasage("new message")

just like worker_threads modules

@skr316
Copy link

@skr316 skr316 commented Jun 11, 2020

Is 'require' now supported in Graal Context as per this release ?
https://github.com/graalvm/graaljs/blob/master/docs/user/NodeJSVSJavaScriptContext.md#commonjs-modules-cjs

@wirthi
Copy link
Member

@wirthi wirthi commented Jun 12, 2020

It is as written in the docu you link:

The js.commonjs-require option provides a built-in require() function that can be used to load Npm-compatible CommonJS modules in a JavaScript Context. Currently, this is an experimental feature not for production usage.

It is available behind a flag, is experimental and provides only partial support (similar to what tools like browserify or webpack provide). E.g., you cannot access the Node.js eventloop that way.

@cvadatta
Copy link

@cvadatta cvadatta commented Oct 6, 2020

Hi! I'm trying to do a Fetch request from js inside a java program. what is the best way to do this?
Can I have a example please!!!!!!!!!

@wdanilo
Copy link

@wdanilo wdanilo commented Feb 8, 2021

The fundamentally broken promise of GraalVM JavaScript interop

Hi! @wirthi, @swaechter, first of all, let me express my astonishment that this ticket was closed and even a much greater astonishment when I learned that lack of support for utilizing the node.js functionality from within languages based on GraalVM is a design decision, at least that is what I read from the @wirthi reply from almost 3 years ago (1 May 2018). The possibility to use npm modules and call node.js functions from GraalVM / languages built on top of the Truffle framework is a fundamental promise of seamless languages interoperability. The whole power of JavaScript is inherently bound to the npm ecosystem, and therefore, the node.js functionality. The JS runtime alone is not really useful for anything serious – what do we need interoperability with JS for if we cannot use any server-side functionality and we cannot use npm packages?

The confusing / misleading promises on the GraalVM website.

Also, I believe that the documentation on the official GraalVM website is very confusing. There are a lot of fragments like these:

image

Or this one:
image

Recently, both our team and some people from our community were stunned that we cannot use node.js modules from within our language, after spending weeks on implementing the JS support, because the website is (at least for us) selling this idea, while being very unclear that GraalVM doesn't support interoperability with JavaScript and Node. It only provides a special Node version that can be used to run JavaScript which can call other languages but not vice versa.

Summary

To sum this up, I believe this ticket should be open and we should consider calling node.js modules from within languages implemented on the Truffle framework as one of the top priorities of the JS interop, as without this functionality, the interop with JS from other languages is not very useful to anything serious.

@wirthi
Copy link
Member

@wirthi wirthi commented Feb 9, 2021

Hi @wdanilo

You can execute Node.js application. node yourApplication.js will work.

You can interop from your Node.js application to Java and to all Polyglot Truffle languages we support (Ruby, Python, LLVM, etc.) - and then call back to the JavaScript (Node.js) level. Once you started from node, everything is possible.

True, what you cannot do is to interop from a Java application to a Node.js application. Why? Because Node.js is a binary application, not a language. You cannot interop from, say, Ruby to Node.js for the same reason you cannot interop from Ruby to Eclipse, or vi, or Microsoft Word: those are applications that expect that their own launcher is executed, in order to set everything up. So in order to get the (full) support of Node.js - event loop, etc - your application needs to start from the node binary.

There are several workarounds for this limitation, quoted both here above in this ticket an in e.g. #321, or in our documentation at https://github.com/oracle/graaljs/blob/master/docs/user/NodeJSVSJavaScriptContext.md You can package your Node.js application, you can use our experimental module loading support, and you can use the NodeJVM approach to wrap an existing Java Application with the Node launcher.

(Re-)Implementing Node.js in Java so you can REALLY call it as a language is a possibility. It requires some engineering effort, but it is technically feasible. The problem is that you will have to follow up all future changes and include them to this port to ensure compatibility. In our current approach, we take the Node.js source code (mostly) unmodified, which allows for more straightforward updates when a new Node.js version comes out. If anyone is up for this task, we are happy to support you and help you get it done. There are no efforts from our side in this respect currently, because we believe the current approach is good enough to support reasonable applications and the engineering effort of re-implementing the core of Node.js in Java is too high with too big a risk of having to maintain a still-not-fully-compatible port (been there, done that: Project Avatar).

The open ticket we have in this area is #321 - but again, we are not actively working on this solution at the moment. Happy to receive reasonable external contributions, of course.

Best,
Christian

@swaechter
Copy link
Author

@swaechter swaechter commented Feb 10, 2021

Hey @wdanilo Although I understand your astonishment/frustration I want to share some of my own thoughts about one of your statements:

The possibility to use npm modules and call node.js functions from GraalVM / languages built on top of the Truffle framework is a fundamental promise of seamless languages interoperability

In theory language interoperability (Like using Node.js from a JVM and even vice versa) sounds great: You can share & design code/components in different languages so these components can interop with each other and code can be reused. But this flexibilty has its price: Mainly complexity. In the end I gave up on tools like J2V8 (https://github.com/eclipsesource/J2V8) or GraalVM not because of the tool itself, but because the developer experience suffers. In my Angular server side rendering project I had to develop an own hot reload mechanism for the JS code and the Java application that loads this code (If JS restarts, Java has to reload too). Debugging it accross the language barrier...let's just say difficult. I guess this is just the polyglot nature or in short: There is no seamless languages interoperability as you say (Sorry Graal devs, I still love you for the native-image ).

To sum this up: Rethink if you really need language interoperability. In the case of Angular SSR I switched to a Java backend that communicates via pipe/gRPC mechanism with a standalone Node.js backend that does the SSR (A bit slower, but a lot more flexible). In other situations like Oracle DB with JS support, it helps a lot.

@jessica-d-pennell-at-xcelenergy
Copy link

@jessica-d-pennell-at-xcelenergy jessica-d-pennell-at-xcelenergy commented Feb 16, 2021

I have had some luck with PurpleJS to close this gap, https://github.com/purplejs/purplejs . I submitted the patches I needed to use to build it with OpenJDK8, along with a build script. They are in the issues tab.

Strong caveat : this is a very old (in computer years) project.

In my case there is a real need; I need the ability to allow other team members to write Cypress tests in a larger framework which uses java SDKs to communicate with several cloud services. Most API testers are familiar with JavaScript frameworks and most API test engines support JavaScript; it is the tool that is used and any professional will expect these tools be available and will be forced to reinvent several wheels if they are not.

In the general case, there are many real world cases where there is an expectation that JavaScript be a "lingua franca". Every year since 2018 there has been increasing demand and expectation for JavaScript to be supported in many more contexts than it ever has been for this reason. It is no accident that JSON has replaced XML as the default transport layer.

There is a very real need to at least adopt some concepts such as the ability to require modules. GraalVM doesn't have to be a Node replacement. But currently Java has no scalable JavaScript support whatsoever, and it seems the shortest path to gaining scalability may well be to simply offer full node support as Sten Roger Sandvik has done with PurpleJS.

Many of us on the internet have been reinventing this wheel for decades. If GraalVM truly is meant to replace Nashorn, it may be time to pull this out of the backlog before a larger influx of developers comes to this board with exactly this complaint.

Be gentle, but the (work in progress) update I have for PurpleJS that adds GraalVM support is here, https://gitlab.com/jessica.d.pennell/purplejs-with-graalvm/ . I have been able to get require to work correctly, but have more work to do with my CompatScriptObjectMirror class and other differences between Nashorn and GraalVM before this can be considered a real solution. Nonetheless, if you are just googling and found this page, this should be enough to get you started with importing modules in your own project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked pull requests

Successfully merging a pull request may close this issue.

None yet