Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is there a grammatically way to execute Node.js scripts via the Context API #2

Closed
swaechter opened this issue Apr 30, 2018 · 32 comments
Assignees

Comments

@swaechter
Copy link

Hey there

Is it possible to execute/eval a Node.js script from the Context API? I can execute a hello world script via the GraalVM node binary:

const http = require('http');

const name = 'node-hello-world';
const port = '8888';

const app = new http.Server();

app.on('request', (req, res) => {
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.write('Hello World');
    res.end('\n');
});

app.listen(port, () => {
    console.log(`${name} is listening on port ${port}`);
});

But that doesn't work for the Java API:

import org.apache.commons.io.IOUtils;
import org.graalvm.polyglot.Context;

import java.io.InputStream;
import java.nio.charset.StandardCharsets;

public class NodeJsLauncher {

    public static void main(String[] args) throws Exception {
        new NodeJsLauncher();
    }

    public NodeJsLauncher() throws Exception {
        InputStream inputStream = getClass().getResourceAsStream("/HelloWorld.js");
        String helloWorldScript = IOUtils.toString(inputStream, StandardCharsets.UTF_8);

        Context context = Context.create();
        context.eval("js", helloWorldScript);
    }
}

Output:

Exception in thread "main" ReferenceError: require is not defined
	at <js> :program(Unnamed:1:13-19)
	at org.graalvm.polyglot.Context.eval(Context.java:336)
	at NodeJsLauncher.<init>(NodeJsLauncher.java:18)
	at NodeJsLauncher.main(NodeJsLauncher.java:10)

Process finished with exit code 1

Does the js language even support JavaScript with the Node.js module loading system? Or is it just not enabled:

Graal.js can execute Node.js applications. It provides high compatibility with existing npm packages, with high likelyhood that your application will run out of the box. This includes npm packages with native implementations. Note that you will need to re-compile from source with Graal.js if you want to run binaries that have beeen compiled for Node.js based on V8, or any other compatible engine.

Background: I am in the process of developing a Java solution that is able to server side render Angular SPA applications. The server side rendering requires some Node.js modules, so solutions like J2V8 (or maybe GraalVM) are required. For more information see https://github.com/swaechter/angularj-universal

@wirthi
Copy link
Member

wirthi commented May 1, 2018

Hi Simon,

thanks for your interested in our engine.

Yes, what you ask for makes sense of course. However, consider that Node.js is a native application. Our approach is to base on Node.js sources directly - with only the JavaScript engine replaced - without major changes in the Node.js source code or its build process. This implies that, for Node.js support, you currently need to start Node.js as main (native) application. From there, you are free to call into Java and start whatever you want. Starting from Java and only then calling into Node.js would mean to deprive Node.js of all native access to e.g. signals, and it's unclear how the event loop would be started this way - we would have to significantly interfere the way it is started and managed.

This limitation is the reason why the Polyglot feature gives you a "pure" JavaScript (ECMAScript) engine, and not Node.js. Note that you can use load to eval external files, although that will only work for code not depending on the event loop and other core Node.js features.

We are currently working on APIs and code examples to do this nicely without running into threading problems. We could already show in experiments how to submit entries back into the Node.js event loop from Java. You want to avoid a scenario where your interop call (e.g., your Java application) blocks the event loop. We will address this in one of our next blog posts on Graal.js.

Best,
Christian

dougxc pushed a commit that referenced this issue May 25, 2018
Remove a pointless adapter frame  by fixing up the function's formal
parameter count.  Before:

    frame #0: 0x000033257ea446d5 onParserExecute(...)
    frame #1: 0x000033257ea3b93f <adaptor>
    frame #2: 0x000033257ea41959 <internal>
    frame #3: 0x000033257e9840ff <entry>

After:

    frame #0: 0x00000956287446d5 onParserExecute(...)
    frame #1: 0x0000095628741959 <internal>
    frame #2: 0x00000956286840ff <entry>

PR-URL: nodejs/node#17693
Reviewed-By: Anna Henningsen <anna@addaleax.net>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Daniel Bevenius <daniel.bevenius@gmail.com>
Reviewed-By: Khaidi Chu <i@2333.moe>
@swaechter
Copy link
Author

Hey Christian

Thank you very much for the detailed explanation. This of course makes sense, so it's easier to keep up to date with the latest Node.js/V8 engine and profit from the work done there (Also you don't have to reinvent the wheel).

Best,
Simon

@edopalmieri
Copy link

Hi,
Does enabling the v8-compat option when initialising a Context for JS execution bring any of the Node functionality (obviously not the core stuff like event loop, etc)?

I am wondering specifically about the require module. I have done a quick test:

Context ctx = Context.newBuilder().option("js.v8-compat", "true").build();
String jsCode = "const someLib = require('/someLib.js');";
ctx.eval("js", jsCode);
ctx.close();

But I get the same error that @swaechter was getting: require is not defined. Perhaps the require module is part of the native Node package and not the V8 engine?

Thanks
Edoardo.

@wirthi
Copy link
Member

wirthi commented Sep 7, 2018

Hi Edoardo,

exactly, we don't implement require, that's provided by Node.js.

As an alternative, we offer load (https://github.com/graalvm/graaljs/blob/master/docs/user/JavaScriptCompatibility.md#loadsource), which might or might not work on npm modules.

Best,
Christian

@StephenOTT
Copy link

StephenOTT commented Oct 15, 2018

@edopalmieri is there a way to run a command from within the context? such as node myJsFile.js --jvm ? So that we can pass the result of the command back into the Java application just as if it was normally run with context.eval("js", "....");

@edopalmieri
Copy link

edopalmieri commented Oct 16, 2018

Hi @StephenOTT
From my understanding so far, what you are proposing is not possible. I think only two alternatives exist:

  1. Use load() inside the JS code to execute your myJsFile.js
  2. Execute myJsFile.js using ctx.eval() and then inject then result back into the ctx when running the code that needs it using the bindings.

Perhaps the Graal guys can suggest more approaches. I would be interested to know myself as I am trying to figure out a way to provide some sort of dependency importer/manager to my guest code.

Cheers
Edoardo.
PS: note that it is not possible to execute "node" code via the Polyglot API (ie. context) , only via the native node executable.

@StephenOTT
Copy link

@edopalmieri, I had been playing around for using something like Process Builder to execute a node command as a CLI command and execute the specific .js file and then read back the result into a stream.
Had mixed results depending on argument usage, but generally it worked.

would be nice if this was built into the gralljs usage rather than having to jump through the hoops of processbuilder and streams. There is definitely the startup time factor of the node command, but if overall speed is not a concern and you are more concerned about specific usage of node modules, then this works well.

@edopalmieri
Copy link

Hi @StephenOTT, I am guessing you mean java.lang.ProcessBuilder right?
That makes sense! It sounds like a nice work around to the problem, but I agree: it would be nice if it was offered as a feature of Graaljs. Perhaps it is in their roadmap ...

Thanks for sharing your solution
Edoardo.

@StephenOTT
Copy link

@edopalmieri yes Java lang processbuilder. 👍🏻

@mahesh-kharat
Copy link

mahesh-kharat commented Mar 18, 2019

Hi @wirthi @edopalmieri I am trying to load index.js from my java code

message.js

function prepareMessage(message) { var str = "Hello "; str += message; return str; } exports.prepareMessage = prepareMessage;


index.js

var index = load('./message.js'); console.log("Display message:"+index.prepareMessage('World!!!!'));


IndexTest.java

Context context = Context.newBuilder("js").allowIO(true).build(); context.eval("js", "load('index.js');"); context.close();

I am getting Exception in thread "main" TypeError: Cannot load script: index.js at <js> :program(Unnamed:1:0-15) at org.graalvm.polyglot.Context.eval(Context.java:361) at com.ge.hac.test.main.IndexTest.callIndex(IndexTest.java:29) at com.ge.hac.test.main.IndexTest.main(IndexTest.java:18)

Does this work? Or I am missing something?

Basically I want to evaluate that If I can use Node JS script (containing require feature...excluding event loop etc) from java 11 or not?

Any suggestion will help me to understand use of graaljs.

Thanks
Mahesh

@edopalmieri
Copy link

Hi @mahesh-kharat

As @wirthi explained above, node features are not implemented in the Polyglot API. As a result you will not be able to use "require".
Looking at your code, I get the impression that you are trying to use Graal's "load" function as if it was Node's require. Careful with this as the two functions do not do the same thing.
Graal's load() simply loads a JS file for execution (with no concept of Node, modules, etc.).

Finally, it looks to me like your code should not throw an exception, and while it may not behave as you expect (by using load() that way), it should still run. Perhaps, the code simply cannot find the index.js file?
In any case, @wirth will be able to help you more than I can.

Cheers
Edoardo.

@wirthi
Copy link
Member

wirthi commented Mar 19, 2019

Hi @mahesh-kharat

(thanks @edopalmieri for your answer)

It is working for me. As Edoardo wrote, most likely, the index.js file might not be found, it needs to be in the root of your java app (i.e. in the same directory as your java file, if this uses the default package).

But then, as Edoardo wrote, you need to use pure JavaScript syntax and cannot use Node.js concepts (like export). load will parse and evaluate the file; the prepareMessage function will then be in the global scope, you cannot put it in exports (which does not exist - of course you could provide it from index.js).

index.js (with the variant of providing exports from there):

var exports={};
load('./message.js');
console.log("Display message:"+exports.prepareMessage('World!!!!'));

Cleaner would be if you created the exports object in message.js and returned that. But you have to manually do that, this is not magically happening like in Node.js.

Best,
Christian

@mahesh-kharat
Copy link

mahesh-kharat commented Mar 19, 2019

Thank you for replying @edopalmieri @wirthi

@wirthi I was able to run above example after putting those index.js and message.js in project's default location. Thanks for pointing out.

I wanted to know if we can run "Node modules from jdk 11 using graalvm". In above example I have changed 'require' to 'load' in javascript file to fetch dependency (as it's custom implementation). I have a requirement where I have to use third party dependencies like d3, d3-node and many other where I can't change 'require' to 'load'. In that case will it be possible to execute those node modules from jdk 11 using graalvm? Is there any way of doing it?

@wirthi
Copy link
Member

wirthi commented Mar 19, 2019

Hi @mahesh-kharat

short answer: no, not possible.

long answer: Node.js is a binary application. It cannot trivially be started from within a running JVM. Longer-term, we are working towards enabling such a setup, but it will require some modifications in core Node.js code that we typically want to avoid in order not to break compatibility. The easiest way forward, if that is possible for you, is to use <GraalVM>/bin/node --jvm and start you Java code from there. https://medium.com/graalvm/multi-threaded-java-javascript-language-interoperability-in-graalvm-2f19c1f9c37b can give you some hints how to be do that.

Best,
Christian

@ralleman
Copy link

IMO, being able to run Node modules from within a Java-based Graal application is likely to be a common request. I'm looking into it too. I'm wondering if a project is needed to create Javaised polyfills for the built-in Node modules. If they existed, anyone could run Node modules in their Java apps.

@psanders
Copy link

I had this issue before with Nashorn, and endup using Nodyn's package loader. I created a repo with an example of my approach. You can also see this working in a larger project.

I hope this helps someone!

@jfrantzius
Copy link

Hi @wirthi ,
is there per chance any publicly accessible information about this longer-term work you mentioned? It would be great if it was tracked as a Github issue, so people can get notified on eventual progress :)

On a sidenote, in our particular case we'd just like to be able to require any npm modules from Java, which in turn should be able to use Node API like fs. If this was possible without spinning up a Node.js event loop, that would be even better!

I guess that in the case of a Java web server that wants to make use of npm modules, there probably is no need for a Node.js event loop?

@hitsmaxft
Copy link

hitsmaxft commented Jul 27, 2019

hey @wirthi .

We are currently working on APIs and code examples to do this nicely without running into threading problems. We could already show in experiments how to submit entries back into the Node.js event loop from Java. You want to avoid a scenario where your interop call (e.g., your Java application) blocks the event loop. We will address this in one of our next blog posts on Graal.js.

Is there any code example for message passing from java back to node under node --jvm mode ?

As you mentioned , submit entries back into the Node.js event loop from Java . node thread may receive message from java call by a blocking worker , but it's no an elegant way. ince i've found this in the source code https://github.com/graalvm/graaljs/blob/a6c93a837805b8041fabd59d9f822b3887bc4d66/graal-nodejs/test/graal/unit/javaMessages.js

it('Java can schedule back to the main Node.js event loop using a (blocking) worker', function(done) {

I still look for a postMessage style api for java side to pass new message into node event pool.

for example.

// node side 

javaPort.on("message", handler)

 
//java side 

nodePort.postMesasage("new message")

just like worker_threads modules

@skr316
Copy link

skr316 commented Jun 11, 2020

Is 'require' now supported in Graal Context as per this release ?
https://github.com/graalvm/graaljs/blob/master/docs/user/NodeJSVSJavaScriptContext.md#commonjs-modules-cjs

@wirthi
Copy link
Member

wirthi commented Jun 12, 2020

It is as written in the docu you link:

The js.commonjs-require option provides a built-in require() function that can be used to load Npm-compatible CommonJS modules in a JavaScript Context. Currently, this is an experimental feature not for production usage.

It is available behind a flag, is experimental and provides only partial support (similar to what tools like browserify or webpack provide). E.g., you cannot access the Node.js eventloop that way.

@cvadatta
Copy link

cvadatta commented Oct 6, 2020

Hi! I'm trying to do a Fetch request from js inside a java program. what is the best way to do this?
Can I have a example please!!!!!!!!!

@wdanilo
Copy link

wdanilo commented Feb 8, 2021

The fundamentally broken promise of GraalVM JavaScript interop

Hi! @wirthi, @swaechter, first of all, let me express my astonishment that this ticket was closed and even a much greater astonishment when I learned that lack of support for utilizing the node.js functionality from within languages based on GraalVM is a design decision, at least that is what I read from the @wirthi reply from almost 3 years ago (1 May 2018). The possibility to use npm modules and call node.js functions from GraalVM / languages built on top of the Truffle framework is a fundamental promise of seamless languages interoperability. The whole power of JavaScript is inherently bound to the npm ecosystem, and therefore, the node.js functionality. The JS runtime alone is not really useful for anything serious – what do we need interoperability with JS for if we cannot use any server-side functionality and we cannot use npm packages?

The confusing / misleading promises on the GraalVM website.

Also, I believe that the documentation on the official GraalVM website is very confusing. There are a lot of fragments like these:

image

Or this one:
image

Recently, both our team and some people from our community were stunned that we cannot use node.js modules from within our language, after spending weeks on implementing the JS support, because the website is (at least for us) selling this idea, while being very unclear that GraalVM doesn't support interoperability with JavaScript and Node. It only provides a special Node version that can be used to run JavaScript which can call other languages but not vice versa.

Summary

To sum this up, I believe this ticket should be open and we should consider calling node.js modules from within languages implemented on the Truffle framework as one of the top priorities of the JS interop, as without this functionality, the interop with JS from other languages is not very useful to anything serious.

@wirthi
Copy link
Member

wirthi commented Feb 9, 2021

Hi @wdanilo

You can execute Node.js application. node yourApplication.js will work.

You can interop from your Node.js application to Java and to all Polyglot Truffle languages we support (Ruby, Python, LLVM, etc.) - and then call back to the JavaScript (Node.js) level. Once you started from node, everything is possible.

True, what you cannot do is to interop from a Java application to a Node.js application. Why? Because Node.js is a binary application, not a language. You cannot interop from, say, Ruby to Node.js for the same reason you cannot interop from Ruby to Eclipse, or vi, or Microsoft Word: those are applications that expect that their own launcher is executed, in order to set everything up. So in order to get the (full) support of Node.js - event loop, etc - your application needs to start from the node binary.

There are several workarounds for this limitation, quoted both here above in this ticket an in e.g. #321, or in our documentation at https://github.com/oracle/graaljs/blob/master/docs/user/NodeJSVSJavaScriptContext.md You can package your Node.js application, you can use our experimental module loading support, and you can use the NodeJVM approach to wrap an existing Java Application with the Node launcher.

(Re-)Implementing Node.js in Java so you can REALLY call it as a language is a possibility. It requires some engineering effort, but it is technically feasible. The problem is that you will have to follow up all future changes and include them to this port to ensure compatibility. In our current approach, we take the Node.js source code (mostly) unmodified, which allows for more straightforward updates when a new Node.js version comes out. If anyone is up for this task, we are happy to support you and help you get it done. There are no efforts from our side in this respect currently, because we believe the current approach is good enough to support reasonable applications and the engineering effort of re-implementing the core of Node.js in Java is too high with too big a risk of having to maintain a still-not-fully-compatible port (been there, done that: Project Avatar).

The open ticket we have in this area is #321 - but again, we are not actively working on this solution at the moment. Happy to receive reasonable external contributions, of course.

Best,
Christian

@swaechter
Copy link
Author

swaechter commented Feb 10, 2021

Hey @wdanilo Although I understand your astonishment/frustration I want to share some of my own thoughts about one of your statements:

The possibility to use npm modules and call node.js functions from GraalVM / languages built on top of the Truffle framework is a fundamental promise of seamless languages interoperability

In theory language interoperability (Like using Node.js from a JVM and even vice versa) sounds great: You can share & design code/components in different languages so these components can interop with each other and code can be reused. But this flexibilty has its price: Mainly complexity. In the end I gave up on tools like J2V8 (https://github.com/eclipsesource/J2V8) or GraalVM not because of the tool itself, but because the developer experience suffers. In my Angular server side rendering project I had to develop an own hot reload mechanism for the JS code and the Java application that loads this code (If JS restarts, Java has to reload too). Debugging it accross the language barrier...let's just say difficult. I guess this is just the polyglot nature or in short: There is no seamless languages interoperability as you say (Sorry Graal devs, I still love you for the native-image ).

To sum this up: Rethink if you really need language interoperability. In the case of Angular SSR I switched to a Java backend that communicates via pipe/gRPC mechanism with a standalone Node.js backend that does the SSR (A bit slower, but a lot more flexible). In other situations like Oracle DB with JS support, it helps a lot.

@newk5
Copy link

newk5 commented May 12, 2021

For those familiar with J2V8, there is an alternative called Javet, which provides the same features (and more) and is up to date with Node.js LTS. Allows you to embed a full node.js runtime with support for nodejs modules and nodejs event loop. I recenly found out about it and I can say so far it's been working pretty well and all of the features discussed above are possible. It allows language interoperability from java to node and node to java.

For my use case Java is the host language, and nodejs is the "guest" language, so I couldn't use GraalVM unfortunately, but Javet allowed me to do that.

@gordhosted
Copy link

many (if not most) javascript libraries being published are tied into node and npm ecosystem eg AWS SDK. there needs to be a project to port the built in modules (fs/http/...) of node so a java context can truly run node modules. as pointed out this won't be easy and is a moving target.

it's a shame the JS community decided to mostly code to node.js, essentially forcing node into any js enabled stack. node is almost a de-facto standard at this point and that is at the heart of the problem. js engines without node support are simply not very useful, the "solutions" (javet, wrappers, processbuilder, hacks, ...) are not clean or supported enough for production use and don't give you what you really want - full node support from a java context that's clean and well supported.

no easy answers, hopefully the experimental node support in graalvm improves to try to cover a few of the most used modules like fs, that's the best option. i would love to hear from anyone who has the aws sdk working from a java context. for now it seems you can't use any major cloud vendor sdk, cutting us off from pretty much everything.

@wirthi
Copy link
Member

wirthi commented Aug 27, 2021

@gordhosted I agree to all you wrote.

Just to be clear on one detail:

what you really want - full node support from a java context that's clean and well supported

GraalVM does support this - but you have to start from the node executable and interop to a Java application from there. From then on, everything is possible (both in JavaScript and Java contexts; you have to take care around multi-threading though, see Multithreading.md).

What is not possible right now is to boot a Java application and only then start to moving into Node territory.

@gordhosted
Copy link

gordhosted commented Aug 27, 2021 via email

@newk5
Copy link

newk5 commented Aug 27, 2021

The solution I described above (javet) allows you to do all of that, its a full node.js runtime and allows you to use all node.js libraries (fs,http, etc..). For my use case I am starting a java application that does exactly that, runs node.js snippets.

@gordhosted
Copy link

gordhosted commented Aug 27, 2021 via email

@newk5
Copy link

newk5 commented Aug 27, 2021

Yea I dont think there is a Java.type with Javet, that is true, maybe GraalJS might be more production ready at this point. About the crash, I cant help since I'm not a Javet developer, all I can say is in my application I'm using Javet to run many .js files and I haven't had any crashes, the only JVM crashes I've had before were due to me doing something wrong while setting up the Node.js runtime so it was an issue on my part, but I'd encourage you to contact the lead dev on the discord or gitter channel, he's very active right now, there's usually a new public release each week, he's very helpful and has implemented alot of my suggestions to accomodate my project's needs.

@caoccao
Copy link

caoccao commented Aug 28, 2021

I'm the author of Javet. I'd like to share my 2 cents for your reference.

Javet is Beyond Java.type()

Javet comes with a different mindset from what GraalVM defines. In Javet, you may inject java.lang.Class.class in Node.js, then simply call Class.forName(...) in Node.js to load whatever you want to load. That's a native solution which allows you to even specify the classloader. Following the same way, you may inject File.class or whatever classes. Please refer to this doc for more details.

Why is Javet Designed That Way?

It's simply one word Security. Javet is being used by many applications in production, not a toy or buzzword for promotion. In real world, your application exposes the Node.js runtime to external users who are capable of sending arbitrary scripts and breaking past security boundaries. A built-in Java.type() would be a nightmare to server application developers. Javet gives granular control over any Java API exposed to the Node.js runtime.

Why did You Get That Crash?

If you were trying to load Node.js native modules in Javet without patching them properly, you would experience crashes like that. That's not a Javet problem but the whole Node.js ecosystem problem. E.g. Electron has a dedicated project electron-rebuild which is designed for patching the native modules. Javet has its own patching system as well. If you follow the Javet doc, you will be fine with loading native modules. @newk5 worked with me closing that gap a few months ago. Thank @newk5 for the help.

How is Node.js Embedded in JVM by Javet?

Javet embeds the Node.js itself in JVM instead of reinventing the wheels. So, the Node.js runtime in Javet is genuine. That means anything in Node.js ecosystem is out-of-box in Javet. require(), Promise, fs, vm, etc. all work well. In addition, Javet offers full capability of interoperating with Node.js runtime. Obviously, Javet is able to keep up with Node.js releasing cycle from time to time. In other solutions, you would have to file issues for new Node.js features, then wait months for the mercy from the maintainers. No, Javet keeps that awful experience away from you.

Javet Community Size

Javet has been alive for half a year. Its community is growing up. I always believe it is the right direction that makes a product go further, not the community size. For instance: Oracle Nashorn was branded as A Next-Generation JavaScript Engine for the JVM, but was abandoned. Many Nashorn users are migrating to Javet now. Regarding GraalVM, based on its current design, JavaScript and Java interoperability is not available in native image. I wonder once you were full in GraalVM and wanted to create native image, what would you think about the initial decision on getting onboard to GraalVM? If you have faith in the Javet way, come and join the community so that it won't be a depending-on-one-or-two-developers project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests