Turn all GraalVM Languages artifacts into standard Maven dependencies for Polyglot Embedders

[chumer]

TL;DR

Currently, to use GraalVM languages as a polyglot embedder, you have to install languages using the GraalVM Updater tool (gu).
In the future, we want to ship GraalPy, Graal.js, TruffleRuby, and other GraalVM languages entirely as Maven artifacts.
Polyglot embedders will need to depend on those artifacts using the application module path.

Goals

• The configuration of the application module path should be the same, whether you run on OpenJDK, Oracle JDK, or a GraalVM JDK.
• Languages can be picked up from the application module path and have priority over languages installed using GraalVM Updater.
• JVM built-in tools that support module dependencies like jlink work and are tested with GraalVM languages.
• To prevent accidental use of internal APIs, Truffle and language implementations are isolated from the embedder using Java module-based isolation. This means that all GraalVM languages are migrated to Java modules.
• All Truffle artifacts, including non-open-source parts, are published as Maven artifacts.
• Backward and forward compatibility of all Truffle artifacts to the latest JDK LTS, starting with JDK 21.
• Truffle and its languages will continue to be versioned using a year-based versioning scheme. For example, GraalVM for JDK 21 will be compatible with version 23.1 of the GraalVM languages.

Non-Goals

• Implement extensions for standalone builds using Maven artifacts, e.g. to install new languages.
• Deprecate gu (see [GR-46219] Remove the GraalVM Updater #6855).

[chumer]

We’re looking for feedback from Java developers who utilize the org.graalvm.polyglot API to embed languages such as JavaScript, Python, Ruby, Espresso, WebAssembly, LLVM, etc. within Java. Version 23.1 (September release) is set to introduce some significant changes to how embedding in Java applications works.
In previous versions, you had to install a language with gu, our Graal Updater tool (e.g., gu install python). Post 23.1, this will no longer be possible. You’ll need to download the artifacts from a Maven Repository and run them using the module-path instead.

To clarify how this will work, here’s a brief example for a new Maven configuration that includes Python.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.yourcorp.yourapp</groupId>
    <artifactId>yourartefact</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>
    <dependencies>
        <dependency> <!-- We also move from graal-sdk => polyglot --> 
            <groupId>org.graalvm.polyglot</groupId>
            <artifactId>polyglot</artifactId>
            <version>23.1.0</version>
        </dependency>
        <dependency> <!-- you can also depend on org.graalvm.polyglot:languages for all languages --> 
            <groupId>org.graalvm.python</groupId>
            <artifactId>python</artifactId>
            <version>23.1.0</version>
            <type>pom</type>
        </dependency>
        <dependency> <!-- all tools we include in Oracle GraalVM --> 
            <groupId>org.graalvm.polyglot</groupId>
            <artifactId>tools</artifactId>
            <version>23.1.0</version>
            <type>pom</type>
        </dependency>
    </dependencies>
</project>

Note: If you want Open Source versions of all these components you need to change the artefact names to python-community and tools-community.

Note: We also broke the org.graalvm.sdk module (graal-sdk.jar) into several parts: nativeimage, polyglot, word and collections. We will still deploy a deprecated version of this module in the next release to not break your code.

I think there are some advantages to moving to Maven dependencies:

• You can use the dependency mechanism (Maven) that you are probably already using for other libraries. (Please comment if you are not)
• You can use the same module-path/Maven configuration on all JDKs: GraalVM/Oracle/Open JDK.
• You can upgrade the JDK separately from polyglot languages as long as it is still compatible with the JDK (the exact details are still tbd).
• The module path configuration is directly portable between the java command and native-image command.
• The Java jlink tool will work with all the languages modules.
• All the Maven dependencies will be portable across platforms (yes you will get a portable python runtime with this!)

And there are also some disadvantages:

• This will change the way you used languages like Python in GraalVM JDK in previous releases.
• It might be slightly more complicated to download all dependencies and put the on the module-path, than just calling gu install python.
• Languages and Truffle need to be put on the Java module-path. Note: your application can stay on the class-path!

We’d appreciate your thoughts on this change, in particular on the following points:

Question 1) In order to run with optimized/compiled Truffle, you’ll need to download all language modules from Maven and put them on the module-path. If you put them on the classpath, the “optimized” mode will no longer function.
How challenging would it be to put Maven artifacts on the module-path instead of the classpath in your system/infrastructure? Does your infrastructure already support this?

Question 2) For users who build native images of their Java embedding, the previous --macro:truffle and --language:python flags will no longer be available. You’ll need to put the languages on the module-path for them to be included, just like when running the Java application. If you were using those flags, this change might disrupt your build process.
How difficult would it be to transition from --language:python usage to the module-path approach?

Question 3) For users deploying their own class-loaders: At present, you can only load the optimized/compiled Truffle runtime modules once per JVM, even if they are in completely separate class loaders. Attempting to load the optimized Truffle runtime twice per JVM will result in a failed boot. This is due to a current limitation in how our newly decoupled integration with the compiler works. We plan to remove this restriction in the future, but it’s unlikely to happen by 23.1. You can still use interpreter-only Truffle or share the Truffle/polyglot modules in a common parent class loader.
Is this issue relevant to you? If so, can you share Truffle in a common parent class loader starting with 23.1?
Feel free to comment below with your own questions, comments, or concerns on this topic.

For more details, please also see the slack conversation:
https://graalvm.slack.com/archives/CNBFR78F9/p1691241119810719
You can get an invite here:
https://www.graalvm.org/slack-invitation/

[sgammon]

@chumer we are trying this new style in Github Actions, with the dev version of the VM

It's warning us that we can't install via gu anymore (makes sense):

Warning: Unable to install component(s): 'js,wasm,ruby,python,llvm,regex,icu4j,trufflejson'. The latest GraalVM dev builds and the upcoming GraalVM for JDK 21 no longer include the GraalVM Updater: https://github.com/oracle/graal/issues/6855

Are Maven coordinates up yet? I searched and couldn't find any yet

[chumer]

@sgammon Maven coordinates go up today with the release of 23.1.
The sample Maven repo is already up: https://github.com/graalvm/polyglot-embedding-demo

[chumer]

We have implemented this for the GraalVM for JDK 21 / Polyglot 23.1 release.
We have also published a blog post to explain the change in detail:
https://medium.com/graalvm/truffle-unchained-13887b77b62c

[sbkohel]

What happened to R? We used to be able to use gu to add it (which the documentation still states). I have not found any jar to install R.

[sbkohel]

Also it appears there is a service name collision when trying to implement multiple languages in an uber jar. I can run scripts successfully within an IDE, but when I create an uber jar, and run the same script it throws an error a language was not implemented

java.lang.ExceptionInInitializerError: Exception java.lang.IllegalStateException: No language and polyglot implementation was found on the class-path. Make sure at last one language is added on the class-path. If you put a language on the class-path and you encounter this error then there could be a problem with isolated class loading. Use -Dpolyglotimpl.TraceClassPathIsolation=true to debug class loader islation problems. For best performance it is recommended to use polyglot from the module-path instead of the class-path.

[JaroslavTulach]

multiple languages in an uber jar.

Putting multiple modular JAR files into single uber JAR has an obvious problem: module-info.class should be merged. Unless I am mistaken, no such merging implementation exists yet.

`java.lang.ExceptionInInitializerError: Exception java.lang.IllegalStateException: No language and polyglot implementation was found on the class-path. Make sure at last one language is added on the class-path.

I interpret it as a sign of missing module-info. Try to write one manually or (better imo) give up on uber JAR.

[chumer]

What happened to R?

Since 23.0 Oracle is no longer providing builds of FastR. The community is welcome to pick this effort up and provide builds for it. Internally we are still maintaining it to keep building and provide limited testing with the latest release. For 23.1 there is still more work to be done to make it compatible again with Truffle Unchained.

[chumer]

@sbkohel I wrote a comment on uber jars here: oracle/graaljs#764 (comment)

[frank-montyne]

Can java be used as a scripting language in Graalvm If I add

<dependency>
	    <groupId>org.graalvm.polyglot</groupId>
	    <artifactId>java</artifactId>
            <version>23.1.2</version>
	    <type>pom</type>
</dependency>

When I create a Context
engine = Engine.newBuilder().build();
context = Context.newBuilder(languageId)
.engine(engine)
.allowAllAccess(true)
.allowHostAccess(HostAccess.ALL)
.allowCreateProcess(true)
.allowCreateThread(true)
.allowIO(IOAccess.ALL)
.allowNativeAccess(true)
.allowHostClassLookup((Predicate) className -> true)
.allowHostClassLoading(true)
.build();

I get "Could not find internal resources for configuration native." when getting the bindings.

bindings = context.getBindings(languageId);

Is that normal?