Build native application using GraalVM

[ghost]

Description
Renjin appears to prevent building a functioning native application using GraalVM.

Replicate

1. Create a new JavaFX application.
2. Apply the GluonFX plugin.
3. Invoke Renjin.
4. Build a native binary using GraalVM.

See my StackOverflow question for contextual information.

Expected
A native binary is built and can evaluate R expressions within a JavaFX application.

Actual
Using Gluon's GraalVM 22.1, or even latest GraalVM 23.0-dev, this error appears:

Error: Non-reducible loop requires too much duplication. Setting MaxDuplicationFactor to a value higher than 2.0 may resolve this. Call path from entry point to org.renjin.stats.portsrc__.dg7itb_(Ptr, Ptr, Ptr, ...): no path found from entry point to target method.

Additional details
To test the GluonFX plugin, create a Maven project with just a launcher class that uses KeenWrite's uberjar and JavaFX dependencies.

Workaround
It was suggested to use FastR instead (see issue tracker). I'd rather not go down that road because FastR isn't pure Java, which will likely raise more (or different) issues.

[ghost]

Here's a shell script I used when attempting to build a native version of KeenWrite. Uncomment one of the last two lines, depending on whether GraalVM or BellSoft's NIK (recommended) is used to build.

#!/usr/bin/env bash

cd /tmp
git clone https://github.com/DaveJarvis/keenwrite
cd keenwrite
mkdir -p src/main/resources/META-INF/native-image
export LD_LIBRARY_PATH="$(pwd)/nik/lib:$LD_LIBRARY_PATH"

# Gradle and Java must be installed and available via the PATH.
gradle jar

wget "https://download.bell-sw.com/vm/22.3.0/bellsoft-liberica-vm-full-openjdk17.0.5+8-22.3.0+2-linux-amd64.tar.gz"
tar xf *gz
rm *gz
mv bell* nik

# Run the app and use many options.
java \
  -agentlib:native-image-agent=config-output-dir=src/main/resources/META-INF/native-image \
  --add-opens=javafx.controls/javafx.scene.control=ALL-UNNAMED \
  --add-opens=javafx.controls/javafx.scene.control.skin=ALL-UNNAMED \
  --add-opens=javafx.graphics/javafx.scene.text=ALL-UNNAMED \
  --add-opens=javafx.graphics/com.sun.javafx.css=ALL-UNNAMED \
  --add-opens=javafx.graphics/com.sun.javafx.text=ALL-UNNAMED \
  --add-exports=javafx.base/com.sun.javafx.event=ALL-UNNAMED \
  --add-exports=javafx.graphics/com.sun.javafx.application=ALL-UNNAMED \
  --add-exports=javafx.graphics/com.sun.javafx.geom=ALL-UNNAMED \
  --add-exports=javafx.graphics/com.sun.javafx.text=ALL-UNNAMED \
  --add-exports=javafx.graphics/com.sun.javafx.scene=ALL-UNNAMED \
  --add-exports=javafx.graphics/com.sun.javafx.scene.text=ALL-UNNAMED \
  --add-exports=javafx.graphics/com.sun.javafx.scene.traversal=ALL-UNNAMED \
  -jar build/libs/keenwrite.jar $@

# Fails due to missing reflective class lookup.
./nik/bin/native-image \
  --add-modules=javafx.controls,javafx.swing \
  --verbose \
  -H:+ReportExceptionStackTraces \
  --no-fallback \
  --report-unsupported-elements-at-runtime \
  -Djava.awt.headless=false \
  -cp src/main/resources/META-INF/native-image \
  -jar build/libs/keenwrite.jar

# ./graalvm/bin/gu install native-image
# ./graalvm/bin/native-image \

[perNyfelt]

Which version of renjin are you using? Master and latest versions are built on Java 8. There is a Java 11 branch that might be more successful in attempting this.

[ghost]

From https://github.com/DaveJarvis/keenwrite/blob/master/build.gradle:

  // R
  implementation 'org.renjin:renjin-script-engine:3.5-beta76'
  implementation 'org.renjin.cran:rjson:0.2.15-renjin-21'

The application uses Java 19 and JavaFX 19.

Is there a more recent version? (Note: I tried building Renjin, but didn't want to pollute my dev environment with a different GCC version. It'd be great if the dependencies on a particular version could be eliminated.)

[akbertram]

Interesting problem!

The offending class is created by Renjin's C/Fortran to JVM bytecode compiler. One of the recurring challenges has been that this compiler can generate very long methods. Sometimes this because the C or Fortran method is very long. I believe this is the guilty subroutine here:

https://github.com/wch/r-source/blob/431d589d479358a11ace2a0b7443fb519bf0ce05/src/library/stats/src/portsrc.f#L3995

Long functions are not a problem for machine code, but* many tools like OpenJDK's JIT or possibly the native imaging tools simply won't accept this kind of complexity becuase the optimization algorithms they are using have exponential (or worse) running time.

One solution we came up with this is to run the compiler's output through Soot, a JVM-bytecode optimizer. TBH I'm not sure if this is running in the beta3.5 branch.

[akbertram]

You could also try removing this file from the source branch and rebuilding. (or excluding it from the jar file).

This might to lead to failures at runtime if you use these specific functions from the stats package, but should be ok if you're not using this particular part of the stats package.

[ghost]

should be ok if you're not using this particular part of the stats package.

KeenWrite has 10,000 downloads and no telemetry. I don't see a way for me to know whether that part is being used. I'm wondering:

• What would it take to get a version of Renjin released that doesn't have this issue (preferably with no loss in functionality)?
• Do you have build instructions that show how to build Renjin using GCC (4.x?) such that it doesn't install into /usr/local (i.e., non-polluting)?
• Is it possible to verify whether Soot was run on renjin-3.5-beta76? (I suspect it wasn't because if it was, the error shouldn't have appeared? I'm a little unclear on this point.)
• What/Where are the step-by-step instructions that show how to run the compiler's output through Soot?

I tried building Renjin a little while ago and found the steps to be a little convoluted without much in the way of documentation. Also, I really don't want to install an older version of GCC into the system. Installing GCC into $HOME/dev/renjin/gcc would be fine, but I couldn't see an easy way to do so.

(Aside, there seems to be a lot of duplication with respect to the GCC version number, which may make upgrading to a newer GCC version a bit of a chore.)

[perNyfelt]

Set up a virtual environment e.g. using Vagrant. GCC 4.2 is installed there saving your environment from "pollution". Instructions can be found here: https://github.com/bedatadriven/renjin/blob/master/BUILDING.md

I had to make a few changes to get soot to work in Java 11 (j11-beta1 branch). It is integrated into the build i.e. runs automatically upon build, no ned to do anythig.

[ghost]

Thanks @perNyfelt . I'm not confident that the instructions will work on my machine (Arch Linux), nor that they are exhaustive enough for me to successfully create a build, nor do I want to install another virtual environment on my machine.

What would it take to get a version of Renjin released that doesn't have this issue (preferably with no loss in functionality)?

Ideally, I could change the following line to a new version:

implementation 'org.renjin:renjin-script-engine:3.5-beta76'

Thoughts?

[akbertram]

I doubt there is any easy fix here. I'm not super familiar with GrailVM but it sounds like it has some limits that Renjin is going to cross one way or another. Is there a way to exclude Renjin from AOT compilation? I can't imagine Renjin would benefit from native AOT given how much reflection and dynamic code generation is used. Will GraalVM work with custom classloaders for example? Again, I'm not familiar with the technology so I might be wrong here.

…

On Wed, Nov 30, 2022, 19:54 Dave Jarvis ***@***.***> wrote: Thanks @perNyfelt <https://github.com/perNyfelt> . I'm not confident that the instructions will work on my machine (Arch Linux), nor that they are exhaustive enough for me to successfully create a build, nor do I want to install another virtual environment on my machine. What would it take to get a version of Renjin released that doesn't have this issue (preferably with no loss in functionality)? Ideally, I could change the following line to a new version: implementation 'org.renjin:renjin-script-engine:3.5-beta76' Thoughts? — Reply to this email directly, view it on GitHub <#564 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADO5Q7T64QOKOBBPLKK5F3WK6PFZANCNFSM6AAAAAASMWLDEQ> . You are receiving this because you modified the open/close state.Message ID: ***@***.***>

[ghost]

I doubt there is any easy fix here.

Darn. This blocks giving native binaries to users for all platforms. I'll take a look at switching to FastR. Thanks for investigating.

[akbertram]

Let me know what you find, I'm curious now. FastR is part of the GraalVM project, so it should be possible to compile R code to a native image. But it sounds like what you you're interested in is dynamically interpreting R code at runtime as your users include R expressions in text documents, at runtime. That seems by nature something very dynamic and not at all amenable to ahead of time compilation.

[ghost]

• https://medium.com/graalvm/faster-r-with-fastr-4b8db0e0dceb
• https://github.com/graalvm/examples/blob/master/r_java_embedding/src/main/java/com/oracle/truffle/r/fastrembedding/FastREmbeddingDemo.java

Context ctx = Context.newBuilder("R").allowAllAccess(true).build();
ctx.eval("R", "sum").execute(new int[] {1,2,3});

It looks like dynamic execution is possible.