Importing (some of) Standard.Base works from NI runner #9866

JaroslavTulach · 2024-05-06T15:37:48Z

Pull Request Description

part of Standard.Base is compiled into runner executable and can IO.println fast
proves Investigate compilation of Standard.Base library with Native Image #9774 is viable
achieved by updating list of classes needing reflection by re-running the NI agent for enso --run test/Base_Tests
the updated configuration .json files are part of this PR
including JAR files of Standard.Base library on -cp when building the engine/runner/buildNativeImage
got ENSO_JAVA=espresso ./runner working again by avoiding java.desktop via jlink
that requires changes to SnakeYaml - send to upsteam
to unblock this PR two Apache licensed SnakeYaml files are temporarily copied to our repository

Checklist

Please ensure that the following checklist has been satisfied before submitting the PR:

All code follows the
Scala,
Java,
Integration test have been updated to use Standard.Base library

…erge-dir=./engine/runner/src/main/resources/META-INF/native-image/org/enso/runner" enso --run test/Base_Tests/

…e/runner

engine/runner/src/test/resources/Factorial.enso

Akirathan · 2024-05-06T16:55:41Z

Are you going to experiment with other standard libraries, like Standard.Table as well in this PR? Or are you going to postpone other libs in different PRs? In any case, it would be nice to start increasing test coverage of native-image-produced engine-runner. We used to have only Factorial.enso that did not import anything from Standard.Base.

build.sbt

JaroslavTulach · 2024-05-07T04:31:28Z

Are you going to experiment with other standard libraries, like Standard.Table as well in this PR? Or are you going to postpone other libs in different PRs?

If this PR gets green, I'd like to integrated it as it improves the usability a lot. However I don't claim this is a fix for #9774. The work will continue. I still want to create a Feature that will scan a library for polyglot java import and register such classes for reflection automatically.

In any case, it would be nice to start increasing test coverage of native-image-produced engine-runner.

Ideally, when we train the native image agent on test/Base_Tests we should be able to run the tests on NI version. Right now something is broken. We want to fix it.

We used to have only Factorial.enso that did not import anything from Standard.Base.

Now we have Factorial.enso that imports something from Standard.Base.

JaroslavTulach · 2024-05-07T07:03:59Z

If this PR gets green

All's well including Factorial.enso invocation with regular NI runner, but...

there is a failure of Factorial.enso invocation with Espresso built NI runner.

Looks like our Espresso support in NI is broken.

hubertp

Rejecting (for now) because I don't want this PR to accidentally get in with the wrong sbt definition.

build.sbt

project/NativeImage.scala

…Espresso

…run without java.desktop

JaroslavTulach · 2024-05-15T03:43:39Z

.../runtime-instrument-common/src/main/java/org/yaml/snakeyaml/introspector/MethodProperty.java

+ */
+package org.yaml.snakeyaml.introspector;
+
+// remove once https://github.com/snakeyaml/snakeyaml/pull/12 gets integrated


Remove this file once snakeyaml/snakeyaml#12 gets integrated.

E.g. when upgrading to SnakeYaml 2.3 to be released in ~August. Tracked as

Use SnakeYAML for YAML parsing (instead of circe) #9113

engine/runner/src/main/java/org/enso/runner/ContextFactory.java

project/GraalVM.scala

project/NativeImage.scala

…Base9774

JaroslavTulach · 2024-05-15T12:38:28Z

engine/runner/src/test/resources/Factorial.enso

-    fac number
+main number=5 =
+    v = fac number
+    IO.println v


Using Standard.Base library calls like IO.println

JaroslavTulach · 2024-05-15T12:57:40Z

The important parts of this PR are green. Let's get ready for the integration.

Factorial.enso is now using Standard.Base APIs
./built-distribution/enso-engine-*/enso-*/bin/enso --run engine/runner/src/test/resources/Factorial.enso runs in 3225 ms
./runner --run Factorial.enso uses classes compiled into native executable and runs in 1221 ms
ENSO_JAVA=espresso ./runner --run Factorial.enso uses Espresso to execute Java code and it runs in 2859 ms

We have experimentally demonstrated compiling Standard libraries with native-image gives us three times speedup when starting and allows us to use Standard.Base from ./runner executable to begin with. Let's integrate, unless there are objections!

hubertp

I don't like the changes in sbt. It very much doesn't feel like sbt'ish (not your fault) and adding some shell scripts into the project directory does not help things.

Also, what I'm missing in the source code comments (or build documentation) is the concrete command on how to generate all native image config files. I know that can be found somehow via NI documentation but I'd like to reduce friction points.

project/smalljdk.sh

- ditch bash script in favour of platform agnostic solution - use proper dependencies between tasks to avoid dummy function calls

hubertp

I had a go at SBT and the changes are now in a more acceptable state. I know that the previous solution worked but the build setup easily starts to become a spaghetti monster when one starts to add random bash scripts.

@JaroslavTulach please add docs with an exact command on how to regenerate NI configs.

JaroslavTulach · 2024-05-17T12:36:08Z

build.sbt

+          )
+      }
+
+      val exec =


Exec in Scala! Amazing, thank you @hubertp!

JaroslavTulach · 2024-05-17T12:42:09Z

I had a go at SBT and the changes are now in a more acceptable state.

Thank you Hubert.

@JaroslavTulach please add docs with an exact command on how to regenerate NI configs.

This commit contains the information 8cc1b2d

JaroslavTulach · 2024-06-20T08:47:24Z

To regenerated Substrate VM reflection metadata as this commit did, use:

JAVA_OPTS="-agentlib:native-image-agent=config-merge-dir=./engine/runner/src/main/resources/META-INF/native-image/org/enso/runner" enso --run test/Base_Tests/

JaroslavTulach added 4 commits May 6, 2024 17:27

Adjust path to engine/runner

dbd5253

Regenerated with env JAVA_OPTS="-agentlib:native-image-agent=config-m…

8cc1b2d

…erge-dir=./engine/runner/src/main/resources/META-INF/native-image/org/enso/runner" enso --run test/Base_Tests/

Include all Standard/Base JARs on classpath when building NI of engin…

a62f747

…e/runner

Importing from Standard.Base works in NI mode again

b9f8fec

JaroslavTulach added the CI: No changelog needed Do not require a changelog entry for this PR. label May 6, 2024

JaroslavTulach self-assigned this May 6, 2024

JaroslavTulach requested review from 4e6, hubertp and Akirathan as code owners May 6, 2024 15:37

Akirathan reviewed May 6, 2024

View reviewed changes

engine/runner/src/test/resources/Factorial.enso Show resolved Hide resolved

JaroslavTulach added the CI: Clean build required CI runners will be cleaned before and after this PR is built. label May 6, 2024

JaroslavTulach commented May 6, 2024

View reviewed changes

build.sbt Outdated Show resolved Hide resolved

enso-bot bot mentioned this pull request May 7, 2024

Expressions not being run if indented by 1 more space than current indentation level #9419

Open

base-polyglot-root may not exist

46b48d6

JaroslavTulach added 2 commits May 7, 2024 06:59

Compute additional path at invocation of the buildNativeImage task

2556a2c

Useful perf and debug switches

0d24b3e

hubertp requested changes May 7, 2024

View reviewed changes

build.sbt Outdated Show resolved Hide resolved

Akirathan reviewed May 7, 2024

View reviewed changes

build.sbt Outdated Show resolved Hide resolved

Akirathan reviewed May 7, 2024

View reviewed changes

project/NativeImage.scala Outdated Show resolved Hide resolved

enso-bot bot mentioned this pull request May 8, 2024

CompilerError isn't syntax error (required by Truffle TCK) #8735

Closed

JaroslavTulach mentioned this pull request May 8, 2024

Too heavyweight org.graalvm.shadowed.icu4j oracle/graal#8889

Closed

JaroslavTulach added 6 commits May 8, 2024 08:46

Merging removal of java.desktop to this branch

0d7a11b

Configuration for NI debugging of the runner

9d75c72

NI is built from -cp - we need other method of detecting presence of …

c507a47

…Espresso

Script to generate 'small JDK' with native-image in jdk directory

2473812

Exclude Graal.js and Graal Python when using Espresso, as they don't …

1933c77

…run without java.desktop

Avoid special Graal.js settings

593e9b0

Patching SnakeYaml to run without java.desktop

e20308b

JaroslavTulach commented May 15, 2024

View reviewed changes

JaroslavTulach added 2 commits May 15, 2024 07:02

Copy all JARs to polyglot/java directory in Espresso mode

38fca73

Shelve SnakeYaml patches into a single project

87a15e0

Akirathan reviewed May 15, 2024

View reviewed changes

engine/runner/src/main/java/org/enso/runner/ContextFactory.java Show resolved Hide resolved

project/GraalVM.scala Outdated Show resolved Hide resolved

project/NativeImage.scala Outdated Show resolved Hide resolved

JaroslavTulach added 2 commits May 15, 2024 12:11

Build smalljdk before buildNativeImage

d29c9a1

Merge remote-tracking branch 'origin/develop' into wip/jtulach/NIwith…

e4cb551

…Base9774

JaroslavTulach requested a review from hubertp May 15, 2024 12:24

JaroslavTulach commented May 15, 2024

View reviewed changes

Hide access to various Pkgs constants

2e3b0ed

JaroslavTulach mentioned this pull request May 15, 2024

Extension methods for Java classes #9129

Closed

3 tasks

JaroslavTulach requested a review from Akirathan May 15, 2024 14:18

JaroslavTulach mentioned this pull request May 15, 2024

Use SnakeYAML for YAML parsing (instead of circe) #9113

Closed

enso-bot bot mentioned this pull request May 16, 2024

Stack trace when Enso script finishes with a Panic is upside down #9934

Closed

hubertp requested changes May 16, 2024

View reviewed changes

project/smalljdk.sh Outdated Show resolved Hide resolved

hubertp added 4 commits May 16, 2024 17:04

Rewrite SBT support

4d90834

- ditch bash script in favour of platform agnostic solution - use proper dependencies between tasks to avoid dummy function calls

fix NPE

9cf1543

Correctly detect new javaHome for NI

c627c0f

nit

58ed0bc

hubertp approved these changes May 17, 2024

View reviewed changes

JaroslavTulach commented May 17, 2024

View reviewed changes

build.sbt

)

}

val exec =

Copy link

Member Author

JaroslavTulach May 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exec in Scala! Amazing, thank you @hubertp!

JaroslavTulach merged commit 5c06535 into develop May 17, 2024
36 checks passed

JaroslavTulach deleted the wip/jtulach/NIwithBase9774 branch May 17, 2024 12:42

enso-bot bot mentioned this pull request May 18, 2024

Support cycles in the serialized IR structure #9361

Closed

JaroslavTulach mentioned this pull request May 29, 2024

Run whole Standard.Base in Native Image ./runner #10126

Closed

JaroslavTulach mentioned this pull request Oct 10, 2024

Package libraries Java code as Java Modules #10714

Draft

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Importing (some of) Standard.Base works from NI runner #9866

Importing (some of) Standard.Base works from NI runner #9866

JaroslavTulach commented May 6, 2024 •

edited

Loading

Akirathan commented May 6, 2024

JaroslavTulach commented May 7, 2024

JaroslavTulach commented May 7, 2024

hubertp left a comment

JaroslavTulach May 15, 2024

JaroslavTulach May 20, 2024

JaroslavTulach May 15, 2024

JaroslavTulach commented May 15, 2024 •

edited

Loading

hubertp left a comment •

edited

Loading

hubertp left a comment

JaroslavTulach May 17, 2024

JaroslavTulach commented May 17, 2024

JaroslavTulach commented Jun 20, 2024

Importing (some of) Standard.Base works from NI runner #9866

Importing (some of) Standard.Base works from NI runner #9866

Conversation

JaroslavTulach commented May 6, 2024 • edited Loading

Pull Request Description

Checklist

Akirathan commented May 6, 2024

JaroslavTulach commented May 7, 2024

JaroslavTulach commented May 7, 2024

hubertp left a comment

Choose a reason for hiding this comment

JaroslavTulach May 15, 2024

Choose a reason for hiding this comment

JaroslavTulach May 20, 2024

Choose a reason for hiding this comment

JaroslavTulach May 15, 2024

Choose a reason for hiding this comment

JaroslavTulach commented May 15, 2024 • edited Loading

hubertp left a comment • edited Loading

Choose a reason for hiding this comment

hubertp left a comment

Choose a reason for hiding this comment

JaroslavTulach May 17, 2024

Choose a reason for hiding this comment

JaroslavTulach commented May 17, 2024

JaroslavTulach commented Jun 20, 2024

JaroslavTulach commented May 6, 2024 •

edited

Loading

JaroslavTulach commented May 15, 2024 •

edited

Loading

hubertp left a comment •

edited

Loading