Try native image then fallback to pure java version #717

gzm55 · 2022-10-16T11:46:52Z

The new entry script does some light weight checking to select the first available one from:

native image
pure java version

To control the fallback logic, a new environment switch MVND_CLIENT is added. Default auto enables the fallback logic, set to native or jvm to force select the native image or pure java version.

Checking of native image is mainly based on the file name, and on Linux, there is another ldd check to ensure glibc version is OK.

gnodet · 2022-10-16T12:00:11Z

I like the fallback idea, though I wonder if we should move that fallback code directly into the mvnd.sh script, which would delegate to the native binary if possible, eventually renaming mvnd to mvnd-native.
Also, we may want to investigate a way to run the client on a different JVM that the daemon. We could add two mvnd environment variables, one which would point to the default client JVM, and another that would define whether we want to always use that JVM for the client, auto switch to that JVM is the environment one is that JDK 11 compatible, or never use it. The embedded maven would then only be used if the former is not defined or if the later is set to never.

gzm55 · 2022-10-16T12:14:40Z

A more complex fallback strategy sounds OK, we could define the rules clearly. And the main entrypoint well known is now $MVND_HOME/bin/mvnd. To keep the compatibility, can we move the mvnd.sh/mvnd.cmd (with fallback function) to mvnd or mvnd.cmd, and the native binary are moved to mvnd-native/mvnd-native.exe?

gnodet · 2022-10-16T12:20:53Z

A more complex fallback strategy sounds OK, we could define the rules clearly. And the main entrypoint well known is now $MVND_HOME/bin/mvnd. To keep the compatibility, can we move the mvnd.sh/mvnd.cmd (with fallback function) to mvnd or mvnd.cmd, and the native binary are moved to mvnd-native/mvnd-native.exe?

Sounds good !

gzm55 · 2022-10-17T02:36:39Z

Also, we may want to investigate a way to run the client on a different JVM that the daemon. We could add two mvnd environment variables, one which would point to the default client JVM, and another that would define whether we want to always use that JVM for the client, auto switch to that JVM is the environment one is that JDK 11 compatible, or never use it. The embedded maven would then only be used if the former is not defined or if the later is set to never.

@gnodet do you mean add two new env: MVND_CLIENT_JAVA_HOME and MVND_CLIENT_JAVA_HOME_REQUIRE, and when the native version fails, we select like this:

(MVND_CLIENT_JAVA_HOME is unset) or (MVND_CLIENT_JAVA_HOME_REQUIRE=never):
- JAVA_HOME/bin/java or java is JDK 11+: mvnd with os java
- JAVA_HOME/bin/java or java is old JDK: embeded maven with os java
MVND_CLIENT_JAVA_HOME is set, and MVND_CLIENT_JAVA_HOME_REQUIRE=always
- always run mvnd with MVND_CLIENT_JAVA_HOME/bin/java even if MVND_CLIENT_JAVA_HOME points to an old JDK
MVND_CLIENT_JAVA_HOME is set, and MVND_CLIENT_JAVA_HOME_REQUIRE=auto (default value)
- MVND_CLIENT_JAVA_HOME/bin/java is JDK 11+: mvnd with MVND_CLIENT_JAVA_HOME/bin/java
- JAVA_HOME/bin/java or java is JDK 11+: mvnd with os java
- JAVA_HOME/bin/java or java is old JDK: embeded maven with os java

ppalaga

I'd vote for evaluating the performance impact of wrapping mvnd binary inside a shell script before merging this.

gzm55 · 2022-10-18T14:25:33Z

I'd vote for evaluating the performance impact of wrapping mvnd binary inside a shell script before merging this.

when change the native image names, the checking step of native can be replaced by a test -x mvnd-native-<os_arch>, this would has no performance issue. In another pr #722 , the JDK compatible checking step will be also omitted, auto fallback is done by a multi-release jar.

ppalaga · 2022-10-18T19:41:55Z

I wonder what is your goal here, @gzm55 ? You want to have single package installable and runnable on any host of any arch and OS combination?

gzm55 · 2022-10-19T00:05:05Z

I wonder what is your goal here, @gzm55 ? You want to have single package installable and runnable on any host of any arch and OS combination?

Goal:

single official way to startup mvnd without knowing the internal structure. now there is only 4 prebuild binaries which depend on hardware and glibc 2.14, and the users have to switch to another script if he fails at first time.
treat mvnd as a maven distribution which runs faster than or at least same as, the original maven, then the user do not need to install another maven.

ppalaga · 2022-10-19T12:55:28Z

Goal:

single official way to startup mvnd without knowing the internal structure. now there is only 4 prebuild binaries which depend on hardware and glibc 2.14, and the users have to switch to another script if he fails at first time.

treat mvnd as a maven distribution which runs faster than or at least same as, the original maven, then the user do not need to install another maven.

I'd be fine with all of that as long as installing the native binary as the entry point stays possible. For performance reasons package managers like SDKMAN should still prefer doing that if a native client is available for the given platform. What do others think?

gzm55 · 2022-10-19T14:10:40Z

I'd be fine with all of that as long as installing the native binary as the entry point stays possible. For performance reasons package managers like SDKMAN should still prefer doing that if a native client is available for the given platform. What do others think?

we can use a env flag to force the entry script always use the native binary

ppalaga · 2022-10-20T08:06:13Z

Hm... am I the only one thinking that the mvnd native executable should be the entry point (rather than a wrapper shell script) on platforms for which we have mvnd native executable? @gnodet @hboutemy @cstamas @michael-o what's your opinion?

ppalaga · 2022-10-21T11:51:57Z

These changes seem to be addressing what I'd call install time concerns. Whether the given build of mvnd native executable can run on the current machine can be decided once at install time. Why do we need to run all those checks anew on every invocation of mvnd?

gnodet · 2022-10-21T14:17:41Z

These changes seem to be addressing what I'd call install time concerns. Whether the given build of mvnd native executable can run on the current machine can be decided once at install time. Why do we need to run all those checks anew on every invocation of mvnd?

Because we don't have any installer... ? In particular, people often just unzip the tarball / zip.
https://github.com/apache/camel/blob/main/.github/actions/install-mvnd/action.yml
An alternative would be to provide an install script from a known location that would download the latest version, unzip and do the checks maybe ...

gzm55 · 2022-10-21T15:52:06Z

These changes seem to be addressing what I'd call install time concerns. Whether the given build of mvnd native executable can run on the current machine can be decided once at install time. Why do we need to run all those checks anew on every invocation of mvnd?

how about we do a persist optimization at the real installing time for each package manager? almost every package manager has a post installing step, in which we can check via mvnd-native --status, if passed, then mv -f mvnd-native mvnd to overwrite the generic entry script.

ppalaga · 2022-10-21T17:29:29Z

I think both proposals boil down to having a reusable piece of shell/PowerShell for linking/moving mvnd to the right file. I like that idea. @gnodet variant would also be able to download the zip.

gzm55 · 2022-10-23T17:13:55Z

@ppalaga i add a post-install script bin/mvnd-persist-native for detecting and moving the native mage as the default entry

1. rename native binary to 'mvnd-native-<os>-<arch>' 2. add environment switch MVND_ENTRY_FALLBACK, default 'true' enables the fallback logic, set to 'false' to force execute the native mvnd. 3. rename mvnd.sh to mvnd

…ault entry path

gzm55 · 2023-01-10T17:22:35Z

I'd vote for evaluating the performance impact of wrapping mvnd binary inside a shell script before merging this.

@ppalaga I did some basic profiling, comparing to directly execute native binaries:

MVND_ENTRY_FALLBACK=false: fallback script increases about 50ms on linux and macos
MVND_ENTRY_FALLBACK=true: fallback script increases about 60ms on macos
MVND_ENTRY_FALLBACK=true: fallback script increases about 120ms on linux

ppalaga · 2023-01-10T23:31:47Z

I'd dare to vote against making mvnd to point at a shell script by default. It is adding indirection and performance penalties on major platforms for which we have proper native executables, thus worsening both the user experience for the majority of users and reliable testability of the default setup.

I'd be open to accepting the following alternatives that I think also satisfy the original original requirement of having a universal single entry mvnd:

Enhance the existing shell scripts to fall back to native executables (if available and working on the current platform) without renaming them.
Provide install scripts that replace the native mvnd with a shell script if the native mvnd does not work on or does not match the current platform.

What do others think?

gzm55 · 2023-01-11T01:22:27Z

Enhance the existing shell scripts to fall back to native executables (if available and working on the current platform) without renaming them.

Provide install scripts that replace the native mvnd with a shell script if the native mvnd does not work on or does not match the current platform.

Hi @ppalaga , I would like to make this pr focus on the 1st point, implementing a proper fallback logic in the script entrypoint of the current file layout (mvnd.sh and mvnd.cmd). And create another issue/pr for the 2nd point to discuss and dev the post installing process.

gzm55 · 2023-01-11T09:46:37Z

@ppalaga @gnodet

Enhance the existing shell scripts to fall back to native executables (if available and working on the current platform) without renaming them.

done in this pr

Provide install scripts that replace the native mvnd with a shell script if the native mvnd does not work on or does not match the current platform.

record in the issue #771

dist/src/main/distro/bin/mvnd.cmd

ppalaga

Thanks looks great! I left some minor comment text suggestions inline.

dist/src/main/distro/bin/mvnd.cmd

dist/src/main/distro/bin/mvnd.sh

ppalaga

Great work, thanks a lot, @gzm55!

gnodet · 2023-01-18T14:42:51Z

dist/src/main/provisio/maven-distro.xml

@@ -97,6 +97,7 @@
            <include>mvnd</include>
            <include>mvnd.exe</include>
        </directory>
+        <file touch="platform-${os.detected.name}-${os.detected.arch}"/>


What is the purpose of this line ?

the flag file is used to detect whether the bin/mvnd is built for the current os and arch.

* Add script mvnd-auto to auto select native or pure java mvnd * Move fallback logic into main entry script 1. rename native binary to 'mvnd-native-<os>-<arch>' 2. add environment switch MVND_ENTRY_FALLBACK, default 'true' enables the fallback logic, set to 'false' to force execute the native mvnd. 3. rename mvnd.sh to mvnd * change entry name on windows * Add script mvnd-persist-native for moving the native image to the default entry path * improve platform detect * fix error on dash * rollback default entry to the native image * use MVND_CLIENT switch to control the selection of mvnd client * improve comment docs as suggestion

gzm55 changed the title ~~add script mvnd-auto to a proper mvnd~~ Add script mvnd-auto to a proper mvnd Oct 16, 2022

ppalaga requested changes Oct 18, 2022

View reviewed changes

gzm55 changed the title ~~Add script mvnd-auto to a proper mvnd~~ Draft: Add script mvnd-auto to a proper mvnd Oct 18, 2022

gzm55 mentioned this pull request Oct 20, 2022

Use muti-release jar to fallback mvnd-client to original maven #722

Merged

gzm55 force-pushed the work/entry-one branch from 8f8806b to 7c2349a Compare October 20, 2022 16:55

gzm55 changed the title ~~Draft: Add script mvnd-auto to a proper mvnd~~ Draft: Try native image then fallback to pure java version Oct 20, 2022

gzm55 changed the title ~~Draft: Try native image then fallback to pure java version~~ Try native image then fallback to pure java version Oct 21, 2022

gzm55 and others added 5 commits January 10, 2023 15:22

Add script mvnd-auto to auto select native or pure java mvnd

5c07639

Move fallback logic into main entry script

29ecd01

1. rename native binary to 'mvnd-native-<os>-<arch>' 2. add environment switch MVND_ENTRY_FALLBACK, default 'true' enables the fallback logic, set to 'false' to force execute the native mvnd. 3. rename mvnd.sh to mvnd

change entry name on windows

9e24715

Add script mvnd-persist-native for moving the native image to the def…

ceff591

…ault entry path

improve platform detect

6e4cad7

gzm55 force-pushed the work/entry-one branch from f34dd9e to 6e4cad7 Compare January 10, 2023 09:53

fix error on dash

b8094bd

gzm55 requested a review from ppalaga January 10, 2023 17:23

rollback default entry to the native image

1c3d4fa

gzm55 mentioned this pull request Jan 11, 2023

use script mvnd.{sh,cmd} as default entrypoint #771

Open

ppalaga reviewed Jan 11, 2023

View reviewed changes

dist/src/main/distro/bin/mvnd.cmd Outdated Show resolved Hide resolved

use MVND_CLIENT switch to control the selection of mvnd client

4382677

ppalaga reviewed Jan 12, 2023

View reviewed changes

dist/src/main/distro/bin/mvnd.cmd Outdated Show resolved Hide resolved

dist/src/main/distro/bin/mvnd.sh Outdated Show resolved Hide resolved

improve comment docs as suggestion

b6a5e8c

ppalaga approved these changes Jan 16, 2023

View reviewed changes

gnodet self-requested a review January 16, 2023 08:15

gnodet reviewed Jan 18, 2023

View reviewed changes

gnodet merged commit 6f28a18 into apache:master Jan 24, 2023

gnodet mentioned this pull request Jan 24, 2023

runtime jdk requirement of java client (via mvnd.sh) should be aligned to JDK8 #669

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Try native image then fallback to pure java version #717

Try native image then fallback to pure java version #717

gzm55 commented Oct 16, 2022 •

edited

gnodet commented Oct 16, 2022

gzm55 commented Oct 16, 2022 •

edited

gnodet commented Oct 16, 2022

gzm55 commented Oct 17, 2022 •

edited

ppalaga left a comment

gzm55 commented Oct 18, 2022 •

edited

ppalaga commented Oct 18, 2022

gzm55 commented Oct 19, 2022

ppalaga commented Oct 19, 2022

gzm55 commented Oct 19, 2022

ppalaga commented Oct 20, 2022 •

edited

ppalaga commented Oct 21, 2022

gnodet commented Oct 21, 2022

gzm55 commented Oct 21, 2022

ppalaga commented Oct 21, 2022

gzm55 commented Oct 23, 2022

gzm55 commented Jan 10, 2023

ppalaga commented Jan 10, 2023

gzm55 commented Jan 11, 2023

gzm55 commented Jan 11, 2023

ppalaga left a comment

ppalaga left a comment

gnodet Jan 18, 2023

gzm55 Jan 18, 2023

Try native image then fallback to pure java version #717

Try native image then fallback to pure java version #717

Conversation

gzm55 commented Oct 16, 2022 • edited

gnodet commented Oct 16, 2022

gzm55 commented Oct 16, 2022 • edited

gnodet commented Oct 16, 2022

gzm55 commented Oct 17, 2022 • edited

ppalaga left a comment

Choose a reason for hiding this comment

gzm55 commented Oct 18, 2022 • edited

ppalaga commented Oct 18, 2022

gzm55 commented Oct 19, 2022

ppalaga commented Oct 19, 2022

gzm55 commented Oct 19, 2022

ppalaga commented Oct 20, 2022 • edited

ppalaga commented Oct 21, 2022

gnodet commented Oct 21, 2022

gzm55 commented Oct 21, 2022

ppalaga commented Oct 21, 2022

gzm55 commented Oct 23, 2022

gzm55 commented Jan 10, 2023

ppalaga commented Jan 10, 2023

gzm55 commented Jan 11, 2023

gzm55 commented Jan 11, 2023

ppalaga left a comment

Choose a reason for hiding this comment

ppalaga left a comment

Choose a reason for hiding this comment

gnodet Jan 18, 2023

Choose a reason for hiding this comment

gzm55 Jan 18, 2023

Choose a reason for hiding this comment

gzm55 commented Oct 16, 2022 •

edited

gzm55 commented Oct 16, 2022 •

edited

gzm55 commented Oct 17, 2022 •

edited

gzm55 commented Oct 18, 2022 •

edited

ppalaga commented Oct 20, 2022 •

edited