[GR-36905] Implement --link-at-build-time and @<prop-values-file>. #4305

graalvmbot · 2022-02-09T18:40:11Z

Adds the following option to native-image:

    --link-at-build-time  require types to be fully defined at image build-time. If used
                          without args, all classes in scope of the option are required to
                          be fully defined.

Depending on where and how the option is used it behaves differently:

Using the option with args is always possible, e.g., --link-at-build-time=foo.bar.Foobar,.... The arguments are fully qualified class names, or package names (note: no package prefix names).
When used on the module path or class path (embedded in native-image.properties files) only classes and packages defined in the same .jar file can be specified.
@<prop-values-file> can be used to redirect values for the option coming from a file relative to the place where the option is used.
When being used without args --link-at-build-time it depends on where it is used:
- On the command line it means all classes should be seen as link-at-build-time classes
- In modules (embedded in native-image.properties files) all classes defined in that module are treated as link-at-build-time classes
- On classpath (embedded in native-image.properties files) an error message is show:

Error: Using '--link-at-build-time' without args only allowed on module-path. 'META-INF/native-image/org.mylibrary/native-image.properties' in 'file:///home/test/myapp/MyLibrary.jar' not part of module-path.

The PR also deprecates the option --allow-incomplete-classpath. Allowing an "incomplete classpath", i.e., link at image run time, is the new default and therefore does not need to be specified explicitly. The option is still allowed for compatibility reasons, but has no effect.

Motivation for this change (see also the discussions in #2762 (comment), #3932 (comment), and #3929): Before this PR, support for an incomplete class path must be enabled by manually via --allow-incomplete-classpath. This ensures that no linking errors are thrown at run time. But since in practice many large projects needs to specify this option because optional dependencies are not always on the class path. So it is necessary to flip the default and enable the support by default.

Still, well-structured libraries should be able to do all linking at image build time, to avoid surprising errors at image run time. And well-structured applications should be able to do the same for all classes of the application. This is enabled by the new option --link-at-build-time. It is designed in a way so that libraries can only configure their own classes, i.e., avoid any side effects on other classes. The restrictions of the --link-at-build-time argument mentioned above ensure that.

When using the Java Platform Module System, it is easy for a library to link all its classes at image build time by specifying --link-at-build-time without listing classes and packages explicitly. But when using the class path, this is not possible because for .jar files on the class path there is no 1:1 mapping between packages and .jar files - multiple .jar files can contribute to the same package, and multiple .jar files can be merged into a single large .jar file. Therefore, packages for libraries used on the class path need to be listed explicitly. To make this process easier and allow, e.g., automatic generation of package lists, the @<prop-values-file> syntax can be used to provide the package list (or class list) in a separate file.

christianwimmer · 2022-02-28T23:30:19Z

FYI @galderz @Sanne (I cannot add you as reviewers)

Sanne · 2022-03-01T00:17:26Z

Many thanks for the huge progress!

Wondering though, is there any way to apply --link-at-build-time globally and yet allow specific exceptions?

I haven't had the chance to explore this in more depth yet, but from the description it seems I would need to set --link-at-build-time on the command line to have it apply the global effect, but not sure how I would then allow a particular library (and only that particular library) to relax the rule.

olpaw · 2022-03-01T11:52:10Z

Hi @Sanne

Wondering though, is there any way to apply --link-at-build-time globally and yet allow specific exceptions?

That would not make sense. If you use --link-at-build-time on the command line (i.e. globally), every class that the image builder sees during analysis will be required to be fully resolved. If you would like to only have individual classes or packages to be required to be fully resolved at build-time you have the following options:

Provide as many --link-at-build-time options with specific arguments on command line as you like. For example:

native-image --link-at-build-time=foo.bar,foobar.core.MyClass --link-at-build-time=my.other.RandomClass ...

If you are a library developer and you want to guarantee that none of the classes of your library will ever be unresolved at image-runtime (requiring all your library classes to be fully resolved at image build-time) you can add

--link-at-build-time=mylibrary.core,mylibrary.ext,mylibrary.aux,... etc

to the Args = of the native-image.properties (in META-INF/native-image/...) that belongs to your library project.

Note that using --link-at-build-time without arguments in such context (as being part of a classpath entry) is not allowed. E.g. suppose you have mylib.jar and it contains META-INF/native-image/org.example/mylib/native-image.properties with Args = --link-at-build-time and you use it with native-image -cp myapp:mylib.jar myapp.Main, you would get the following error message:

Error: Using '--link-at-build-time' without args only allowed on module-path. 'META-INF/native-image/org.example/mylib/native-image.properties' in 'file:///home/pwoegere/mylib.jar' not part of module-path.

As the error messages indicates, the image builder refuses jar files on classpath that use --link-at-build-time without args.

It is fine though to use --link-at-build-time without arguments with Java modules. Assuming mylib.jar would be a Java module containing a module-info.class and it would be used only on the module-path. E.g. like this:

native-image -p mylib.jar:myapp.jar -m myapp

then the builder would accept that. In this case, since mylib.jar is a Java module, --link-at-build-time is interpreted as: Require all classes in the module mylib.jar to be fully resolved at image build-time.

olpaw · 2022-03-01T12:11:59Z

@christianwimmer, re:

When used on the module path or class path (embedded in native-image.properties files) only classes and packages defined in the same .jar file can be specified.

This is currently not enforced. If you use the --link-at-build-time=foo.bar.Foobar,... form embedded in native-image.properties files there is no checking if the given classes and packages are actually within the jar where the native-image.properties is embedded in. Unfortunately it's not practical to require the above since in real-wolld configs you might have to specify --link-at-build-time with args for some library you depend on, but that you do not have control over so that you could embed --link-at-build-time options in that library itself. Even this PR makes use of the fact that we do not enforce this. See https://github.com/oracle/graal/pull/4305/files#diff-98098d0b73f5a8109ce55af795a5cd911783234ccaadcb2346c9e6313c78fb50

vjovanov

Elegant design. We can use the same approach for class initialization which will make it well-behaved.

...src/com.oracle.svm.core/src/com/oracle/svm/core/configure/ReflectionConfigurationParser.java

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/option/OptionUtils.java

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/option/OptionOrigin.java

vjovanov · 2022-03-01T11:16:21Z

substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/LinkAtBuildTimeSupport.java

+    public String errorMessageFor(ResolvedJavaType type) {
+        Class<?> clazz = ((OriginalClassProvider) type).getJavaClass();
+        if (clazz == null) {
+            return "This error is reported at image build time because class " + type.toJavaName(true) + " is registered for linking at image build time.";


An origin for this decision would be useful. Can we pass it here?

We can't. There is none.

It's just the fault behaviour that is responsible for that. I.e. if we have a ResolvedJavaType that has no corresponding Class instance, we are dealing with a purely synthetic ResolvedJavaType created by the builder itself. Thus the only sane default here is to require such ResolvedJavaTypes always to be fully resolved.

substratevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/LinkAtBuildTimeFeature.java

...tevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/phases/SharedGraphBuilderPhase.java

zakkak · 2022-03-02T14:59:54Z

Note that using --link-at-build-time without arguments in such context (as being part of a classpath entry) is not allowed.

@olpaw my understanding is that this is by design in order to prevent libraries from forcing their consumers to link everything at build time. However, this was the default until now and it would be nice to have the option to enforce it even after this patch goes in. To prevent libraries from abusing it we could require it to be done programmatically instead.

That would not make sense. If you use --link-at-build-time on the command line (i.e. globally), every class that the image builder sees during analysis will be required to be fully resolved.

I think it makes sense if you want everything but a couple of classes/packages to be fully resolved at build time. Instead of having to create an exhaustive list of packages required to be resolved at build time, which might as well become incomplete in a future release, you would only need to define the classes/package that you don't require to be resolved at build time.

maxandersen · 2022-03-02T16:16:49Z

Really happy to see movement on improving this!

What I don't grok is how you foresee this change to be used in practice by frameworks like Quarkus, Micronaut, SpringNative etc. that has the need to extend/customize libraries that not yet have native-image configuration or for cases where a library config is not good for specific usecases.

With the proposed solution we now move from being able to explicit control to something where libraries/jars themself are the one to do customization and it is not possible to change that per library without modifying the jar or generate massive lists of classes/packages?

olpaw · 2022-03-02T16:27:41Z

I think it makes sense if you want everything but a couple of classes/packages to be fully resolved at build time. Instead of having to create an exhaustive list of packages required to be resolved at build time, which might as well become incomplete in a future release, you would only need to define the classes/package that you don't require to be resolved at build time.

That would be a full include/exclude config design. The current PR does not have that.

… to junit macro-option

…me and adjust comments

…n classpath

oracle/graal#4305 makes `--allow-incomplete-classpath` the default behavior (removing the flag) and introduces a new flag `--link-at-build-time` which enables us to set which classes/packages we require to be resolvable at build time. Passing `--link-at-build-time` without arguments to `native-image` requires all classes/packages to be resolvable at build time, essentially bringing the pre-22.1 behavior back.

graalvmbot added the oracle-emp label Feb 9, 2022

graalvmbot force-pushed the github/paw/GR-36905 branch from 010e771 to 3dd0030 Compare February 16, 2022 08:35

olpaw changed the title ~~[GR-36905] Implement --require-complete-classdef and @<prop-values-file>.~~ [GR-36905] Implement --link-at-build-time and @<prop-values-file>. Feb 16, 2022

graalvmbot force-pushed the github/paw/GR-36905 branch 6 times, most recently from 432cb37 to 041c43f Compare February 23, 2022 11:26

christianwimmer assigned olpaw and christianwimmer Feb 28, 2022

This was referenced Feb 28, 2022

Allow incomplete classpath configuration to be more granular #3929

Open

Remove some limitations of GraalVM Native Image #2762

Closed

christianwimmer requested review from cstancu, vjovanov, gradinac and zakkak February 28, 2022 23:28

olpaw added this to To do in Native Image via automation Mar 1, 2022

olpaw moved this from To do to In progress in Native Image Mar 1, 2022

vjovanov approved these changes Mar 1, 2022

View reviewed changes

graalvmbot force-pushed the github/paw/GR-36905 branch 2 times, most recently from 2a87716 to 6429db2 Compare March 1, 2022 13:36

zakkak reviewed Mar 2, 2022

View reviewed changes

...tevm/src/com.oracle.svm.hosted/src/com/oracle/svm/hosted/phases/SharedGraphBuilderPhase.java Outdated Show resolved Hide resolved

olpaw added 13 commits March 8, 2022 18:08

Ensure all images of a GraalVM build require --link-at-build-time

0537b3e

Document OptionOrigin.getRedirectionValues

2381518

Rename resolveOptionValueRedirectionFlatMap

6de6a9f

Allow individual launchers to opt-out of --link-at-build-time

45a8a13

Implement strict checking of --link-at-build-time arguments

a0d139d

Add support for @<prop-values-file> use in macro-options

bf1b83a

Move @<prop-values-file>-use from c.o.s.junit native-image.properties…

ed4525d

… to junit macro-option

Change SharedBytecodeParser allowIncompleteClassPath to linkAtBuildTi…

29ad095

…me and adjust comments

Fix strict checking of --link-at-build-time arguments for jar-files o…

4dd1540

…n classpath

Implement --link-at-build-time-paths

4e330cd

Fully document --link-at-build-time and --link-at-build-time-paths

7838db3

Make OptionOrigin classes package-private

c3227c8

Add changelog entry

1655cb2

graalvmbot force-pushed the github/paw/GR-36905 branch from 3678586 to 1655cb2 Compare March 8, 2022 17:57

graalvmbot merged commit 604d2e1 into master Mar 8, 2022

Native Image automation moved this from In progress to Done Mar 8, 2022

This was referenced Mar 9, 2022

Introduce --link-at-build-time flag for GraalVM 22.1 and later quarkusio/quarkus#24213

Merged

Helidon quickstart failing with 22.1-dev Karm/mandrel-integration-tests#76

Closed

dr-eme mentioned this pull request Mar 23, 2022

Support for Apple M1 (darwin-aarch64) #2666

Closed

lewurm mentioned this pull request Mar 23, 2022

ImageSingletons.add must not overwrite existing key com.oracle.svm.core.jdk.ProtectionDomainSupport #4419

Open

graalvmbot deleted the github/paw/GR-36905 branch March 24, 2022 04:37

graalvmbot restored the github/paw/GR-36905 branch March 24, 2022 04:47

graalvmbot deleted the github/paw/GR-36905 branch March 31, 2022 01:51

graalvmbot restored the github/paw/GR-36905 branch March 31, 2022 01:55

graalvmbot deleted the github/paw/GR-36905 branch April 4, 2022 04:06

graalvmbot restored the github/paw/GR-36905 branch April 4, 2022 04:38

graalvmbot deleted the github/paw/GR-36905 branch April 4, 2022 23:53

GoodforGod mentioned this pull request May 12, 2022

[0.19.0] GoodforGod/graalvm-hint#10

Merged

violetagg mentioned this pull request Jan 27, 2023

Native Image UnresolvedElementException sun.misc.SharedSecrets since version 3.5.2 reactor/reactor-core#3334

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GR-36905] Implement --link-at-build-time and @<prop-values-file>. #4305

[GR-36905] Implement --link-at-build-time and @<prop-values-file>. #4305

graalvmbot commented Feb 9, 2022 •

edited by christianwimmer

Loading

christianwimmer commented Feb 28, 2022

Sanne commented Mar 1, 2022 •

edited

Loading

olpaw commented Mar 1, 2022 •

edited

Loading

olpaw commented Mar 1, 2022 •

edited

Loading

vjovanov left a comment

vjovanov Mar 1, 2022

olpaw Mar 1, 2022 •

edited

Loading

zakkak commented Mar 2, 2022

maxandersen commented Mar 2, 2022

olpaw commented Mar 2, 2022 •

edited

Loading

[GR-36905] Implement --link-at-build-time and @<prop-values-file>. #4305

[GR-36905] Implement --link-at-build-time and @<prop-values-file>. #4305

Conversation

graalvmbot commented Feb 9, 2022 • edited by christianwimmer Loading

christianwimmer commented Feb 28, 2022

Sanne commented Mar 1, 2022 • edited Loading

olpaw commented Mar 1, 2022 • edited Loading

olpaw commented Mar 1, 2022 • edited Loading

vjovanov left a comment

Choose a reason for hiding this comment

vjovanov Mar 1, 2022

Choose a reason for hiding this comment

olpaw Mar 1, 2022 • edited Loading

Choose a reason for hiding this comment

zakkak commented Mar 2, 2022

maxandersen commented Mar 2, 2022

olpaw commented Mar 2, 2022 • edited Loading

graalvmbot commented Feb 9, 2022 •

edited by christianwimmer

Loading

Sanne commented Mar 1, 2022 •

edited

Loading

olpaw commented Mar 1, 2022 •

edited

Loading

olpaw commented Mar 1, 2022 •

edited

Loading

olpaw Mar 1, 2022 •

edited

Loading

olpaw commented Mar 2, 2022 •

edited

Loading