[GR-66110] [Native Image] Some .so libs are not considered Produced artifacts

[scrocquesel]

Describe the Issue

I'm compiling a Quarkus application using AWS CRT package. This package relies on a native library which is embeded in the package.

The lib is extracted to the native image path by a GraalVM Feature after image write but is not considered as a build artifact and is not in the output of -H:+GenerateBuildArtifactsFile.

Using the latest version of GraalVM can resolve many issues.

• I tried with the latest version of GraalVM.

GraalVM Version

Java version: 21.0.7+6-LTS, vendor version: Mandrel-23.1.7.0-Final

Operating System and Version

Linux

Build Command

-J-Djava.util.logging.manager=org.jboss.logmanager.LogManager -J-Duser.language=fr -J-Duser.country=FR -J-Dlogging.initial-configurator.min-level=500 -H:+UnlockExperimentalVMOptions -H:IncludeLocales=fr-FR -H:-UnlockExperimentalVMOptions -J-Dfile.encoding=UTF-8 -J--add-exports=org.graalvm.nativeimage.builder/com.oracle.svm.core.jdk=ALL-UNNAMED --features=io.quarkus.runner.Feature,io.quarkus.runtime.graal.DisableLoggingFeature,io.quarkus.runtime.graal.SkipConsoleServiceProvidersFeature -J--add-exports=java.security.jgss/sun.security.krb5=ALL-UNNAMED -J--add-exports=java.security.jgss/sun.security.jgss=ALL-UNNAMED -J--add-opens=java.base/java.text=ALL-UNNAMED -J--add-opens=java.base/java.io=ALL-UNNAMED -J--add-opens=java.base/java.lang.invoke=ALL-UNNAMED -J--add-opens=java.base/java.util=ALL-UNNAMED -H:+UnlockExperimentalVMOptions -H:BuildOutputJSONFile=quarkus-aws-sso-native-0.0.0-SNAPSHOT-runner-build-output-stats.json -H:-UnlockExperimentalVMOptions -H:+UnlockExperimentalVMOptions -H:+GenerateBuildArtifactsFile -H:-UnlockExperimentalVMOptions --strict-image-heap -H:+UnlockExperimentalVMOptions -H:+AllowFoldMethods -H:-UnlockExperimentalVMOptions -J-Djava.awt.headless=true --no-fallback --link-at-build-time -H:+UnlockExperimentalVMOptions -H:+ReportExceptionStackTraces -H:-UnlockExperimentalVMOptions -H:-AddAllCharsets --enable-url-protocols=http,https --enable-monitoring=heapdump -H:+UnlockExperimentalVMOptions -H:-UseServiceLoaderFeature -H:-UnlockExperimentalVMOptions -J--add-exports=org.graalvm.nativeimage/org.graalvm.nativeimage.impl=ALL-UNNAMED quarkus-aws-sso-native-0.0.0-SNAPSHOT-runner -jar quarkus-aws-sso-native-0.0.0-SNAPSHOT-runner.jar

Expected Behavior

Produced artifacts should list libaws-crt-jni.so because it is used by the application and exists in the native image target folder like it does when using jdk lib like libawt.so for example.

Actual Behavior

Produced artifacts:
 /project/build-artifacts.json (build_info)
 /project/quarkus-aws-sso-native-0.0.0-SNAPSHOT-runner (executable)
 /project/quarkus-aws-sso-native-0.0.0-SNAPSHOT-runner-build-output-stats.json (build_info)

Steps to Reproduce

1. Add dependency

   <dependency>
       <groupId>software.amazon.awssdk</groupId>
       <artifactId>aws-crt-client</artifactId>
       <version>0.38.2</version>
   </dependency>
   

2. Make a call to init CRT and import the lib

   new software.amazon.awssdk.crt.CRT();
   

Additional Context

• GraalVMFeature extracting the lib : https://github.com/awslabs/aws-crt-java/blob/main/src/main/java/software/amazon/awssdk/crt/internal/GraalVMNativeFeature.java
• The lib is loaded with System.loadLibrary : https://github.com/awslabs/aws-crt-java/blob/83d451ebfe7b66dcd3f361295d0010ecc647ef9a/src/main/java/software/amazon/awssdk/crt/CRT.java#L39-L52

Build Log Output and Error Messages

No response

[selhagani]

Hi @scrocquesel,

Thank you for reaching out to us about this.
Could you please create a minimal reproducer alongside the steps needed to encounter the issue using a Github repository?

[scrocquesel]

Hello, here it is https://github.com/scrocquesel/quarkus-amazon-services-crt-client-issue

The output will show

Produced artifacts:
 /project/build-artifacts.json (build_info)
 /project/quarkus-amazon-services-crt-client-issue-1.0.0-SNAPSHOT-runner (executable)
 /project/quarkus-amazon-services-crt-client-issue-1.0.0-SNAPSHOT-runner-build-output-stats.json (build_info)

and build-artifacts.json

{"build_info":["quarkus-amazon-services-crt-client-issue-1.0.0-SNAPSHOT-runner-build-output-stats.json"],"executables":["quarkus-amazon-services-crt-client-issue-1.0.0-SNAPSHOT-runner"]}

[selhagani]

Thank you for providing the reproducer. I was able to use it and observed the same output you described.
I've shared this with our team and will ensure you are kept informed of any updates.

[selhagani]

Hi @scrocquesel,

Thank you again for reaching out to us.

I’ve discussed this with our team and wanted to share an update. It turns out that the extraction of the shared library in question is not handled by our code directly, but rather by an external feature:

https://github.com/awslabs/aws-crt-java/blob/main/src/main/java/software/amazon/awssdk/crt/internal/GraalVMNativeFeature.java

Since this feature is not part of our image builder codebase, it does not have a mechanism to contribute to the list of output artifacts reported by the native image process. As such, this behavior is not due to a bug in Native Image itself.

You may want to consider reaching out to the aws-crt-java team to see if they can provide guidance or support on this matter.

[scrocquesel]

@selhagani I know, but could it be reconsidered to offer a way for features to contribute to the list of of output ? Quarkus is also using this output to copy the necessary files when building the native image from a remote container instance.
This would be really handy to have centralized file for this purpose. It could be in a different property of the json like userLibs.

[zakkak]

This would be really handy to have centralized file for this purpose.

+1

You may want to consider reaching out to the aws-crt-java team to see if they can provide guidance or support on this matter.

As mentioned in quarkusio/quarkus#47865 (comment)

One option would be to have the GraalVMNativeFeature run something like the following:

BuildArtifacts.singleton().add(ArtifactType.IMPORT_LIBRARY, pathToNativeBuild);

However, this API is not public.

@vjovanov Do you think it would make sense to make this API public and allow 3rd party Features to amend the build-artifacts.json file or is this scenario too niche?

[adampoplawski]

Hello
I see this task is closed. Is it solved in current Quarkus version?