SfnAsyncClient.create(); fails in latest azul/zulu-openjdk:11

[pguedes]

Describe the bug

The latest azul/zulu-openjdk:11 container seems to have removed some certs that this sdk uses.

public class AwsTrustStoreTest {
    public static void main(String[] args) {
        var c = SfnAsyncClient.create();
        System.out.println(c);
    }
}

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

default setup should run with latest version of container (tags: 11.0.26, 11, 11-latest) as it does with previous version (tag: 11.0.25)

Unable to find image 'azul/zulu-openjdk:11.0.25' locally
11.0.25: Pulling from azul/zulu-openjdk
Digest: sha256:66f96754ecb0e56c259283b2cd6cdb4f62489cfe42b7fb15489a92ed7e38bc52
Status: Downloaded newer image for azul/zulu-openjdk:11.0.25
root@f586ab210322:/# java -jar /craps/api-app/build/libs/api-app-1.0-SNAPSHOT.jar
07:48:51.428 [main] DEBUG software.amazon.awssdk.regions.providers.AwsRegionProviderChain - Unable to load region from software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@6a396c1e:Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region).
07:48:51.649 [main] DEBUG software.amazon.awssdk.core.internal.http.loader.ClasspathSdkHttpServiceProvider - The HTTP implementation loaded is software.amazon.awssdk.http.crt.AwsCrtSdkHttpService@5c6648b0
software.amazon.awssdk.services.sfn.DefaultSfnAsyncClient@71a8adcf

Current Behavior

this code now fails with:

docker run -it -v "$(pwd)":/craps -v /home/pedro/.aws:/root/.aws --entrypoint bash azul/zulu-openjdk:11
root@6c10ed513867:/# java -jar /craps/api-app/build/libs/api-app-1.0-SNAPSHOT.jar
07:49:15.238 [main] DEBUG software.amazon.awssdk.regions.providers.AwsRegionProviderChain - Unable to load region from software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@6a396c1e:Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region).
07:49:15.459 [main] DEBUG software.amazon.awssdk.core.internal.http.loader.ClasspathSdkHttpServiceProvider - The HTTP implementation loaded is software.amazon.awssdk.http.crt.AwsCrtSdkHttpService@5c6648b0
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: software.amazon.awssdk.crt.CrtRuntimeException: TlsContext.tls_ctx_new: Failed to create new aws_tls_ctx (aws_last_error: AWS_IO_TLS_ERROR_DEFAULT_TRUST_STORE_NOT_FOUND(1173), Default TLS trust store not found on this system. Trusted CA certificates must be installed, or "override default trust store" must be used while creating the TLS context.) AWS_IO_TLS_ERROR_DEFAULT_TRUST_STORE_NOT_FOUND(1173)
        at software.amazon.awssdk.crt.io.TlsContext.tlsContextNew(Native Method)
        at software.amazon.awssdk.crt.io.TlsContext.<init>(TlsContext.java:24)
        at software.amazon.awssdk.http.crt.AwsCrtHttpClientBase.<init>(AwsCrtHttpClientBase.java:84)
        at software.amazon.awssdk.http.crt.AwsCrtAsyncHttpClient.<init>(AwsCrtAsyncHttpClient.java:52)
        at software.amazon.awssdk.http.crt.AwsCrtAsyncHttpClient.<init>(AwsCrtAsyncHttpClient.java:49)
        at software.amazon.awssdk.http.crt.AwsCrtAsyncHttpClient$DefaultAsyncBuilder.buildWithDefaults(AwsCrtAsyncHttpClient.java:259)
        at software.amazon.awssdk.core.internal.http.loader.DefaultSdkAsyncHttpClientBuilder.lambda$buildWithDefaults$0(DefaultSdkAsyncHttpClientBuilder.java:43)
        at java.base/java.util.Optional.map(Optional.java:265)
        at software.amazon.awssdk.core.internal.http.loader.DefaultSdkAsyncHttpClientBuilder.buildWithDefaults(DefaultSdkAsyncHttpClientBuilder.java:43)
        at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.lambda$resolveAsyncHttpClient$20(SdkDefaultClientBuilder.java:468)
        at java.base/java.util.Optional.orElseGet(Optional.java:369)
        at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.resolveAsyncHttpClient(SdkDefaultClientBuilder.java:468)
        at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.lambda$finalizeAsyncConfiguration$6(SdkDefaultClientBuilder.java:322)
        at software.amazon.awssdk.utils.AttributeMap$DerivedValue.primeCache(AttributeMap.java:604)
        at software.amazon.awssdk.utils.AttributeMap$DerivedValue.get(AttributeMap.java:593)
        at software.amazon.awssdk.utils.AttributeMap$Builder.resolveValue(AttributeMap.java:400)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
        at software.amazon.awssdk.utils.AttributeMap$Builder.build(AttributeMap.java:362)
        at software.amazon.awssdk.core.client.config.SdkClientConfiguration$Builder.build(SdkClientConfiguration.java:224)
        at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.finalizeAsyncConfiguration(SdkDefaultClientBuilder.java:324)
        at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.asyncClientConfiguration(SdkDefaultClientBuilder.java:234)
        at software.amazon.awssdk.services.sfn.DefaultSfnAsyncClientBuilder.buildClient(DefaultSfnAsyncClientBuilder.java:37)
        at software.amazon.awssdk.services.sfn.DefaultSfnAsyncClientBuilder.buildClient(DefaultSfnAsyncClientBuilder.java:25)
        at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:169)
        at software.amazon.awssdk.services.sfn.SfnAsyncClient.create(SfnAsyncClient.java:5924)
        at AwsTrustStoreTest.main(AwsTrustStoreTest.java:7)
        ... 8 more

Reproduction Steps

run sample code from description inside container as shown in current behavior

Possible Solution

not sure but looks like some certs were removed? at least document how this should work

Additional Information/Context

No response

AWS Java SDK version used

2.31.22

JDK version used

11.0.26

Operating System and version

azul/zulu-openjdk:11

[bhoradc]

Hello @pguedes,

Thank you for reporting this issue and providing the detailed information.

Based on the error message, it appears that the root cause of the problem is related to the missing CA certificates in the azul/zulu-openjdk:11.0.26 container image, which is causing the Java SDK to fail when creating a new TLS context.

Installing the required/missing CA certificates can resolve the issue for now as mentioned in awslabs/aws-c-io#561, but if you need to root cause the problem, it would be more appropriate to report the issue to the zulu-openjdk repository.

Regards,
Chaitanya

[pguedes]

yeah looks container related... runs fine from outside container with same jdk

$=> sdk use java 11.0.26-zulu

Using java version 11.0.26-zulu in this shell.

$=> java -jar api-app/build/libs/api-app-1.0-SNAPSHOT.jar
10:28:31.735 [main] DEBUG software.amazon.awssdk.regions.providers.AwsRegionProviderChain - Unable to load region from software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@49070868:Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region).
10:28:31.940 [main] DEBUG software.amazon.awssdk.core.internal.http.loader.ClasspathSdkHttpServiceProvider - The HTTP implementation loaded is software.amazon.awssdk.http.crt.AwsCrtSdkHttpService@235834f2
software.amazon.awssdk.services.sfn.DefaultSfnAsyncClient@4659191b

$=> java --version
openjdk 11.0.26 2025-01-21 LTS
OpenJDK Runtime Environment Zulu11.78+15-CA (build 11.0.26+4-LTS)
OpenJDK 64-Bit Server VM Zulu11.78+15-CA (build 11.0.26+4-LTS, mixed mode)

$=> docker run -it -v "$(pwd)":/craps -v /home/pedro/.aws:/root/.aws --entrypoint bash azul/zulu-openjdk:11
root@601d638ca126:/# java -version
openjdk version "11.0.26" 2025-01-21 LTS
OpenJDK Runtime Environment Zulu11.78+15-CA (build 11.0.26+4-LTS)
OpenJDK 64-Bit Server VM Zulu11.78+15-CA (build 11.0.26+4-LTS, mixed mode)

root@601d638ca126:/# java -jar /craps/api-app/build/libs/api-app-1.0-SNAPSHOT.jar
07:49:15.238 [main] DEBUG software.amazon.awssdk.regions.providers.AwsRegionProviderChain - Unable to load region from software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@6a396c1e:Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or  system property (aws.region).
07:49:15.459 [main] DEBUG software.amazon.awssdk.core.internal.http.loader.ClasspathSdkHttpServiceProvider - The HTTP implementation loaded is software.amazon.awssdk.http.crt.AwsCrtSdkHttpService@5c6648b0
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
        at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
        at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: software.amazon.awssdk.crt.CrtRuntimeException: TlsContext.tls_ctx_new: Failed to create new aws_tls_ctx (aws_last_error: AWS_IO_TLS_ERROR_DEFAULT_TRUST_STORE_NOT_FOUND(1173), Default TLS trust store not found on this system. Trusted CA certificates must be installed, or "override default trust store" must be used while creating the TLS context.) AWS_IO_TLS_ERROR_DEFAULT_TRUST_STORE_NOT_FOUND(1173)
        at software.amazon.awssdk.crt.io.TlsContext.tlsContextNew(Native Method)
        at software.amazon.awssdk.crt.io.TlsContext.<init>(TlsContext.java:24)
        at software.amazon.awssdk.http.crt.AwsCrtHttpClientBase.<init>(AwsCrtHttpClientBase.java:84)
        at software.amazon.awssdk.http.crt.AwsCrtAsyncHttpClient.<init>(AwsCrtAsyncHttpClient.java:52)
        at software.amazon.awssdk.http.crt.AwsCrtAsyncHttpClient.<init>(AwsCrtAsyncHttpClient.java:49)
        at software.amazon.awssdk.http.crt.AwsCrtAsyncHttpClient$DefaultAsyncBuilder.buildWithDefaults(AwsCrtAsyncHttpClient.java:259)
        at software.amazon.awssdk.core.internal.http.loader.DefaultSdkAsyncHttpClientBuilder.lambda$buildWithDefaults$0(DefaultSdkAsyncHttpClientBuilder.java:43)
        at java.base/java.util.Optional.map(Optional.java:265)
        at software.amazon.awssdk.core.internal.http.loader.DefaultSdkAsyncHttpClientBuilder.buildWithDefaults(DefaultSdkAsyncHttpClientBuilder.java:43)
        at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.lambda$resolveAsyncHttpClient$20(SdkDefaultClientBuilder.java:468)
        at java.base/java.util.Optional.orElseGet(Optional.java:369)
        at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.resolveAsyncHttpClient(SdkDefaultClientBuilder.java:468)
        at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.lambda$finalizeAsyncConfiguration$6(SdkDefaultClientBuilder.java:322)
        at software.amazon.awssdk.utils.AttributeMap$DerivedValue.primeCache(AttributeMap.java:604)
        at software.amazon.awssdk.utils.AttributeMap$DerivedValue.get(AttributeMap.java:593)
        at software.amazon.awssdk.utils.AttributeMap$Builder.resolveValue(AttributeMap.java:400)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
        at software.amazon.awssdk.utils.AttributeMap$Builder.build(AttributeMap.java:362)
        at software.amazon.awssdk.core.client.config.SdkClientConfiguration$Builder.build(SdkClientConfiguration.java:224)
        at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.finalizeAsyncConfiguration(SdkDefaultClientBuilder.java:324)
        at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.asyncClientConfiguration(SdkDefaultClientBuilder.java:234)
        at software.amazon.awssdk.services.sfn.DefaultSfnAsyncClientBuilder.buildClient(DefaultSfnAsyncClientBuilder.java:37)
        at software.amazon.awssdk.services.sfn.DefaultSfnAsyncClientBuilder.buildClient(DefaultSfnAsyncClientBuilder.java:25)
        at software.amazon.awssdk.core.client.builder.SdkDefaultClientBuilder.build(SdkDefaultClientBuilder.java:169)
        at software.amazon.awssdk.services.sfn.SfnAsyncClient.create(SfnAsyncClient.java:5924)
        at AwsTrustStoreTest.main(AwsTrustStoreTest.java:7)
        ... 8 more

root@601d638ca126:/# 
exit

this could be the cause zulu-openjdk/zulu-openjdk#308

[bhoradc]

Hi @pguedes,

Thanks for the confirmation. Therefore, I will go ahead and close this issue.

Regards,
Chaitanya

[github-actions]

This issue is now closed. Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.