SunCertPathBuilderException: unable to find valid certification path to requested target in binary container with Mailer

[NinjaTycoon]

Describe the bug

Getting this error when I try to access my test REST for the Mailer.

ERROR [io.qua.ver.htt.run.QuarkusErrorHandler] (executor-thread-1) HTTP Request to /scs/test/cc/v1/emailer/simple failed, error id: 711cd7ac-dbfc-4455-b304-c8d3e9db2bb8-1: org.jboss.resteasy.spi.UnhandledException: java.util.concurrent.CompletionException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

I got the same error in dev (with mock set to false) and in a binary container. I was able to resolve it in dev by launching with this:

./mvnw compile quarkus:dev
-Djavax.net.ssl.trustStore=/home/ninja/dev2/svn/sc-trunk/microservices/sc-cloud-collector/conf/cacerts
-Djavax.net.ssl.trustStorePassword=changeit

I received the email and the REST completed without error in dev. So, I know supplying these JVM parameters to the truststore resolves it when the JVM can find them.

Following this documentation, I've been unable to get it to work. I'm using yml for application config, which includes:

additional-build-args: -H:ResourceConfigurationFiles=resources-config.json,-J-Djavax.net.ssl.trustStore=/home/ninja/dev2/svn/sc-trunk/microservices/sc-cloud-collector/conf/cacerts,-J-Djavax.net.ssl.trustStorePassword=changeit

under quarkus.native. Note that additional-build-args continues to work for the first parameter I've been setting for some time for resources-config.json. It wouldn't boot if that didn't load property. I checked with a text editor that barring the syntax differences, the -D parameters match what I have working for dev.

Expected behavior

The REST to complete w/o error and the email to be sent and received, as happens in dev currently.

Actual behavior

The container binary produces the above error.

To Reproduce

The challenge here is this is talking to a real SMTP server with a real SSL certificate in the truststore. Can't realistically mock it.

Configuration

Here is my mailer config obfuscated for privacy since I'm using a real SMTP server.

quarkus:
  mailer:
    from: e_noreply@MYDOMAIN.com
    host: smtp.MYDOMAIN.com
    port: 25
    mock: false

### Screenshots

## Environment (please complete the following information):

Linux.  The SMTP server is both on a local LAN (trusted by SMTP server) and public to the application.  

### Output of `uname -a` or `ver`

### Output of `java -version`

Linux client.MYDOMAIN.com 4.18.19-100.fc27.x86_64 #1 SMP Wed Nov 14 22:04:34 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

### GraalVM version (if different from Java)

not installed.  It uses docker.  

### Quarkus version or git rev

    <quarkus-plugin.version>1.13.3.Final</quarkus-plugin.version>
    <quarkus.platform.artifact-id>quarkus-universe-bom</quarkus.platform.artifact-id>
    <quarkus.platform.group-id>io.quarkus</quarkus.platform.group-id>
    <quarkus.platform.version>1.13.3.Final</quarkus.platform.version>

### Build tool (ie. output of `mvnw --version` or `gradlew --version`)

$ ./mvnw --version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /home/ninja/.m2/wrapper/dists/apache-maven-3.6.3-bin/1iopthnavndlasol9gbrbg6bf2/apache-maven-3.6.3
Java version: 1.8.0_212, vendor: Oracle Corporation, runtime: /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.fc28.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.18.19-100.fc27.x86_64", arch: "amd64", family: "unix"

## Additional context

[NinjaTycoon]

As a sidenote relative path doesn't work with dev, returning

java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

It only works with the absolute path. That suggests you get this different error when it can find the parameters, but can't find the truststore. Ditto if you put a file name that doesn't exist.

I'm pointing this out cuz in dev, you only get the SunCertPathBuilderException error that the native container is giving if it cannot find the JVM parameters. I understand from your documentation and issues that Quarkus somehow packs this into the build, so this suggests this packing part is failing and that java is either supposed to be satisfied and thus not need these parameters, or that Quarkus is somehow supposed to set the properties for the JVM.

[cescoffier]

I would need a reproducer.

[NinjaTycoon]

I would need a reproducer.

How? It has to try an SSL handshake with an SMTP. Just take one you have working and add those 3 parameters, changing path to an absolute path with a truststore with 1 cert in it for w/e SMTP you are talking to.

As for the Mailer part, it's just a basic REST test...

@Path("test/emailer")
public class EmailerTestREST {
	@Inject
	Mailer mailer;

	@GET
	@Path("/simple")
	public Response sendASimpleEmail() {
	    mailer.send(Mail.withText("you@whereever.com", "A simple email from quarkus", "This is my body"));
	    return Response.accepted("OK").build();
	}	
}

You still have to supply your own SMTP server and create your own truststore, as you'll be trusting a different SMTP server than me.

Then do a container build

./mvnw clean package -e -Pnative -DskipTests -Dquarkus.container-image.build=true

and hit the REST with a browser.

Use these modules

  <artifactId>quarkus-mailer</artifactId>
  <artifactId>quarkus-config-yaml</artifactId>
  <artifactId>quarkus-resteasy</artifactId>

You have my application.yml config in the OP.

You can get rid of the first parameter -H:ResourceConfigurationFiles=resources-config.json, and just use this:

    additional-build-args: -J-Djavax.net.ssl.trustStore=/home/YOUR_PROJECT_PATH/conf/cacerts,-J-Djavax.net.ssl.trustStorePassword=changeit

Just discovered it doesn't blow up my boot as long as my DB is bootstrapped. So, I was able to reproduce the error with just those two parameters.

[NinjaTycoon]

Here is a starter with the same version properties in the pom.xml. This just works in dev with mock.

https://github.com/NinjaTycoon/quarkus-mailer-reproducer

Hit

http://localhost:9090/test/emailer/simple

and you get OK, with the mock output to log.

[quarkus-bot]

/cc @cescoffier

[NinjaTycoon]

Completed the reproducer so that I was able to reproduce the same issue with it.

https://github.com/NinjaTycoon/quarkus-mailer-reproducer

Had to add quarkus-container-image-docker to the POM. Updated the README with instructions including template files where you need to copy and replace certain things specific to your environment.

[NinjaTycoon]

Further diagnosing it

• the native runner build produces the same error
• does not help to put cacerts in /tmp/cacerts
• reproduced error with 1.13.6.Final

Looking at Quarkus code. Noticed this config magic of trustStore happens in the handleAdditionalProperties method in NativeImageBuildStep.java. Here is my build output from that file in case it helps:

[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Building native image from /home/ninja/dev2/git/quarkus-mailer-reproducer/target/quarkus-mailer-reproducer-1.0.0-SNAPSHOT-native-image-source-jar/quarkus-mailer-reproducer-1.0.0-SNAPSHOT-runner.jar
[WARNING] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Cannot find the `native-image` in the GRAALVM_HOME, JAVA_HOME and System PATH. Install it using `gu install native-image` Attempting to fall back to container build.
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildContainerRunner] Using docker to run the native image builder
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildContainerRunner] Checking image status quay.io/quarkus/ubi-quarkus-native-image:21.0-java11
21.0-java11: Pulling from quarkus/ubi-quarkus-native-image
Digest: sha256:affbd4a26c932fa5e507a529c57802ec76c9dc4e97bf96f85ddc40727d3a91d7
Status: Image is up to date for quay.io/quarkus/ubi-quarkus-native-image:21.0-java11
[INFO] [io.quarkus.deployment.pkg.steps.NativeImageBuildStep] Running Quarkus native-image plugin on GraalVM Version 21.0.0.2 (Java Version 11.0.10+8-jvmci-21.0-b06)

If it logged the path of outputDir I'd look for "trustStore" there. Can't see how to verify it called handleAdditionalProperties. Unfortunately, I'm not setup to build quarkus.

[cescoffier]

Unfortunately, I don't have an SMTP server where I need to customize the Keystore. My corporate one, SendGrid and Gmail are all working.

Just to verify, did you try setting quarkus.mailer.trust-all=true ? I'm actually surprised that we don't expose truststore options for the mailer.

[NinjaTycoon]

That resolved it! Perhaps you can update this documentation to suggest that. Thanks for the help! :)

Does this just bypass the trustStore?

[cescoffier]

Yes, I will open a PR for the doc (don't close the issue, so I don't forget)

[NinjaTycoon]

Well, if it bypasses the trustStore, then it technically doesn't resolve the core issue which does appear to be a possible bug. If you need to hit my SMTP server to test I can give it to you privately. I just don't want a reference to it here.

You can ping me at e_ninjatycoon-2021@servicecraze.com

[cescoffier]

The mailer config must be extended to allow trust store configuration.
I will add that.

Thanks for the offer, I will ping you by email.