6.3 regression: CA keystore missing certificates, AWS plugins not working

[xose]

• Docker image used: docker.elastic.co/elasticsearch/elasticsearch:6.3.0
• Operating System: Amazon Linux ECS 2018.03.a

Bug Description

The Java CA keystore that ships with the 6.3.0 Docker image is missing many root certificates, which makes both AWS plugins unusable by default as the cert presented by their endpoints will not validate.

I'm updating my ElasticSearch deployments on AWS from 6.2 to 6.3. I have a very simple Dockerfile that just installs the discovery-ec2 and repository-s3 plugins on top of the official ES image. Updating the base image from 6.2.4 to 6.3.0 makes the containers unable to find the cluster with the following exception (note that it's logged as INFO):

[INFO ][o.e.d.e.AwsEc2UnicastHostsProvider] [Aae6Wze] Exception while retrieving instance list from AWS API: Unable to execute HTTP request: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Digging into the issue, the CA keystore in the 6.3 image is half the size of the one in 6.2:

$ docker run docker.elastic.co/elasticsearch/elasticsearch:6.3.0 sh -c '$JAVA_HOME/bin/keytool -cacerts -storepass changeit -list | head -n 4'
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 80 entries


$ docker run docker.elastic.co/elasticsearch/elasticsearch:6.2.4 sh -c '$JAVA_HOME/bin/keytool -keystore "$JAVA_HOME/lib/security/cacerts" -storepass changeit -list | head -n 4'
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 155 entries

For comparison, this is the one on the official OpenJDK 10 Docker image:

$ docker run openjdk:10 sh -c '$JAVA_HOME/bin/keytool -cacerts -storepass changeit -list | head -n 4'
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 133 entries

Workaround

For anyone who finds this problem, the workaround to make the AWS plugins work is to add the AWS root cert on the Docker image. Here is my complete Dockerfile with both plugins and certificate:

FROM docker.elastic.co/elasticsearch/elasticsearch:6.3.0

RUN curl -s https://www.amazontrust.com/repository/SFSRootCAG2.pem | $JAVA_HOME/bin/keytool -cacerts -storepass changeit -importcert -noprompt

RUN for PLUGIN in discovery-ec2 repository-s3; do elasticsearch-plugin install --batch $PLUGIN; done

[fxdgear]

Doing a bit of digging it would appear that AWS CA is not included with Java 10.

Java 1.4.2_12, Java 5 update 2, and all newer versions, including Java 6, Java 7, and Java 8

Thanks for providing a workaround
We're working on a fix for this.
Thank you.

In 6.2 we are using Java from the OS vendor and it uses a Mozilla CA bundle provided by an OS-wide package:

export old='docker run --rm -it docker.elastic.co/elasticsearch/elasticsearch:6.2.4'
export new='docker run --rm -it docker.elastic.co/elasticsearch/elasticsearch:6.3.0'

$ $old realpath /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre/lib/security/cacerts
/etc/pki/ca-trust/extracted/java/cacerts

$ $old yum whatprovides /etc/pki/ca-trust/extracted/java/cacerts
<--snip-->
ca-certificates-2018.2.22-70.0.el7_5.noarch : The Mozilla CA root certificate bundle
Repo        : updates
Matched from:
Filename    : /etc/pki/ca-trust/extracted/java/cacerts
<--snip-->

In 6.3, we install OpenJDK from Oracle and it ships with its own CA bundle:

$ $new realpath /opt/jdk-10.0.1/lib/security/cacerts
/opt/jdk-10.0.1/lib/security/cacerts

@xose has shown that the CA bundle that ships with OpenJDK 10 is not as comprehensive as the Mozilla bundle. Thank you, by the way, for the excellent analysis.

So we could decide that the old way is better and replace the OpenJDK keystore with the Mozilla one. That would, presumably, restore the old behaviour. However, there could be certs in the OpenJDK store that are useful.

I propose that we be conservative (and greedy) and just use all the certs. Like this:

$ $new bash

[root@ec7792ebd8d4 elasticsearch]# $JAVA_HOME/bin/keytool -cacerts -storepass changeit -list | egrep 'contains [0-9]+ entries'
Your keystore contains 80 entries

[root@ec7792ebd8d4 elasticsearch]# echo | $JAVA_HOME/bin/keytool -importkeystore -noprompt -srckeystore /etc/pki/java/cacerts -destkeystore /opt/jdk-10.0.1/lib/security/cacerts -deststorepass changeit 2>&1 | egrep '[0-9]+ entries'
Import command completed:  133 entries successfully imported, 0 entries failed or cancelled

[root@ec7792ebd8d4 elasticsearch]# $JAVA_HOME/bin/keytool -cacerts -storepass changeit -list | egrep 'contains [0-9]+ entries'
Your keystore contains 213 entries

[root@410a1cb07b16 elasticsearch]# $JAVA_HOME/bin/keytool -cacerts -list -storepass changeit | grep amazon
amazonrootca4, Jun 21, 2018, trustedCertEntry,
amazonrootca3, Jun 21, 2018, trustedCertEntry,
amazonrootca2, Jun 21, 2018, trustedCertEntry,
amazonrootca1, Jun 21, 2018, trustedCertEntry,

The image docker.elastic.co/elasticsearch/elasticsearch:6.3.0-cacerts contains the above fix. I've manually tested that it does not produce the exception in the original report.

Please try this image out, @xose, if you'd like to. Thanks.

[xose]

Thanks for the quick update! I tried the image on a new cluster and it works fine. It can connect to AWS endpoints and join each other.

---

I've also been looking into the cert stores and found the following:

• The importkeystore command will copy all certs to the destination keystore without accounting for duplicates. That's why you see 213 entries (133+80) after the import. Actual number of unique certificates after the merge is 159:

$ docker run --rm -it docker.elastic.co/elasticsearch/elasticsearch:6.3.0-cacerts sh -c '$JAVA_HOME/bin/keytool -cacerts -list -storepass changeit | grep "SHA-256" | sort | uniq | wc -l'
159

• The OpenJDK CA Store has 26 CA certs that are not on the Mozilla CA Bundle. I've extracted the list to this gist: https://gist.github.com/xose/8107d582aaa68e2ee156e84ae4dab018

• Many of those certs have been revoked due to the Symantec distrust. I have not tested all certificates to see if they should all be distrusted:

[1] https://security.googleblog.com/2018/03/distrust-of-symantec-pki-immediate.html
[2] https://blog.mozilla.org/security/2018/03/12/distrust-symantec-tls-certificates/

Very interesting, thanks.

So, I think we have now demonstrated that the certificate bundle that ships with OpenJDK is both insufficient and actually harmful, since it contains CA certificates that are widely considered by experts to be untrustworthy.

Given that observation and the absence of reported problems under the old model of using the Mozilla CA bundle, I think we should revert to that behaviour by simply symlinking the keystore file from the OS package into the OpenJDK location.