Custom keystore file always overwritten by open-liberty upon docker container startup

[jkjome]

My Dockerfile is pretty simple...

FROM open-liberty:javaee8-java8-ibm

ENV KEYSTORE_REQUIRED false

COPY --chown=1001:0 . /config

I build/run using...

docker build -t ol-runtime .
docker run -d -p 9080:9080 ol-runtime

Among the config copied into the open-liberty Docker image, the following is most relevant to the current issue....

/config/configDropins/defaults/keystore.xml
/config/resources/security/key.jks

My keystore.xml file looks like...

<server>
    <keyStore id="defaultKeyStore" location="${server.output.dir}/resources/security/key.jks" password="[my password]" type="JKS"/>
</server>

Note that I've verified that (prior to copy into the Docker image) my "key.jks" file contains the certificates I expect to be there using the Java keytool, as well as successfully using this keystore file when running open-liberty directly on my local desktop. However, when I try to run my application under open-liberty in a Docker container, outbound calls to a service requiring cert-auth fail with...

[ERROR   ] CWPKI0022E: SSL HANDSHAKE FAILURE:  A signer with SubjectDN CN=somesubdomain.somedomain.com, OU=Some Certificate, OU=Some Sub-Organization, O=Some Organization, L=Some Location, ST=Some State, C=US was sent from the target host.  The signer might need to be added to local trust store /opt/ol/wlp/output/defaultServer/resources/security/key.jks, located in SSL configuration alias defaultSSLConfig.  The extended error message from the SSL handshake exception is: PKIX path building failed: java.security.cert.CertPathBuilderException: PKIXCertPathBuilderImpl could not build a valid CertPath.; internal cause is:
        java.security.cert.CertPathValidatorException: The certificate issued by OU=Some Intermediate Certificate, O=Some Organization, L=Some Location, ST=Some State, C=US is not trusted; internal cause is:
        java.security.cert.CertPathValidatorException: Certificate chaining error

Upon open-liberty startup under Docker, I see the following log output that I don't ever see when running open-liberty directly on my local desktop...

[AUDIT   ] CWWKS4104A: LTPA keys created in 1.527 seconds. LTPA key file: /opt/ol/wlp/output/defaultServer/resources/security/ltpa.keys
[AUDIT   ] CWPKI0803A: SSL certificate created in 6.658 seconds. SSL key file: /opt/ol/wlp/output/defaultServer/resources/security/key.jks

When I run a bash shell, within Docker, to check the keystore contents of "key.jks", I see that my original keystore contents have been wiped out, though the file maintains the password that I set in my keystore.xml config...

default@993deec726cf:/output/resources/security$ ls -lsa
total 16
4 drwxr-x--- 2 default root 4096 Apr 24 04:19 .
4 drwxrwx--- 1 root    root 4096 Apr 24 04:19 ..
4 -rw-r----- 1 default root 2171 Apr 24 04:19 key.jks
4 -rw------- 1 default root  897 Apr 24 04:19 ltpa.keys
default@993deec726cf:/output/resources/security$ keytool -v -list -keystore key.jks
Enter keystore password:

Keystore type: jks
Keystore provider: IBMJCE

Your keystore contains 1 entry

Alias name: default
Creation date: Apr 24, 2019
Entry type: keyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=localhost, OU=defaultServer, O=ibm, C=us
Issuer: CN=localhost, OU=defaultServer, O=ibm, C=us
Serial number: 3b7ac341
Valid from: 4/24/19 4:19 AM until: 4/23/20 4:19 AM
Certificate fingerprints:
         MD5:  C4:52:A4:B2:5A:A3:26:9C:B3:70:8C:27:09:A4:3B:5E
         SHA1: 15:0A:66:E3:57:1F:C9:C6:A4:17:C0:6A:CE:F9:84:C7:AC:FD:2D:EE
         SHA256: A4:75:5B:72:4B:B6:50:35:DA:D8:6C:CF:92:48:51:C9:D0:B5:98:02:DD:09:F8:A1:07:C3:90:64:FE:4C:79:84
         Signature algorithm name: SHA256withRSA
         Version: 3

Extensions:

#1: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 7e 8e f2 3f b4 6e 2c bf  ab d7 80 94 73 4d d1 c2  .....n......sM..
0010: be c0 1d a5                                        ....
]
]



*******************************************
*******************************************


default@993deec726cf:/output/resources/security$

Other than this keystore issue, everything else seems to work fine in open-liberty under Docker. But this issue is so fundamental to the working of my application, that it's blocking my ability to use open-liberty under Docker. What perplexes me is how this would not already have been reported by someone else? Does no one else running open-liberty under Docker use cert-auth (thus haven't fully exercised this feature) or am I incorrectly applying my keystore configuration? That my keystore configuration works fine when running open-liberty directly on my local desktop indicates to me that it should be valid.

So, is this a bug or am I doing something wrong here?

[arthurdm]

hey @jkjome - in the Docker image the output_dir is different than the config_dir (see this line) so it seems you are copying your key into the config dir, but your XML snippet points to the output dir..which causes OL to create a key for you.

I believe if you changed your XML snippet to be:

<server>
    <keyStore id="defaultKeyStore" location="${server.config.dir}/resources/security/key.jks" password="[my password]" type="JKS"/>
</server>

then it should work (noticed I changed output to config)

[jkjome]

Thanks @arthurdm. It works now. Now I realize that my local desktop installation of open-liberty must set "WLP_OUTPUT_DIR" to the same as the config dir (or leave it unspecified). That's why it works there. I guess I'll start referencing ${server.config.dir} for both docker and non-docker config files to maintain consistency.

https://openliberty.io/docs/ref/config/serverConfiguration.html#server-env
https://www.ibm.com/support/knowledgecenter/en/SSEQTP_liberty/com.ibm.websphere.wlp.doc/ae/twlp_admin_customvars.html