Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bitnami/zookeeper] Got error "the trustAnchors parameter must be non-empty" when using PKCS12 truststore for zookeeper tls communication #6182

Closed
shenqinb-star opened this issue Apr 22, 2021 · 10 comments
Closed

[bitnami/zookeeper] Got error "the trustAnchors parameter must be non-empty" when using PKCS12 truststore for zookeeper tls communication #6182

shenqinb-star opened this issue Apr 22, 2021 · 10 comments
Labels
stale 15 days without activity

Comments

@shenqinb-star
Copy link

shenqinb-star commented Apr 22, 2021 •
edited
Loading

Which chart:
Chart Version: 6.2.0
Zookeeper Version: 3.6.2

Describe the bug
Got error "the trustAnchors parameter must be non-empty" when using PKCS12 truststore for zookeeper tls communication.

2021-04-22 02:22:42,987 [myid:1] - WARN  [QuorumConnectionThread-[myid=1]-1:QuorumCnxManager@396] - Cannot open secure channel to 3 at election address zookeeper-2.zookeeper-headless.poc.svc.cluster.local:3888
org.apache.zookeeper.common.X509Exception$SSLContextException: Failed to create TrustManager
        at org.apache.zookeeper.common.X509Util.createSSLContextAndOptionsFromConfig(X509Util.java:371)
        at org.apache.zookeeper.common.X509Util.createSSLContextAndOptions(X509Util.java:328)
        at org.apache.zookeeper.common.X509Util.createSSLContextAndOptions(X509Util.java:282)
        at org.apache.zookeeper.common.X509Util.getDefaultSSLContextAndOptions(X509Util.java:262)
        at org.apache.zookeeper.common.X509Util.createSSLSocket(X509Util.java:517)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:378)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.zookeeper.common.X509Exception$TrustManagerException: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
        at org.apache.zookeeper.common.X509Util.createTrustManager(X509Util.java:512)
        at org.apache.zookeeper.common.X509Util.createSSLContextAndOptionsFromConfig(X509Util.java:369)
        ... 9 more
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
        at java.base/java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
        at java.base/java.security.cert.PKIXParameters.<init>(PKIXParameters.java:157)
        at java.base/java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:130)
        at org.apache.zookeeper.common.X509Util.createTrustManager(X509Util.java:489)
        ... 10 more

To Reproduce
Steps to reproduce the behavior:

  1. enable tls for both client and quorum
  2. specific the keystore and truststore as below in values.yaml:
service:
  type: ClusterIP
  port: 2181
  followerPort: 2888
  electionPort: 3888
  publishNotReadyAddresses: true
  tls:
    client_enable: true
    quorum_enable: true
    disable_base_client_port: false

    client_port: 3181

    client_keystore_path: /opt/bitnami/zookeeper/certs/zookeeper-keystore.p12
    client_keystore_password: "{{ tls.keystore_pass }}"
    client_truststore_path: /opt/bitnami/zookeeper/certs/zookeeper-truststore.p12
    client_truststore_password: "{{ tls.truststore_pass }}"

    # quorum tls certificates are for both quorum port (follower port) and leader selection port.
    quorum_keystore_path: /opt/bitnami/zookeeper/certs/zookeeper-keystore.p12
    quorum_keystore_password: "{{ tls.keystore_pass }}"
    quorum_truststore_path: /opt/bitnami/zookeeper/certs/zookeeper-truststore.p12
    quorum_truststore_password: "{{ tls.truststore_pass }}"
  annotations: {}
  headless:
    annotations: {}
  1. it works when using PKCS12 keystore
  2. it works when using JKS truststore, but got error when using PKCS12 truststore.
  3. for truststore, I only put the CA in it.
  4. both keystore and truststore has password
  5. I have read Zookeepr Admin Doc (https://github.com/apache/zookeeper/blob/master/zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md#Communication+using+the+Netty+framework), it should support PKCS12 truststore, zookeeper will detect by filename if we didn't define the keystore/truststore type.
    image
@migruiz4
Copy link
Member

migruiz4 commented Apr 22, 2021 •
edited
Loading

Hi @shenqinb-star ,

The error message the trustAnchors parameter must be non-empty could mean that your truststore is empty, the password is incorrect or the permissions are not correct.

You could check if the above is the cause of the issue by opening a bash session inside your pod and running:

# Check password and content are correct
keytool -list -storetype PKCS12 -keystore /opt/bitnami/zookeeper/certs/zookeeper-truststore.p12 -storepass <your_password>
# Check ownership and permissions
ls -la /opt/bitnami/zookeeper/certs/zookeeper-truststore.p12

I tried to reproduce your issue, but I was able to run Zookeeper using PKCS12 truststore and keystore.

@shenqinb-star
Copy link
Author

@migruiz4 Thanks for your replay, below are my test results, pleas help to see if there is any problems:

  1. the truststore is not empty, see below:
 have no name!@zookeeper-0:/opt/bitnami/zookeeper/certs$ keytool -list -storetype PKCS12 -keystore /opt/bitnami/zookeeper/certs/zookeeper-truststore.p12 -storepass <store pass>
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry

maximormro-ca, Apr 22, 2021, PrivateKeyEntry, 
Certificate fingerprint (SHA-256): 89:D8:F1:F3:C5:17:7B:F8:23:E9:EC:52:1E:5C:76:37:19:5F:2C:19:C5:CF:AB:FA:F9:7E:EE:E9:0E:37:81:4E
  1. check ownership and permission, see below:
I have no name!@zookeeper-0:/opt/bitnami/zookeeper/certs$ ls -la /opt/bitnami/zookeeper/certs/zookeeper-truststore.p12
lrwxrwxrwx 1 root root 31 Apr 22 12:47 /opt/bitnami/zookeeper/certs/zookeeper-truststore.p12 -> ..data/zookeeper-truststore.p12
  1. I'm using command openssl pkcs12 -export -in <cert fule> -inkey <private key> -passin pass:<private key path> -name <friendly name> -out zookeeper-truststore.p12 -passout pass:<store pass> to generate both keystore and truststore.

@migruiz4
Copy link
Member

Hi,

Your truststore is using a symlink, to discard the permissions issue, could you please check the permissions of the file under /opt/bitnami/zookeeper/data/zookeeper-truststore.p12?

Additionally, could you please ensure your truststore password is configured correctly inside the zookeeper configuration?

cat /opt/bitnami/zookeeper/conf/zoo.cfg | grep trustStore.password

I'd like to ensure the container configured it without issues and it matches your truststore password.

@shenqinb-star
Copy link
Author

@migruiz4, below are my test results:

  1. the permission for both keystore and truststore are -r--r----- 1 root 1001 2508 Apr 22 12:47 zookeeper-truststore.p12. I can not understand why keystore works but truststore doesn't.
  2. I confirmed the truststore password is configured correctly.

@migruiz4
Copy link
Member

migruiz4 commented Apr 23, 2021 •
edited
Loading

Maybe you could try adding javax.net.debug=all to your Java options using:

extraEnvVars:
  - name: JVMFLAGS
    value: "-Djavax.net.debug=all"

The output is a bit noisy but may help us find the root cause of the issue.

In my case, it shows the following after reading the PKCS12 truststore:

...
ZooKeeper JMX enabled by default
Using config: /opt/bitnami/zookeeper/bin/../conf/zoo.cfg
javax.net.ssl|DEBUG|01|main|2021-04-23 07:26:30.408 UTC|X509TrustManagerImpl.java:96|adding as trusted certificates (
  "certificate" : {
    "version"            : "v1",
    "serial number"      : "00 A2 11 65 68 23 A0 39 3A",
    "signature algorithm": "SHA256withRSA",
    "issuer"             : "...",
    "not before"         : "2021-04-22 08:28:44.000 UTC",
    "not  after"         : "2021-05-22 08:28:44.000 UTC",
    "subject"            : "..",
    "subject public key" : "RSA"}
)
javax.net.ssl|DEBUG|1F|QuorumConnectionThread-[myid=1]-1|2021-04-23 07:26:31.000 UTC|X509TrustManagerImpl.java:96|adding as trusted certificates (
  "certificate" : {
    "version"            : "v1",
    "serial number"      : "00 A2 11 65 68 23 A0 39 3A",
    "signature algorithm": "SHA256withRSA",
    "issuer"             : "...",
    "not before"         : "2021-04-22 08:28:44.000 UTC",
    "not  after"         : "2021-05-22 08:28:44.000 UTC",
    "subject"            : "...",
    "subject public key" : "RSA"}
)

@shenqinb-star
Copy link
Author

@migruiz4 I enabled javax.net.debug=all show following errors:

21-04-23 08:00:26,679 [myid:3] - WARN  [QuorumConnectionThread-[myid=3]-2:QuorumCnxManager@396] - Cannot open secure channel to 2 at election address zookeeper-1.zookeeper-headless.poc.svc.cluster.local:3888
org.apache.zookeeper.common.X509Exception$SSLContextException: Failed to create TrustManager
        at org.apache.zookeeper.common.X509Util.createSSLContextAndOptionsFromConfig(X509Util.java:371)
        at org.apache.zookeeper.common.X509Util.createSSLContextAndOptions(X509Util.java:328)
        at org.apache.zookeeper.common.X509Util.createSSLContextAndOptions(X509Util.java:282)
        at org.apache.zookeeper.common.X509Util.getDefaultSSLContextAndOptions(X509Util.java:262)
        at org.apache.zookeeper.common.X509Util.createSSLSocket(X509Util.java:517)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:378)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.zookeeper.common.X509Exception$TrustManagerException: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
        at org.apache.zookeeper.common.X509Util.createTrustManager(X509Util.java:512)
        at org.apache.zookeeper.common.X509Util.createSSLContextAndOptionsFromConfig(X509Util.java:369)
        ... 9 more
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
        at java.base/java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
        at java.base/java.security.cert.PKIXParameters.<init>(PKIXParameters.java:157)
        at java.base/java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:130)
        at org.apache.zookeeper.common.X509Util.createTrustManager(X509Util.java:489)
        ... 10 more
2021-04-23 08:00:26,679 [myid:3] - WARN  [QuorumConnectionThread-[myid=3]-1:QuorumCnxManager@396] - Cannot open secure channel to 1 at election address zookeeper-0.zookeeper-headless.poc.svc.cluster.local/172.30.252.103:3888
org.apache.zookeeper.common.X509Exception$SSLContextException: Failed to create TrustManager
        at org.apache.zookeeper.common.X509Util.createSSLContextAndOptionsFromConfig(X509Util.java:371)
        at org.apache.zookeeper.common.X509Util.createSSLContextAndOptions(X509Util.java:328)
        at org.apache.zookeeper.common.X509Util.createSSLContextAndOptions(X509Util.java:282)
        at org.apache.zookeeper.common.X509Util.getDefaultSSLContextAndOptions(X509Util.java:262)
        at org.apache.zookeeper.common.X509Util.createSSLSocket(X509Util.java:517)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:378)
        at org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.zookeeper.common.X509Exception$TrustManagerException: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
        at org.apache.zookeeper.common.X509Util.createTrustManager(X509Util.java:512)
        at org.apache.zookeeper.common.X509Util.createSSLContextAndOptionsFromConfig(X509Util.java:369)
        ... 9 more
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
        at java.base/java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
        at java.base/java.security.cert.PKIXParameters.<init>(PKIXParameters.java:157)
        at java.base/java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:130)
        at org.apache.zookeeper.common.X509Util.createTrustManager(X509Util.java:489)
        ... 10 more

@shenqinb-star
Copy link
Author

can you share the p12 keystore and truststore which works on your side, I can have tested in my side.

@migruiz4
Copy link
Member

Hi @shenqinb-star,

I tried two approaches, creating PEM certs and then inserting them into the keystore:

#Create CA key and cert
openssl genrsa -out ca-key.pem 2048
openssl req -new -x509 -sha256 -key ca-key.pem -out ca.pem

#Create node key and cert (replace node with zookeeper-0,1,2...)
openssl genrsa -out node-key-temp.pem 2048
openssl pkcs8 -inform PEM -outform PEM -in node-key-temp.pem -topk8 -nocrypt -v1 PBE-SHA1-3DES -out node-key.pem
openssl req -new -key node-key.pem -out node.csr
openssl x509 -req -in node.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -sha256 -out node.pem

#Insert CA into the truststore
keytool -importcert -storetype PKCS12 -keystore zookeper-truststore.pkcs12 \
-storepass zoopass -alias ca -file ca.pem -noprompt

#Convert PEM certificates to PKCS12
openssl pkcs12 -in zookeeper0.pem -inkey zookeeper0-key.pem -passin pass:pass -certfile ca.pem \
  -export -out zookeeper0.pkcs12 -passout pass:zoopass
openssl pkcs12 -in zookeeper1.pem -inkey zookeeper1-key.pem -passin pass:pass -certfile ca.pem \
  -export -out zookeeper1.pkcs12 -passout pass:zoopass 
openssl pkcs12 -in zookeeper2.pem -inkey zookeeper2-key.pem -passin pass:pass -certfile ca.pem \
  -export -out zookeeper2.pkcs12 -passout pass:zoopass
 
#Insert them into a single keystore
keytool -importkeystore -deststorepass zoopass -srcstorepass zoopass -srckeystore zookeeper0.pkcs12 \
-destkeystore zookeeper-keystore.p12 -srcstoretype pkcs12 -alias zookeeper-0.zookeeper-headless.poc.svc.cluster.local
keytool -importkeystore -deststorepass zoopass -srcstorepass zoopass -srckeystore zookeeper1.pkcs12 \
-destkeystore zookeeper-keystore.p12 -srcstoretype pkcs12 -alias zookeeper-1.zookeeper-headless.poc.svc.cluster.local
keytool -importkeystore -deststorepass zoopass -srcstorepass zoopass -srckeystore zookeeper2.pkcs12 \
-destkeystore zookeeper-keystore.p12 -srcstoretype pkcs12 -alias zookeeper-2.zookeeper-headless.poc.svc.cluster.local

The other approach is the one described in the Zookeeper documentation (https://zookeeper.apache.org/doc/r3.5.8/zookeeperAdmin.html) but adding -storetype PKCS12:

#Create keystore
keytool -genkeypair -alias zookeeper-0.zookeeper-headless.poc.svc.cluster.local -keyalg RSA -keysize 2048 -dname "cn=zookeeper-0.zookeeper-headless.poc.svc.cluster.local,cn=" -keypass zoopass -keystore keystore.p12 -storepass zoopass -storetype PKCS12
keytool -genkeypair -alias zookeeper-1.zookeeper-headless.poc.svc.cluster.local -keyalg RSA -keysize 2048 -dname "cn=zookeeper-1.zookeeper-headless.poc.svc.cluster.local,cn=" -keypass zoopass -keystore keystore.p12 -storepass zoopass -storetype PKCS12
keytool -genkeypair -alias zookeeper-2.zookeeper-headless.poc.svc.cluster.local -keyalg RSA -keysize 2048 -dname "cn=zookeeper-2.zookeeper-headless.poc.svc.cluster.local,cn=" -keypass zoopass -keystore keystore.p12 -storepass zoopass -storetype PKCS12

#Create selfsigned truststore
keytool -exportcert -alias zookeeper-0.zookeeper-headless.poc.svc.cluster.local -keystore keystore.p12 -file zookeeper-0.zookeeper-headless.poc.svc.cluster.local.cer -rfc
keytool -exportcert -alias zookeeper-1.zookeeper-headless.poc.svc.cluster.local -keystore keystore.p12 -file zookeeper-1.zookeeper-headless.poc.svc.cluster.local.cer -rfc
keytool -exportcert -alias zookeeper-2.zookeeper-headless.poc.svc.cluster.local -keystore keystore.p12 -file zookeeper-2.zookeeper-headless.poc.svc.cluster.local.cer -rfc

keytool -importcert -alias zookeeper-0.zookeeper-headless.poc.svc.cluster.local -file zookeeper-0.zookeeper-headless.poc.svc.cluster.local.cer -keystore truststore.jks -storepass zoopass -storetype PKCS12
keytool -importcert -alias zookeeper-1.zookeeper-headless.poc.svc.cluster.local -file zookeeper-1.zookeeper-headless.poc.svc.cluster.local.cer -keystore truststore.jks -storepass zoopass -storetype PKCS12
keytool -importcert -alias zookeeper-2.zookeeper-headless.poc.svc.cluster.local -file zookeeper-2.zookeeper-headless.poc.svc.cluster.local.cer -keystore truststore.jks -storepass zoopass -storetype PKCS12

@github-actions
Copy link

github-actions bot commented May 9, 2021

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

@github-actions github-actions bot added the stale 15 days without activity label May 9, 2021
@github-actions
Copy link

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stale 15 days without activity
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@migruiz4 @shenqinb-star