SaslAuthenticationException after installation on ICP 3.1.1

[johbossle]

Hi,
I just installed the IBM Event Streams Community Edition-2018.3.1 into a fresh installation of ICP 3.1.1 (CE) and get a org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed, invalid credentials when I'm trying to access the UI.
Checking the logs (as noted in #10) lead me to the fact, that the log of the eventstreams-ibm-es-access-controller is full of errors like this

{"error":"Post https://xxx.yyy.zzz:8443/iam-token/oidc/token: x509: certificate signed by unknown authority","message":"failed to obtain token from IAM","message_template":"failed to obtain token from IAM","mh_file":"renewer.go","mh_line":104,"mh_ts":"Feb  6 07:53:09.287","transaction_id":"a8411f35-bde7-4524-9cf3-73972746908c"}

The installation seems to be correct, since there are no failed jobs or other obvious strange pods.
I installed the ICP with having a valid singed certificate and a custom domain name (not only published with an IP address). I also installed the eventstreams with a provided certificate/ private key (section Kafka external access configuration in the values.yaml).
Is there any chance to import the certificate of the iam service to a keystore, so it is considered as a valid certificate?
Any help is highly appreciated.

[nictownsend]

Hi Johannes,

The error that you have supplied suggests that the certificate presented by the web server serving https://xxx.yyy.zzz:8443/iam-token/oidc/token is not providing a certificate that eventstreams-ibm-es-access-controller can verify.

Would you be able to provide the public certificate that you supplied when installing ICP, and the public certificate you supplied when installing Event Streams?

Additionally, could you provide the output from kubectl -n kube-system get secret cluster-ca-cert -o=jsonpath='{.data.tls\.crt}' | base64 --decode | openssl x509 -text.

Additionally, when you installed ICP, how was the certificate signed? Was it through a known CA, self signed, or signed by your own CA?

[johbossle]

Hi Nic,

I have installed the ICP with a icp-router.crt containing the complete certificate chain (according to https://www.ibm.com/support/knowledgecenter/en/SSBS6K_3.1.1/installing/create_ca_cert.html#existing). The certificate is a wildcard certificate for our domain (*.yyy.zzz) that was signed by an known CA (thawte). The management console does seamlessly work with this certificate.

For installation of eventstreams I used the following settings in ´values.yaml`

proxy:
  # external IP address for access that the proxy should use
  externalEndpoint: "xxx.yyy.zzz"
# Secure connection settings for the proxy
tls:
  type: "provided"
  key: "<key for my certificate | base 64>"
  cert: "<my wildcard certificate without the chain | base 64>"
  cacert: "<the chain parts of my wildcard certificate | base 64>"

Nevertheless the output of the cluster-ca-cert points to a self signed certificate. Please find it here
kube-system.crt.txt .

Thanks in advance

[nictownsend]

Hi Johannes,

Thank you for the update, it is very useful to see that the cluster-ca-cert is self signed.

Please could you post the output of the following command: curl -v https://xxx.yyy.zzz:8443/iam-token/oidc/token - this will show the details of the certificate being served by that endpoint.

I am digging further to understand why the cluster-ca-cert is pointing to a self signed certificate.

[johbossle]

Hi Nic,

please find attached the masked output of the curl. I replaced the values in a consistent way. Hope this helps.
curl.txt

Maybe we will end up in a general ICP installation/configuration issue for the cluster-ca-cert...

[nictownsend]

Hi Johannes,

That is showing that the server itself is responding with the expected certificate, and not the selfsigned as seen in cluster-ca-cert. I will now start investigating why the certificates are not matching, and why access-controller is not verifying that signed certificate.

Also, please could you provide the output of so that I can confirm whether the signed certificate is visible to access-controller:

• kubectl -n <event-streams-namespace> get secret <event-streams-release>-ibm-es-proxy-secret -o jsonpath={'.data.tls\.cert'} | base64 --decode | openssl x509 -text
• kubectl -n <event-streams-namespace> get secret <event-streams-release>-ibm-es-proxy-secret -o jsonpath={'.data.tls\.cacert'} | base64 --decode | openssl x509 -text
• kubectl -n <event-streams-namespace> get secret <event-streams-release>-ibm-es-proxy-secret -o jsonpath={'.data.tls\.cluster'} | base64 --decode | openssl x509 -text

[nictownsend]

In addition could you please provide the public certificate you supplied to ICP on install, so that I can verify the certificate chain.

[johbossle]

To put the output in a nutshell

tls.cert --> *.yyy.zzz wildcard certificate
tls.cacert --> Thawte TLS RSA CA G1 certificate, but not the complete chain - this would consist of two certificates
tls.cluster --> IBM Cloud Private certificate

I will provide the files later on (tomorrow in Germany). Perhaps there is a more private way of sharing the certificates - knowing that sharing a public certificate should be no security issue at all.

[johbossle]

Please find attached the additional required files
certificate used for the installation: install.crt.txt
output of tls.cacert tls.cacert.txt
output of tls.cluster tls.cluster.txt
(partially masked) output of tls.crt tls.crt.txt

Thanks!

[nictownsend]

Hi Johannes,

Thank you for that output - one more thing I request, could you please send the output of kubectl -n kube-system get secret router-certs -o jsonpath='{.data.tls\.crt}' | base64 --decode (assuming nothing you deem sensitive). I am trying to identify whether the full certificate chain you supplied on install of ICP is available to access-controller.

[johbossle]

Hi Nic,

the access-controller seems to have the full certificate chain. Please double-check it with the provided output
router-certs.tls.txt

[nictownsend]

Hi Johannes,

Given that router-certs contains the expected certificate, please could you try the following:

1. kubectl -n <event-streams-namespace> get secret <event-streams-release>-ibm-es-proxy-secret --export -o yaml > proxySecret.backup.yaml
2. Copy proxySecret.backup.yaml to proxySecret.yaml
3. Edit proxySecret.yaml and replace the value stored in tls.cluster with the output of kubectl -n kube-system get secret router-certs -o jsonpath='{.data.tls\.crt}'
4. Apply the updated secret - kubectl -n <eventStreamsNamespace> apply -f proxySecret.yaml
5. Scale down and up the access controller deploy: kubectl -n <eventStreamsNamespace> scale deploy <eventStreamsRelease>-ibm-es-access-controller-deploy --replicas=0 and then kubectl -n <eventStreamsNamespace> scale deploy <eventStreamsRelease>-ibm-es-access-controller-deploy --replicas=2 to restart the access controllers.

This should then allow the connection between access controller and the https://xxx.yyy.zzz:8443/iam-token/oidc/token endpoint to succeed.

[nictownsend]

I have opened #19 to track this issue through to resolution inside the product.

Please continue to use this thread to confirm whether your particular Event Streams install is functional.

[johbossle]

Hi Nic,

I folllowed the suggested steps to replace the cacert part of the secret - but unfortunately it did not help at all. I'm still facing the problem that the call to https://xxx.yyy.zzz:8443/iam-token/oidc/tokenis rejected.
The resolving of the URL to the correct public IP works inside the access-controller-container. Unfortunately there is only wget available and no curl to validate the certificate in the container directly.
Any other ideas?
Thanks in advance and kind regards

[nictownsend]

Hi Johannes,

Could you please provide the output of the following command to confirm that access controller has now picked up the correct cert:

kubectl get pods -n <eventStreamsNamespace> | grep access-controller - copy the pod name of the first line returned

Then execute

kubectl exec -n <eventStreamsNamespace> <accessControllerPod> -c access-controller -- cat /etc/ssl/certs/ca-certificates.crt | openssl x509 -text

If this certificate is not the correct certificate, the next step is to let the container verify the certificate using its default public CA list. Please could you edit the -ibm-es-access-controller-deploy deployment to remove the volumeMount entry inside the access-controller container for ca-certs-volume. The path to that value in the deployment is spec.template.spec.containers[name=access-controller].volumeMounts

You can either edit the deployment in place via kubectl edit <releaseName>-ibm-es-access-controller-deploy, or use kubectl get deploy <releaseName>-ibm-es-access-controller-deploy -o yaml --export > access-deploy.yaml (and backup file), edit it locally with a text editor, then add back to ICP via kubectl apply -f access-deploy.yaml

[nictownsend]

Hi Johannes, I'm afraid I mistyped in my comment above : #18 (comment). Please can you attempt the following before the steps in the previous post.

When editing proxySecret.yaml the replaced value should have been tls.cluster not tls.cacert.

Please could you perform the following steps:

1. Copy proxySecret.backup.yaml to proxySecret.yaml
2. Edit proxySecret.yaml and replace the value stored in tls.cluster with the output of kubectl -n kube-system get secret router-certs -o jsonpath='{.data.tls\.crt}'
3. Apply the updated secret - kubectl -n <eventStreamsNamespace> apply -f proxySecret.yaml
4. Scale down and up the access controller deploy: kubectl -n <eventStreamsNamespace> scale deploy <eventStreamsRelease>-ibm-es-access-controller-deploy --replicas=0 and then kubectl -n <eventStreamsNamespace> scale deploy <eventStreamsRelease>-ibm-es-access-controller-deploy --replicas=2 to restart the access controllers.

[johbossle]

Hi Nic,
that was the trick... Replacing the tls.cluster certificate helped. Now I'm able to use the UI without the SaslAuthenticationException. Also the log states, that getting the iam token was successful "successfully obtained token from IAM"
Thank you very much!
I will open another issue tomorrow because the UI is only available via IP Address, since there is an extra certificate used in the section https.cert of the <event-streams-release>-ibm-es-proxy-secret