Custom CA requires trailing newline to function?

[rcreasey]

Bug description
While upgrading some clusters with a pre-seeded CA from 1.5.4 to 1.6.1, I've noticed that certificates stored in cacerts is rather particular about their formatting. Specifically if the secret key values do not contain an encoded trailing slash.

Prior to installing istio, I created a cacerts secret with fresh copies of my CA by using:

kubectl create secret generic cacerts -n istio-system \
  --from-file=ca-cert.pem \
  --from-file=ca-key.pem \
  --from-file=root-cert.pem \
  --from-file=cert-chain.pem

Upon startup, istiod correctly detected the existing CA and began its usual certificate signing:

2020-06-11T01:07:01.843955Z	info	Use local CA certificate
2020-06-11T01:07:01.948649Z	info	Generating istiod-signed cert for [istiod.istio-system.svc istiod-remote.istio-system.svc istio-pilot.istio-system.svc]
2020-06-11T01:07:02.271394Z	info	Use plugged-in cert at etc/cacerts/ca-key.pem

However, ingresscontroller never seems to stabilize properly:

2020-06-11T01:10:16.770852Z	warning	envoy config	[external/envoy/source/common/config/grpc_subscription_impl.cc:101] gRPC config for type.googleapis.com/envoy.api.v2.auth.Secret rejected: Failed to load certificate chain from <inline>
2020-06-11T01:10:16.771802Z	error	sds	resource:default received error: code:13 message:"Failed to load certificate chain from <inline>" . Will not respond until next secret update
...
2020-06-11T01:11:13.001734Z	warn	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
...

After uninstalling istio and deleting the namespace, I tried installing the certificates from my 1.5.4 clusters directly (rather than the newly acquired copies I had previously used). This worked. istiod and ingressgateway both went stable and healthy as expected.

Comparing the difference between the two certificate file pairs, the only difference was a trailing newline at the end of each certificate (after the -----END CERTIFICATE----- or -----END RSA PRIVATE KEY-----). After doing a echo "" >> ca-cert.pem to the newer copies and retrying once more, everything went stable.

Additionally, documentation previously stated that the following was needed in order to properly detect a non-self signed CA:

  values:
    security:
      selfSigned: false

This value has been removed in the 1.6 release with no deprecation notice and the documentation does not seem to reference needing it any longer. While this is fine, it took a few minutes to realize that the default behavior does proper detection without requiring a helm option to configure this.

[X] Docs
[X] Security

Expected behavior
What is documented here: https://istio.io/latest/docs/tasks/security/cert-management/plugin-ca-cert/

Steps to reproduce the bug
See above description

Version (include the output of istioctl version --remote and kubectl version and helm version if you used Helm)

❯ ic version --remote
client version: 1.6.1
control plane version: 1.6.1
data plane version: 1.6.1 (4 proxies)

How was Istio installed?
Via istioctl manifest apply --set tag=1.6.1 using an updated version of this IstioOperatorSpec.

Environment where bug was observed (cloud vendor, OS, etc)
GKE 1.16

[incfly]

This will require a follow up error message improvement in Envoy I believe. Reading from Slack channel.

[myidpt]

Thanks @rcreasey

Here's what I think:

1. If a PEM file contains multiple certs, a newline is required between the certs.
2. A newline isn't be required at the end of the cert or key PEM file.
   Is (2) the case today? If not, we should fix it or at least make it clear in the doc.

[rcreasey]

Yeah, for clarity's sake each .pem file contained only one certificate block.

[myidpt]

@rcreasey Got it. I'll see if Lei or I get time to fix. Meanwhile would also be great if anyone from the community is willing to help.