helm: cilium kvstore clustermesh failing without clustermesh-apiserver-remote-cert in 1.15

[taraspos]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

After upgrade from 1.14.8 to 1.15.3 Cilium is failing to connect to remote etcd with:

level=info msg="Waiting for all etcd configuration files to be available" error="open /var/lib/cilium/clustermesh/common-etcd-client-ca.crt: no such file or directory" subsys=kvstore

---

Possible cause:

1. common-etcd-client-ca.crt is supposed to be included via clustermesh-secrets from clustermesh-apiserver-remote-cert:

• cilium/install/kubernetes/cilium/templates/cilium-agent/daemonset.yaml

  Lines 863 to 893 in a758d21

  	- name: clustermesh-secrets
  	projected:
  	# note: the leading zero means this number is in octal representation: do not remove it
  	defaultMode: 0400
  	sources:
  	- secret:
  	name: cilium-clustermesh
  	optional: true
  	# note: items are not explicitly listed here, since the entries of this secret
  	# depend on the peers configured, and that would cause a restart of all agents
  	# at every addition/removal. Leaving the field empty makes each secret entry
  	# to be automatically projected into the volume as a file whose name is the key.
  	- secret:
  	name: clustermesh-apiserver-remote-cert
  	optional: true
  	items:
  	- key: tls.key
  	path: common-etcd-client.key
  	- key: tls.crt
  	path: common-etcd-client.crt
  	{{- if not .Values.tls.caBundle.enabled }}
  	- key: ca.crt
  	path: common-etcd-client-ca.crt
  	{{- else }}
  	- {{ .Values.tls.caBundle.useSecret | ternary "secret" "configMap" }}:
  	name: {{ .Values.tls.caBundle.name }}
  	optional: true
  	items:
  	- key: {{ .Values.tls.caBundle.key }}
  	path: common-etcd-client-ca.crt
  	{{- end }}

2. clustermesh-apiserver-remote-cert is created only when clustermesh.useAPIServer is set to true

• cilium/install/kubernetes/cilium/templates/clustermesh-apiserver/tls-provided/remote-secret.yaml

  Lines 1 to 5 in a758d21

  	{{- if and .Values.clustermesh.useAPIServer (not .Values.clustermesh.apiserver.tls.auto.enabled) }}
  	apiVersion: v1
  	kind: Secret
  	metadata:
  	name: clustermesh-apiserver-remote-cert

3. Since after the 1.15.x upgrade kvstore can't be used together with clustermesh.useAPIServer, otherwise helm chart is failing with:

   Helm upgrade failed for release kube-system/cilium with chart cilium@1.15.3: execution error at (cilium/templates/validate.yaml:85:7): The clustermesh-apiserver cannot be enabled in combination with .Values.identityAllocationMode=kvstore. To establish a Cluster Mesh, directly configure the parameters to access the remote kvstore through .Values.clustermesh.config

• This change seems to be introduced in: Minor helm improvements concerning clustermesh #28763

4. As result, Cilium agent can't join remote etcd because of missing tls certs

Cilium Version

1.15.3

Kernel Version

NA

Kubernetes Version

N/A

Regression

1.14.8

Sysdump

No response

Relevant log output

level=info msg="Creating etcd client" ConfigPath=/var/lib/cilium/clustermesh/cluster-2 KeepAliveHeartbeat=15s KeepAliveTimeout=25s ListLimit=256 MaxInflight=100 RateLimit=100 subsys=kvstore
level=info msg="Creating etcd client" ConfigPath=/var/lib/etcd-config/etcd.config KeepAliveHeartbeat=15s KeepAliveTimeout=25s ListLimit=256 MaxInflight=100 RateLimit=100 subsys=kvstore

Anything else?

No response

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct

[giorio94]

Hi @taraspos,

You need to manually specify through the dedicated section the TLS key/certificate used to connect to the kvstore in remote clusters when running Cilium in kvstore mode, as those cannot be automatically managed by Cilium.

[ti-mo]

@giorio94 What's the takeaway? Does the upgrade documentation need to be improved? Sounds like this is a bit of a breaking change.

[taraspos]

Hey @giorio94, thanks for the reply!

Our configuration works in 1.14.8, but we had useAPIServer set to true with replicas scaled down, that was a workaround because some resources were not created otherwise.

After upgrading to 1.15 useAPIServer=true can't be used together with kvstore, so I had to disable it and to face current issue. I will take a look at the tls configuration section you provided when attempting 1.15 upgrade somewhere later.

[giorio94]

What's the takeaway? Does the upgrade documentation need to be improved? Sounds like this is a bit of a breaking change.

Hmm, specifying the clustermesh configuration through helm in combination with Cilium running in kvstore mode had never been supported properly, and required using the trick of enabling the clustermesh-apiserver, but then scaling the replicas to 0.

That was addressed by #28763, which also added an explicit validation to prevent running the clustermesh-apiserver when Cilium is running in kvstore mode. Indeed, this combination does not work correctly and is not supported by design, and allowing them to be enabled at the same time caused confusion to several users in the past.

@taraspos Could you please share some more details about your clustermesh setup, and in particular about how the TLS certificates are configured for the external kvstores? The reason for not automatically creating the TLS certificates when the clustermesh-apiserver is disabled is that they wouldn't be trusted by the external kvstore, except in the very specific case in which the same CA is configured there as well (which I guess it could be the case for you).