Spring cloud kuberentes watcher spring boot 3.x help required

[Nomi67]

Hi,
I'm trying to implement spring cloud kubernetes in our projects. My main goals are to read configMap, secrets and enable reload using spring cloud kubernetes config watcher.

Since there are very less examples available about how to use spring cloud for kubernetes apps, I read the documentation and figured out how to read secrets and configmap from k8s. There are 2 ways you can read these i.e

1. Volume mount
2. Kubernetes API

The 2nd one is not recommended due to security problems which can occur so I'm preferring 1st one. But when it comes to reloading properties using watcher then there is zero examples available with spring boot 3.x.

While using below properties in bootstrap.properties I get below warning logs
bootstrap.properties
spring.cloud.kubernetes.discovery.enabled=false spring.cloud.kubernetes.secrets.paths=/etc/secrets/secret-k8s spring.cloud.kubernetes.secrets.fail-fast=true spring.cloud.kubernetes.secrets.enabled=true

warning log
o.s.c.k.c.c.SecretsPropertySourceLocator : path support is deprecated and will be removed in a future release. Please use **spring.config.import**

But I couldn't find any help about how to use this property with kubernetes using volume mounts.

second thing is if possible could you please share an example of watcher with spring boot 3.x which should update mounted volume(secret/configmap) ?

build.gradle
dependencies { implementation 'org.springframework.boot:spring-boot-starter-actuator' implementation 'org.springframework.boot:spring-boot-starter-web' implementation group: 'org.springframework.cloud', name: 'spring-cloud-starter-kubernetes-fabric8-config', version: '3.0.4' compileOnly 'org.projectlombok:lombok' annotationProcessor 'org.projectlombok:lombok' testImplementation 'org.springframework.boot:spring-boot-starter-test' }

[ryanjbaxter]

But I couldn't find any help about how to use this property with kubernetes using volume mounts.

spring.config.import is a Spring Boot property so its usage is documented in the Spring Boot documentation https://docs.spring.io/spring-boot/docs/3.1.4/reference/htmlsingle/#features.external-config.files

second thing is if possible could you please share an example of watcher with spring boot 3.x which should update mounted volume(secret/configmap) ?

I am not sure what exactly you are looking for with regards to using the watcher and spring boot 3.x.
There is an integration test which provides an example of using the watcher
https://github.com/spring-cloud/spring-cloud-kubernetes/tree/main/spring-cloud-kubernetes-integration-tests/spring-cloud-kubernetes-configuration-watcher-it

[Nomi67]

Hi @ryanjbaxter, thank you for first answer. And for second question I'm looking for help regarding.

I want to mount my secrets and configmap to pod as mounted volume and reload them using spring cloud kubernetes config watcher via actuator end point not using amqp bus event. But I couldn't find any example for this. Could you please share any example code for this use case ?

[wind57]

I am going to write an extensive answer here with an example today, FYI.

[ryanjbaxter]

Can you tell me what you have tried and what issue you are running into?

[wind57]

I agree on the sparse sources for examples. I plan to make some youtube videos in the future on how to configure everything and use, unfortunately I don't have much time at the moment for that. It's on my TODO list nevertheless.

A couple of things, using Kubernetes API for secrets/configmaps is still supported and is not planned to go anywhere anytime soon.

Support for paths (not spring.config.import) is at least going to be supported in 3.0.x releases, though it might be dropped in 4.0.x, but that is yet to be finalized. I am not a spring employee, just a contributor here, and this is what I know so far. You can track this, if you want, here.

As interesting as it sounds, I have been recently preparing for dropping path support and using spring.config.import feature, as such, I have been writing some integration tests in this regard, so you are spot-on asking for this :)

Since this question might appear for other people, I'll try to summarize this here and point to this issue in the feature. The example can be seen here, it is called ReloadConfigMapMountDelegate. Now, it might be a little hard to read the source code, but in essence you need to do a couple of things.

• (1) first, deploy configuration watcher image. You need to tell it what namespaces to look at. This can be done via environment variables, for example:

    spec:
      serviceAccountName: spring-cloud-kubernetes-serviceaccount
      containers:
        - name: spring-cloud-kubernetes-configuration-watcher
          image: docker.io/springcloud/spring-cloud-kubernetes-configuration-watcher:<VERSION>
          imagePullPolicy: IfNotPresent
          readinessProbe:
            httpGet:
              port: 8888
              path: /actuator/health/readiness
          livenessProbe:
            httpGet:
              port: 8888
              path: /actuator/health/liveness
          ports:
            - containerPort: 8888
          env:
            - name: LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_CLOUD_KUBERNETES_CONFIGURATION_WATCHER
              value: DEBUG
            - name: LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_CLOUD_KUBERNETES_CLIENT_CONFIG_RELOAD
              value: DEBUG
            - name: LOGGING_LEVEL_ORG_SPRINGFRAMEWORK_CLOUD_KUBERNETES_COMMONS_CONFIG_RELOAD
              value: DEBUG
           - name: SPRING_CLOUD_KUBERNETES_RELOAD_NAMESPACES_0
              value: "namespace-a" 

Please notice a few things (I've added a few options for debugging purposes, this can help you greatly when inspecting the logs on your initial set-up, to understand what is going on). The second part, is that here, we have explicitly defined the namespace(s) where configuration watcher will look for stuff (more on that later). This is how I prefer to do it: explicit. If you don't do it like this, then namespace resolution chapter from the documentation will kick in. So, first see if there are explicit namespaces defined (SPRING_CLOUD_KUBERNETES_RELOAD_NAMESPACES_0 in the example above), if not: find the namespace according to "namespace resolution".

Keep in mind that spring-cloud-kubernetes-serviceaccount will need to have proper rights (RBAC) to watch for changes in those namespaces.

• (2) So we have a configuration watcher that will "monitor" namespace(s). OK, but what exactly it will monitor there? Well, configmaps and secrets. It could monitor all of them, but almost for sure, you do not want to do that, you want to be more specific. As such, there are two things you can configure here:

• (2.1) only secrets/configmaps that have the label: spring.cloud.kubernetes.config.informer.enabled=true set. When your namespace(s) you monitor have lots of secrets/configmaps, this is a very important label. It will decreases the pressure on k8s api server and configuration watcher significantly.

• (2.2) the second parameter you need to configure in case of spring.config.import (or paths support), is a label on the secret/configmap called spring.cloud.kubernetes.config=true or spring.cloud.kubernetes.secret=true. Without it, configuration watcher will simply ignore changes in secrets/configmaps.

So, (2.2) is a must, (2.1) is a good to have.

Let's go to the part on how you configure spring.config.import, but before we need to look on how to mount a config-map.

Let's suppose this is my configmap:

apiVersion: v1
kind: ConfigMap
metadata:
   name: poll-reload-as-mount
   namespace: default
   labels:
      spring.cloud.kubernetes.config: true
data:
   from.properties: "as-mount-initial"

To mount it (I'm going to put it here as json, the way we have it in the integration test)

"spec": {
    "template": {
       "spec": {
          "volumes": [
		{
			"configMap": {
				"defaultMode": 420,
					"name": "poll-reload-as-mount",
					"items": [
						{
						   "key": "from.properties",
						    "path": "key"
						}
					]
				},
				"name": "config-map-volume"
				}
          ],

         "containers": [{
		"name": "spring-k8s-client-reload",
		"image": "image_name_here",
		"volumeMounts": [
			{
			    "mountPath": "/tmp/props",
			    "name": "config-map-volume"
			}
		]
         .....

If you start such a pod, exec into it, and look into /tmp/props you will see a key file that holds the value as-mount-initial.

You need to tell Spring about this configuration, and this is done (in application.yaml) via:

spring:
  config:
    import: "kubernetes:,configtree:/tmp/"

Spring, in turn, having a properties configuration like this:

@ConfigurationProperties("props")
public class ConfigMapProperties {

	private String key;

	public String getKey() {
		return key;
	}

	public void setKey(String key) {
		this.key = key;
	}

}

will be able to inject the key with value as-mount-initial.

It's time to wire these two concepts together: watcher and spring.

---

Once you have everything set-up and change your configmap, for example it becomes:

...
data:
   from.properties: "as-mount-changed"

configuration watcher will detect this, and schedule "something" (will get to it shortly) to happen. First, you need to know one important thing here. When you change a configmap, the path it is mounted to, will change also, so that path inside the pod where key file is will change to now contain: "as-mount-changed". There is a big BUT here, this will not happen instantly:

• you can do kubectl describe configmap poll-reload-as-mount and see that the actual value is as-mount-changed
• if you exec into the pod and check the key file it might still contain as-mount-initial.

The documentation of k8s is clear as mud in this regard:

When a ConfigMap currently consumed in a volume is updated, projected keys are eventually updated as well.

Notice the eventually part ... this has an implication on our side: this means that the configuration watcher might have caught this change (remember that it watches for configmap changes), might schedule a remote refresh to the pod of interest (will get to it), but the value inside the pod (the key file that is mounted) might have the old value. So the refresh/restart might have been useless and you will never see your changes. There is no strong definition on k8s of the max time it could take for the changes to be reflected in the mounted path, as such configuration watcher provides one more setting:

            - name: SPRING_CLOUD_KUBERNETES_CONFIGURATION_WATCHER_REFRESHDELAY
              value: "10000"

This value is specified in milliseconds and it means in plain english something like this:

after I see a change in the configmap, I will schedule a remote refresh/restart with this delay.

Code-wise would be even simpler to explain:

executorService.schedule(() -> {
			// schedule refresh/restart
		},<SPRING_CLOUD_KUBERNETES_CONFIGURATION_WATCHER_REFRESHDELAY>, TimeUnit.MILLISECONDS);

So don't do it instantly, but after a certain "pause" to give the pods a chance to be updated by the k8s in those mounted paths. By default this is 2 minutes.

So after 2 minutes, send a http request to the apps that you want to refresh.

---

Now, what "apps" are supposed to be refreshed/restarted? By default, it is the app that matches the configmap name - that is: poll-reload-as-mount from our example. If you want other apps to be reloaded, you need to add (to the configmap) an annotation. For example:

spring.cloud.kubernetes.configmap.apps: app-a, app-b

These app-a and app-b have to match the service name, an actual k8s service name. We take this service name, and using Discovery Client (which again has a bunch of properties to configure, I will not go into the details here either, but if someone wants, I will in a separate post), find the so-called "service instances" that this service has. It's a bit more complicated, but to simplify: its the pods of the underlying service (the app-a).

Since by default these requests will go to the actuator endpoint (you can use a bus for this too, like rabbit or kafka - not going to go into the details for now), you need to enable those endpoints on the pods where the refresh/restart is supposed to go to:

management:
  endpoint:
    refresh:
      enabled: true
    restart:
      enabled: true

Of course the actuator URI might be different in your case then the default one :) and we do support configuring this also, in case you need explanation on this subject too, just let me know.

Keep in mind, that we will send requests to all the "service instances" (pods) at the same time almost, this is not a rolling refresh. We do not support this at the moment.

---

P.S. I hope I got everything right here, if I forgot something, just let me know and I'll update this answer.

[ryanjbaxter]

This is epic thanks! When I originally wrote the confg watcher I read some where on the k8s docs what the maximum time was before the values in the Pod would be updated, and I added a little bit to that to be safe as the default value. I am having trouble finding that now, but this is a good explanation on the topic
https://ahmet.im/blog/kubernetes-secret-volumes-delay/

[spring-cloud-issues]

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

[Nomi67]

Hi @wind57 and @ryanjbaxter, Thank you for your help and detailed answer about config watcher. With your information I was able to use it correctly.
This issue can be closed now.

[AngelJava78]

@Nomi67 Could you solve the problem? I'm facing the same issue. Spring Cloud Kubernetes Configuration Watcher works fine with Spring Boot 2.x, but not with Spring Boot 3.x. I cloned this repository: https://github.com/alexdefelipe/spring-cloud-k8s-watcher-demo.git, but it has the same problem as my code. When updating the values of the ConfigMap, the application does not automatically update the configuration. I would appreciate some help to solve the issue with Spring Boot 3.x."