Support Kubernetes config fail-fast and retry (#873)

* Add support for fail fast loading ConfigMaps and Secrets * Add support for retry loading ConfigMaps and Secrets on failure
spring-cloud · Oct 26, 2021 · d338511 · d338511
1 parent e7d864d
commit d338511
Show file tree

Hide file tree

Showing 41 changed files with 2,398 additions and 66 deletions.
diff --git a/docs/src/main/asciidoc/property-source-config.adoc b/docs/src/main/asciidoc/property-source-config.adoc
@@ -459,15 +459,37 @@ NOTE:  If you use `spring.cloud.kubernetes.config.paths` or `spring.cloud.kubern
 functionality will not work.  You will need to make a `POST` request to the `/actuator/refresh` endpoint or
 restart/redeploy the application.
 
+[#config-map-fail-fast]
+In some cases, your application may be unable to load some of your `ConfigMaps` using the Kubernetes API.
+If you want your application to fail the start-up process in such cases, you can set
+`spring.cloud.kubernetes.config.fail-fast=true` to make the application start-up fail with an Exception.
+
+[#config-map-retry]
+You can also make your application retry loading `ConfigMap` property sources on a failure. First, you need to
+set `spring.cloud.kubernetes.config.fail-fast=true`. Then you need to add `spring-retry` 
+and `spring-boot-starter-aop` to your classpath. You can configure retry properties such as
+the maximum number of attempts, backoff options like initial interval, multiplier, max interval by setting the
+`spring.cloud.kubernetes.config.retry.*` properties.
+
+NOTE: If you already have `spring-retry` and `spring-boot-starter-aop` on the classpath for some reason
+and want to enable fail-fast, but do not want retry to be enabled; you can disable retry for `ConfigMap` `PropertySources`
+by setting `spring.cloud.kubernetes.config.retry.enabled=false`.
+
 .Properties:
 [options="header,footer"]
 |===
-| Name                                       | Type      | Default                      | Description
-| `spring.cloud.kubernetes.config.enabled`   | `Boolean` | `true`                       | Enable ConfigMaps `PropertySource`
-| `spring.cloud.kubernetes.config.name`      | `String`  | `${spring.application.name}` | Sets the name of `ConfigMap` to look up
-| `spring.cloud.kubernetes.config.namespace` | `String`  | Client namespace             | Sets the Kubernetes namespace where to lookup
-| `spring.cloud.kubernetes.config.paths`     | `List`    | `null`                       | Sets the paths where `ConfigMap` instances are mounted
-| `spring.cloud.kubernetes.config.enableApi` | `Boolean` | `true`                       | Enable or disable consuming `ConfigMap` instances through APIs
+| Name                                                    | Type      | Default                      | Description
+| `spring.cloud.kubernetes.config.enabled`                | `Boolean` | `true`                       | Enable ConfigMaps `PropertySource`
+| `spring.cloud.kubernetes.config.name`                   | `String`  | `${spring.application.name}` | Sets the name of `ConfigMap` to look up
+| `spring.cloud.kubernetes.config.namespace`              | `String`  | Client namespace             | Sets the Kubernetes namespace where to lookup
+| `spring.cloud.kubernetes.config.paths`                  | `List`    | `null`                       | Sets the paths where `ConfigMap` instances are mounted
+| `spring.cloud.kubernetes.config.enableApi`              | `Boolean` | `true`                       | Enable or disable consuming `ConfigMap` instances through APIs
+| `spring.cloud.kubernetes.config.fail-fast`              | `Boolean` | `false`                      | Enable or disable failing the application start-up when an error occurred while loading a `ConfigMap`
+| `spring.cloud.kubernetes.config.retry.enabled`          | `Boolean` | `true`                       | Enable or disable config retry.
+| `spring.cloud.kubernetes.config.retry.initial-interval` | `Long`    | `1000`                       | Initial retry interval in milliseconds.
+| `spring.cloud.kubernetes.config.retry.max-attempts`     | `Integer` | `6`                          | Maximum number of attempts.
+| `spring.cloud.kubernetes.config.retry.max-interval`     | `Long`    | `2000`                       | Maximum interval for backoff.
+| `spring.cloud.kubernetes.config.retry.multiplier`       | `Double`  | `1.1`                        | Multiplier for next interval.
 |===
 
 === Secrets PropertySource
@@ -622,17 +644,35 @@ the `Secret` named `s1` would be looked up in the namespace that the application
 See <<namespace-resolution,namespace-resolution>> to get a better understanding of how the namespace
 of the application is resolved.
 
+<<config-map-fail-fast,Similar to the `ConfigMaps`>>; if you want your application to fail to start 
+when it is unable to load `Secrets` property sources, you can set `spring.cloud.kubernetes.secrets.fail-fast=true`.
+
+It is also possible to enable retry for `Secret` property sources <<config-map-retry,like the `ConfigMaps`>>.
+As with the `ConfigMap` property sources, first you need to set `spring.cloud.kubernetes.secrets.fail-fast=true`. 
+Then you need to add `spring-retry` and `spring-boot-starter-aop` to your classpath. 
+Retry behavior of the `Secret` property sources can be configured by setting the `spring.cloud.kubernetes.secrets.retry.*`
+properties.
+
+NOTE: If you already have `spring-retry` and `spring-boot-starter-aop` on the classpath for some reason
+and want to enable fail-fast, but do not want retry to be enabled; you can disable retry for `Secrets` `PropertySources`
+by setting `spring.cloud.kubernetes.secrets.retry.enabled=false`.
 
 .Properties:
 [options="header,footer"]
 |===
-| Name                                        | Type      | Default                      | Description
-| `spring.cloud.kubernetes.secrets.enabled`   | `Boolean` | `true`                       | Enable Secrets `PropertySource`
-| `spring.cloud.kubernetes.secrets.name`      | `String`  | `${spring.application.name}` | Sets the name of the secret to look up
-| `spring.cloud.kubernetes.secrets.namespace` | `String`  | Client namespace             | Sets the Kubernetes namespace where to look up
-| `spring.cloud.kubernetes.secrets.labels`    | `Map`     | `null`                       | Sets the labels used to lookup secrets
-| `spring.cloud.kubernetes.secrets.paths`     | `List`    | `null`                       | Sets the paths where secrets are mounted (example 1)
-| `spring.cloud.kubernetes.secrets.enableApi` | `Boolean` | `false`                      | Enables or disables consuming secrets through APIs (examples 2 and 3)
+| Name                                                     | Type      | Default                      | Description
+| `spring.cloud.kubernetes.secrets.enabled`                | `Boolean` | `true`                       | Enable Secrets `PropertySource`
+| `spring.cloud.kubernetes.secrets.name`                   | `String`  | `${spring.application.name}` | Sets the name of the secret to look up
+| `spring.cloud.kubernetes.secrets.namespace`              | `String`  | Client namespace             | Sets the Kubernetes namespace where to look up
+| `spring.cloud.kubernetes.secrets.labels`                 | `Map`     | `null`                       | Sets the labels used to lookup secrets
+| `spring.cloud.kubernetes.secrets.paths`                  | `List`    | `null`                       | Sets the paths where secrets are mounted (example 1)
+| `spring.cloud.kubernetes.secrets.enableApi`              | `Boolean` | `false`                      | Enables or disables consuming secrets through APIs (examples 2 and 3)
+| `spring.cloud.kubernetes.secrets.fail-fast`              | `Boolean` | `false`                      | Enable or disable failing the application start-up when an error occurred while loading a `Secret`
+| `spring.cloud.kubernetes.secrets.retry.enabled`          | `Boolean` | `true`                       | Enable or disable secrets retry.
+| `spring.cloud.kubernetes.secrets.retry.initial-interval` | `Long`    | `1000`                       | Initial retry interval in milliseconds.
+| `spring.cloud.kubernetes.secrets.retry.max-attempts`     | `Integer` | `6`                          | Maximum number of attempts.
+| `spring.cloud.kubernetes.secrets.retry.max-interval`     | `Long`    | `2000`                       | Maximum interval for backoff.
+| `spring.cloud.kubernetes.secrets.retry.multiplier`       | `Double`  | `1.1`                        | Multiplier for next interval.
 |===
 
 Notes:

diff --git a/spring-cloud-kubernetes-client-config/pom.xml b/spring-cloud-kubernetes-client-config/pom.xml
@@ -99,6 +99,21 @@
 			<scope>test</scope>
 		</dependency>
 
+		<dependency>
+			<groupId>org.springframework.retry</groupId>
+			<artifactId>spring-retry</artifactId>
+			<scope>test</scope>
+		</dependency>
+		<dependency>
+			<groupId>org.springframework.boot</groupId>
+			<artifactId>spring-boot-starter-aop</artifactId>
+			<scope>test</scope>
+		</dependency>
+		<dependency>
+			<groupId>org.mockito</groupId>
+			<artifactId>mockito-inline</artifactId>
+			<scope>test</scope>
+		</dependency>
 	</dependencies>
 
 

diff --git a/...ringframework/cloud/kubernetes/client/config/KubernetesClientConfigMapPropertySource.java b/...ringframework/cloud/kubernetes/client/config/KubernetesClientConfigMapPropertySource.java
@@ -1,5 +1,5 @@
 /*
- * Copyright 2013-2020 the original author or authors.
+ * Copyright 2013-2021 the original author or authors.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -34,6 +34,7 @@
 
 /**
  * @author Ryan Baxter
+ * @author Isik Erhan
  */
 public class KubernetesClientConfigMapPropertySource extends ConfigMapPropertySource {
 
@@ -42,18 +43,19 @@ public class KubernetesClientConfigMapPropertySource extends ConfigMapPropertySo
 	@Deprecated
 	public KubernetesClientConfigMapPropertySource(CoreV1Api coreV1Api, String name, String namespace,
 			Environment environment) {
-		super(getName(name, namespace), getData(coreV1Api, name, namespace, environment, "", true));
+		super(getName(name, namespace), getData(coreV1Api, name, namespace, environment, "", true, false));
 	}
 
 	public KubernetesClientConfigMapPropertySource(CoreV1Api coreV1Api, String name, String namespace,
-			Environment environment, String prefix, boolean includeProfileSpecificSources) {
+			Environment environment, String prefix, boolean includeProfileSpecificSources, boolean failFast) {
 		super(getName(name, namespace),
-				getData(coreV1Api, name, namespace, environment, prefix, includeProfileSpecificSources));
+				getData(coreV1Api, name, namespace, environment, prefix, includeProfileSpecificSources, failFast));
 	}
 
 	private static Map<String, Object> getData(CoreV1Api coreV1Api, String name, String namespace,
-			Environment environment, String prefix, boolean includeProfileSpecificSources) {
+			Environment environment, String prefix, boolean includeProfileSpecificSources, boolean failFast) {
 
+		LOG.info("Loading ConfigMap with name '" + name + "' in namespace '" + namespace + "'");
 		try {
 			Set<String> names = new HashSet<>();
 			names.add(name);
@@ -77,6 +79,11 @@ private static Map<String, Object> getData(CoreV1Api coreV1Api, String name, Str
 			return result;
 		}
 		catch (ApiException e) {
+			if (failFast) {
+				throw new IllegalStateException(
+						"Unable to read ConfigMap with name '" + name + "' in namespace '" + namespace + "'", e);
+			}
+
 			LOG.warn("Unable to get ConfigMap " + name + " in namespace " + namespace, e);
 		}
 		return Collections.emptyMap();

diff --git a/...mework/cloud/kubernetes/client/config/KubernetesClientConfigMapPropertySourceLocator.java b/...mework/cloud/kubernetes/client/config/KubernetesClientConfigMapPropertySourceLocator.java
@@ -1,5 +1,5 @@
 /*
- * Copyright 2013-2020 the original author or authors.
+ * Copyright 2013-2021 the original author or authors.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -29,6 +29,7 @@
 
 /**
  * @author Ryan Baxter
+ * @author Isik Erhan
  */
 public class KubernetesClientConfigMapPropertySourceLocator extends ConfigMapPropertySourceLocator {
 
@@ -88,7 +89,8 @@ else if (kubernetesClientProperties != null) {
 		}
 
 		return new KubernetesClientConfigMapPropertySource(coreV1Api, name, namespace, environment,
-				normalizedSource.getPrefix(), normalizedSource.isIncludeProfileSpecificSources());
+				normalizedSource.getPrefix(), normalizedSource.isIncludeProfileSpecificSources(),
+				this.properties.isFailFast());
 	}
 
 }
diff --git a/...springframework/cloud/kubernetes/client/config/KubernetesClientSecretsPropertySource.java b/...springframework/cloud/kubernetes/client/config/KubernetesClientSecretsPropertySource.java
@@ -1,5 +1,5 @@
 /*
- * Copyright 2013-2020 the original author or authors.
+ * Copyright 2013-2021 the original author or authors.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -32,6 +32,7 @@
 
 /**
  * @author Ryan Baxter
+ * @author Isik Erhan
  */
 public class KubernetesClientSecretsPropertySource extends SecretsPropertySource {
 
@@ -40,15 +41,17 @@ public class KubernetesClientSecretsPropertySource extends SecretsPropertySource
 	private CoreV1Api coreV1Api;
 
 	public KubernetesClientSecretsPropertySource(CoreV1Api coreV1Api, String name, String namespace,
-			Environment environment, Map<String, String> labels) {
-		super(getSourceName(name, namespace), getSourceData(coreV1Api, environment, name, namespace, labels));
+			Environment environment, Map<String, String> labels, boolean failFast) {
+		super(getSourceName(name, namespace), getSourceData(coreV1Api, environment, name, namespace, labels, failFast));
 
 	}
 
 	private static Map<String, Object> getSourceData(CoreV1Api api, Environment env, String name, String namespace,
-			Map<String, String> labels) {
+			Map<String, String> labels, boolean failFast) {
 		Map<String, Object> result = new HashMap<>();
 
+		LOG.info("Loading Secret with name '" + name + "' or with labels [" + labels + "] in namespace '" + namespace
+				+ "'");
 		try {
 			// Read for secrets api (named)
 			if (StringUtils.hasText(name)) {
@@ -81,6 +84,11 @@ private static Map<String, Object> getSourceData(CoreV1Api api, Environment env,
 			}
 		}
 		catch (Exception e) {
+			if (failFast) {
+				throw new IllegalStateException("Unable to read Secret with name '" + name + "' or labels [" + labels
+						+ "] in namespace '" + namespace + "'", e);
+			}
+
 			LOG.warn("Can't read secret with name: [" + name + "] or labels [" + labels + "] in namespace:[" + namespace
 					+ "] (cause: " + e.getMessage() + "). Ignoring", e);
 		}

diff --git a/...ramework/cloud/kubernetes/client/config/KubernetesClientSecretsPropertySourceLocator.java b/...ramework/cloud/kubernetes/client/config/KubernetesClientSecretsPropertySourceLocator.java
@@ -1,5 +1,5 @@
 /*
- * Copyright 2013-2020 the original author or authors.
+ * Copyright 2013-2021 the original author or authors.
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -31,6 +31,7 @@
 
 /**
  * @author Ryan Baxter
+ * @author Isik Erhan
  */
 public class KubernetesClientSecretsPropertySourceLocator extends SecretsPropertySourceLocator {
 
@@ -90,7 +91,7 @@ else if (kubernetesClientProperties != null) {
 		}
 
 		return new KubernetesClientSecretsPropertySource(coreV1Api, secretName, namespace, environment,
-				normalizedSource.getLabels());
+				normalizedSource.getLabels(), this.properties.isFailFast());
 	}
 
 }