ConfigMapPropertySource can't read configMap. Retry Policy?

[anavidad3]

Related with this Issue #400

Could a retry policy be sense to read ConfigMap?

We work with:
spring-cloud-starter-kubernetes-config 1.0.2.RELEASE version.
gke cluster 1.12.8-gke.10 version.

We have a microservice(MS) where we read a configMap. Most of the time the configmap load is successful but sometimes a warning message like this appears:

2019-07-30 10:18:52,184 [main]WARN org.springframework.cloud.kubernetes.config.ConfigMapPropertySource - Can't read configMap with name: [employee-ms] in namespace:[test]. Ignoring

In our case, load of external configuration in our MS is mandatory.

As we didn't know what the error might be, we made a few changes to the spring-cloud-kubernetes-config.

1. We add the exception to the warning trace.

We discover that the error was for a connection refuse

2019-07-30 11:38:52,184 [main] WARN org.springframework.cloud.kubernetes.config.ConfigMapPropertySource - Can't read configMap with name: [employee-ms] in namespace:[test]. Ignoring io.fabric8.kubernetes.client.KubernetesClientException: Operation: [get] for kind: [ConfigMap] with name: [employee-ms] in namespace: [test] failed. at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:64) at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:72) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:221) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.get(BaseOperation.java:177) at org.springframework.cloud.kubernetes.config.ConfigMapPropertySource.getData(ConfigMapPropertySource.java:101) at org.springframework.cloud.kubernetes.config.ConfigMapPropertySource.<init>(ConfigMapPropertySource.java:77) at org.springframework.cloud.kubernetes.config.ConfigMapPropertySourceLocator.getMapPropertySourceForSingleConfigMap(ConfigMapPropertySourceLocator.java:95) at org.springframework.cloud.kubernetes.config.ConfigMapPropertySourceLocator.lambda$locate$0(ConfigMapPropertySourceLocator.java:78) at java.util.ArrayList.forEach(ArrayList.java:1257) at org.springframework.cloud.kubernetes.config.ConfigMapPropertySourceLocator.locate(ConfigMapPropertySourceLocator.java:77) at org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration.initialize(PropertySourceBootstrapConfiguration.java:97) at org.springframework.boot.SpringApplication.applyInitializers(SpringApplication.java:649) at org.springframework.boot.SpringApplication.prepareContext(SpringApplication.java:373) at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1260) at org.springframework.boot.SpringApplication.run(SpringApplication.java:1248) at com.paradigma.goodly.EmployeeApplication.main(EmployeeApplication.java:13) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) Caused by: java.net.ConnectException: Failed to connect to /10.0.0.1:443 at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:240) at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:158) at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:256) at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:134) at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:113) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at io.fabric8.kubernetes.client.utils.BackwardsCompatibilityInterceptor.intercept(BackwardsCompatibilityInterceptor.java:119) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at io.fabric8.kubernetes.client.utils.ImpersonatorInterceptor.intercept(ImpersonatorInterceptor.java:66) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at io.fabric8.kubernetes.client.utils.HttpClientUtils$2.intercept(HttpClientUtils.java:107) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) at okhttp3.RealCall.execute(RealCall.java:77) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:379) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:313) at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleGet(OperationSupport.java:296) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleGet(BaseOperation.java:794) at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getMandatory(BaseOperation.java:210) ... 22 common frames omitted Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:589) at okhttp3.internal.platform.Platform.connectSocket(Platform.java:125) at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:238) ... 54 common frames omitted

2. We have defined a retry policy with a number between -1 and N> 0, where -1 is infinite retries. Foreach retry we wait 1 second.

[ryanjbaxter]

I am in favor of this, it would be awesome if you wanted to try and submit a PR for it

[anavidad3]

Yes, sure!
I'm going to perform some tests and submit a PR.

[Jumwah]

Any progress on this - I see the PR is ready? I'm getting Connection Refused, more often than I'm not - it's preventing me using Spring Cloud Kubernetes at all since it's not reliable.

[william-tran]

Got bit by this as well. I would far rather the application to crash if it can't start up with its intended config, than to start with incorrect config.

[DiogoMajela]

I have the same error, there is any news about the workaround?

[ryanjbaxter]

Hoping @RoyJacobs can get back to the PR soon to add this functionality.