Regression: vault resources not syncing to k8s secrets

[gaffneyd4]

Expected behavior: vault-crd creates and deletes kubernetes secrets in the cluster when Vault resources are created or deleted.

Observed behavior:

• vault-crd doesn't sync newly created Vault resources with the vault server to create a kubernetes secret.
• On deletion of a Vault resource, its respective kubernetes secret is not deleted.
• Output from the modules d.koudingspawn.vault.kubernetes.Watcher and d.k.vault.kubernetes.KubernetesService no longer appear in logs

Versions: 1.3.2, or 1.4.0 (crd update was applied during deployment of 1.4.0). Tested with kubernetes clusters on versions 1.12.8 and 1.15.6. Same behavior occurs in both clusters.

This might be a regression introduced in version 1.3.2, present in 1.4.0 as well.

Logs from vault version 1.4.0 (demonstrates incorrect behavior):

+ vault-crd-5595bf88d-mngb5 › vault-crd
vault-crd-5595bf88d-mngb5 vault-crd 
vault-crd-5595bf88d-mngb5 vault-crd   .   ____          _            __ _ _
vault-crd-5595bf88d-mngb5 vault-crd  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
vault-crd-5595bf88d-mngb5 vault-crd ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
vault-crd-5595bf88d-mngb5 vault-crd  \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
vault-crd-5595bf88d-mngb5 vault-crd   '  |____| .__|_| |_|_| |_\__, | / / / /
vault-crd-5595bf88d-mngb5 vault-crd  =========|_|==============|___/=/_/_/_/
vault-crd-5595bf88d-mngb5 vault-crd  :: Spring Boot ::        (v2.2.4.RELEASE)
vault-crd-5595bf88d-mngb5 vault-crd 
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:52:18.016  INFO 1 --- [           main] de.koudingspawn.vault.VaultApplication   : Starting VaultApplication v0.0.1-SNAPSHOT on vault-crd-5595bf88d-mngb5 with PID 1 (/opt/vault-crd.jar started by root in /opt)
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:52:18.019  INFO 1 --- [           main] de.koudingspawn.vault.VaultApplication   : No active profile set, falling back to default profiles: default
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:52:19.181  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:52:19.193  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:52:19.193  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet engine: [Apache Tomcat/9.0.30]
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:52:19.251  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:52:19.252  INFO 1 --- [           main] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 1190 ms
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:52:19.595  INFO 1 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 'vaultThreadPoolTaskScheduler'
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:52:20.738  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:52:20.791  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:52:20.793  INFO 1 --- [           main] de.koudingspawn.vault.VaultApplication   : Started VaultApplication in 3.104 seconds (JVM running for 3.53)
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:52:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:52:30.791  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:53:06.413  INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:53:06.413  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:53:06.418  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 5 ms
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:53:06.623  INFO 1 --- [nio-8080-exec-1] o.s.v.a.LifecycleAwareSessionManager     : Scheduling Token renewal
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:53:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:53:30.784  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:54:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:54:30.783  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:55:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:55:30.784  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:56:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:56:30.781  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:57:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:57:30.784  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:58:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:58:30.780  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:59:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 14:59:30.780  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:00:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:00:30.780  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:01:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:01:30.780  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:02:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:02:30.781  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:03:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:03:30.788  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:04:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:04:30.780  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:05:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:05:30.780  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:06:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:06:30.780  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:07:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:07:30.781  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:08:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:08:30.780  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:09:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:09:30.780  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:10:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:10:30.780  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:11:30.774  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-5595bf88d-mngb5 vault-crd 2020-01-27 15:11:30.780  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...

Logs from vault version 1.3.1 (demonstrates correct behavior):

+ vault-crd-586bb99858-kg9zc › vault-crd
vault-crd-586bb99858-kg9zc vault-crd
vault-crd-586bb99858-kg9zc vault-crd   .   ____          _            __ _ _
vault-crd-586bb99858-kg9zc vault-crd  /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
vault-crd-586bb99858-kg9zc vault-crd ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
vault-crd-586bb99858-kg9zc vault-crd  \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
vault-crd-586bb99858-kg9zc vault-crd   '  |____| .__|_| |_|_| |_\__, | / / / /
vault-crd-586bb99858-kg9zc vault-crd  =========|_|==============|___/=/_/_/_/
vault-crd-586bb99858-kg9zc vault-crd  :: Spring Boot ::        (v2.1.1.RELEASE)
vault-crd-586bb99858-kg9zc vault-crd
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:16.616  INFO 1 --- [           main] de.koudingspawn.vault.VaultApplication   : Starting VaultApplication v0.0.1-SNAPSHOT on vault-crd-586bb99858-kg9zc with PID 1 (/opt/vault-crd.jar started by root in /opt)
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:16.619  INFO 1 --- [           main] de.koudingspawn.vault.VaultApplication   : No active profile set, falling back to default profiles: default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:18.058  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat initialized with port(s): 8080 (http)
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:18.086  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Starting service [Tomcat]
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:18.087  INFO 1 --- [           main] org.apache.catalina.core.StandardEngine  : Starting Servlet Engine: Apache Tomcat/9.0.13
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:18.099  INFO 1 --- [           main] o.a.catalina.core.AprLifecycleListener   : The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-1.8-openjdk/jre/lib/amd64:/usr/lib/jvm/java-1.8-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:18.178  INFO 1 --- [           main] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring embedded WebApplicationContext
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:18.178  INFO 1 --- [           main] o.s.web.context.ContextLoader            : Root WebApplicationContext: initialization completed in 1519 ms
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:18.559  INFO 1 --- [           main] o.s.s.c.ThreadPoolTaskScheduler          : Initializing ExecutorService 'vaultThreadPoolTaskScheduler'
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:18.779  WARN 1 --- [           main] i.f.k.client.internal.VersionUsageUtils  : The client is using resource type 'customresourcedefinitions' with unstable version 'v1beta1'
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.529  INFO 1 --- [           main] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.591  INFO 1 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.593  INFO 1 --- [           main] de.koudingspawn.vault.VaultApplication   : Started VaultApplication in 3.304 seconds (JVM running for 3.761)
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.693  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret0 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.702  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret1 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.716  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret2 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.721  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret3 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.726  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret4 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.729  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret5 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.738  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret6 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.744  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret7 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.748  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret8 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.758  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret9 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.762  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret10 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.771  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret11 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.777  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret12 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.782  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret13 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.787  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret14 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.794  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret15 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.797  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret16 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.801  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret17 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.805  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret18 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.809  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret19 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.812  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret20 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.820  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret21 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.824  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret22 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.834  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret23 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.838  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret24 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:19.841  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret25 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:29.566  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:29.695  INFO 1 --- [TaskScheduler-1] o.s.v.a.LifecycleAwareSessionManager     : Scheduling Token renewal
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:30.084  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:35.683  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret26 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:05:35.730  INFO 1 --- [/x.x.x.x/...] d.k.vault.kubernetes.KubernetesService   : Created secret for vault resource secret26 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:06:02.649  INFO 1 --- [nio-8080-exec-1] o.a.c.c.C.[Tomcat].[localhost].[/]       : Initializing Spring DispatcherServlet 'dispatcherServlet'
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:06:02.650  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Initializing Servlet 'dispatcherServlet'
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:06:02.658  INFO 1 --- [nio-8080-exec-1] o.s.web.servlet.DispatcherServlet        : Completed initialization in 7 ms
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:06:29.565  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:06:29.925  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:07:29.565  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:07:29.902  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:08:29.565  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:08:29.902  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:09:29.565  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:34:29.869  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:35:29.565  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:35:29.872  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:36:29.565  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:36:29.879  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:37:29.565  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:37:29.867  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:38:29.565  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:38:29.953  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:39:29.565  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:39:29.868  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:40:00.305  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: DELETED for secret26 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:40:00.317  INFO 1 --- [/x.x.x.x/...] d.k.vault.kubernetes.KubernetesService   : Deleted secret secret26 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:40:29.565  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:40:29.860  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:41:18.063  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: ADDED for secret26 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:41:18.088  INFO 1 --- [/x.x.x.x/...] d.k.vault.kubernetes.KubernetesService   : Created secret for vault resource secret26 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:41:29.565  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:41:29.913  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:42:29.568  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:42:29.853  INFO 1 --- [TaskScheduler-1] d.k.vault.kubernetes.KubernetesService   : Modified secret secret26 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:42:30.084  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:43:29.565  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:43:29.865  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:44:29.565  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:44:29.903  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:45:29.565  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:45:29.876  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:46:29.565  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:46:29.886  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:46:34.800  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: MODIFIED for secret26 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:46:34.827  INFO 1 --- [/x.x.x.x/...] d.k.vault.kubernetes.KubernetesService   : Modified secret secret26 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:47:28.099  INFO 1 --- [/x.x.x.x/...] d.koudingspawn.vault.kubernetes.Watcher  : Received action: DELETED for secret26 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:47:28.107  INFO 1 --- [/x.x.x.x/...] d.k.vault.kubernetes.KubernetesService   : Deleted secret secret26 in namespace default
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:47:29.565  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Start refresh of secret...
vault-crd-586bb99858-kg9zc vault-crd 2020-01-27 15:47:29.897  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...

[DaspawnW]

Hi @gaffneyd4,

you are right, it doesn't worked anymore because of a change in the fabric8io kubernetes-client api. I've provided a fix that is released in version 1.4.1. I already tested on one of my clusters and it seems to work again, sorry for the hassle.

Cheers,
Björn