Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to restart application when vault extension is present #226

Closed
vsevel opened this issue Jan 25, 2024 · 3 comments
Closed

Unable to restart application when vault extension is present #226

vsevel opened this issue Jan 25, 2024 · 3 comments

Comments

@vsevel
Copy link
Contributor

vsevel commented Jan 25, 2024

The vault extension config source blocks the application from restarting in dev mode (when vault itself is not running using devservice).

To reproduce:

start vault:

docker run --rm --cap-add=IPC_LOCK -e VAULT_ADDR=http://localhost:8200 -p 8200:8200 -d --name=dev-vault hashicorp/vault:1.15.2
docker logs dev-vault
docker exec -it dev-vault sh
export VAULT_TOKEN=...
vault kv put secret/myapps/vault-quickstart/config a-private-key=123456

start the application:

mvn compile quarkus:dev
% curl localhost:8080/hello/test
result => Optional[123456]

force restart the application with: s. This fails with:

2024-01-25 18:17:37,466 ERROR [io.qua.run.Application] (Quarkus Main Thread) Failed to start application (with profile [dev]) [Error Occurred After Shutdown]: java.lang.RuntimeException: Failed to start quarkus
	at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
	at io.quarkus.runtime.Application.start(Application.java:101)
	at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
	at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
	at io.quarkus.runner.GeneratedMain.main(Unknown Source)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at io.quarkus.runner.bootstrap.StartupActionImpl$1.run(StartupActionImpl.java:113)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.util.concurrent.RejectedExecutionException
	at org.jboss.threads.RejectingExecutor.execute(RejectingExecutor.java:38)
	at org.jboss.threads.EnhancedQueueExecutor.rejectShutdown(EnhancedQueueExecutor.java:2136)
	at org.jboss.threads.EnhancedQueueExecutor$SchedulerTask.schedule(EnhancedQueueExecutor.java:3094)
	at org.jboss.threads.EnhancedQueueExecutor.schedule(EnhancedQueueExecutor.java:936)
	at io.smallrye.mutiny.operators.uni.UniFailOnTimeout$UniFailOnTimeoutProcessor.onSubscribe(UniFailOnTimeout.java:50)
	at io.smallrye.mutiny.vertx.AsyncResultUni.subscribe(AsyncResultUni.java:27)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniFailOnTimeout.subscribe(UniFailOnTimeout.java:36)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniOnFailureTransform.subscribe(UniOnFailureTransform.java:31)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniOnFailureTransform.subscribe(UniOnFailureTransform.java:31)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniOnFailureTransform.subscribe(UniOnFailureTransform.java:31)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniOnFailureTransform.subscribe(UniOnFailureTransform.java:31)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransform.subscribe(UniOnItemTransform.java:22)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.performInnerSubscription(UniOnItemTransformToUni.java:81)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni$UniOnItemTransformToUniProcessor.onItem(UniOnItemTransformToUni.java:57)
	at io.smallrye.mutiny.operators.uni.builders.UniCreateFromKnownItem$KnownItemSubscription.forward(UniCreateFromKnownItem.java:38)
	at io.smallrye.mutiny.operators.uni.builders.UniCreateFromKnownItem.subscribe(UniCreateFromKnownItem.java:23)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni.subscribe(UniOnItemTransformToUni.java:25)
	at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
	at io.smallrye.mutiny.operators.uni.UniBlockingAwait.await(UniBlockingAwait.java:60)
	at io.smallrye.mutiny.groups.UniAwait.atMost(UniAwait.java:65)
	at io.smallrye.mutiny.groups.UniAwait.indefinitely(UniAwait.java:46)
	at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:133)
	at io.quarkus.vault.runtime.config.VaultConfigSource.lambda$fetchSecrets$2(VaultConfigSource.java:128)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:128)
	at io.quarkus.vault.runtime.config.VaultConfigSource.lambda$fetchSecrets$1(VaultConfigSource.java:124)
	at java.base/java.util.HashMap.forEach(HashMap.java:1421)
	at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecrets(VaultConfigSource.java:124)
	at io.quarkus.vault.runtime.config.VaultConfigSource.fetchSecretsFirstTime(VaultConfigSource.java:104)
	at io.quarkus.vault.runtime.config.VaultConfigSource.getSecretConfig(VaultConfigSource.java:82)
	at io.quarkus.vault.runtime.config.VaultConfigSource.getValue(VaultConfigSource.java:63)
	at io.smallrye.config.ConfigValueConfigSourceWrapper.getConfigValue(ConfigValueConfigSourceWrapper.java:20)
	at io.smallrye.config.SmallRyeConfigSources.getValue(SmallRyeConfigSources.java:29)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.SecretKeysConfigSourceInterceptor.getValue(SecretKeysConfigSourceInterceptor.java:24)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.RelocateConfigSourceInterceptor.getValue(RelocateConfigSourceInterceptor.java:25)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.ProfileConfigSourceInterceptor.getProfileValue(ProfileConfigSourceInterceptor.java:51)
	at io.smallrye.config.ProfileConfigSourceInterceptor.getValue(ProfileConfigSourceInterceptor.java:36)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.LoggingConfigSourceInterceptor.getValue(LoggingConfigSourceInterceptor.java:20)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:43)
	at io.smallrye.config.ExpressionConfigSourceInterceptor.getValue(ExpressionConfigSourceInterceptor.java:35)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.SecretKeysHandlerConfigSourceInterceptor.getValue(SecretKeysHandlerConfigSourceInterceptor.java:25)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.FallbackConfigSourceInterceptor.getValue(FallbackConfigSourceInterceptor.java:24)
	at io.smallrye.config.SmallRyeConfigSourceInterceptorContext.proceed(SmallRyeConfigSourceInterceptorContext.java:20)
	at io.smallrye.config.SmallRyeConfig.getConfigValue(SmallRyeConfig.java:322)
	at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:229)
	at io.smallrye.config.SmallRyeConfig.getOptionalValue(SmallRyeConfig.java:358)
	at io.smallrye.config.SmallRyeConfig.getOptionalValues(SmallRyeConfig.java:372)
	at io.quarkus.resteasy.reactive.common.runtime.JaxRsSecurityConfig273591402Impl.<init>(Unknown Source)
	at java.base/jdk.internalt.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)
	at io.smallrye.config.ConfigMappingLoader.configMappingObject(ConfigMappingLoader.java:65)
	at io.smallrye.config.ConfigMappingContext.constructGroup(ConfigMappingContext.java:82)
	at io.smallrye.config.ConfigMappingContext.constructRoot(ConfigMappingContext.java:78)
	at io.smallrye.config.ConfigMappingProvider.mapConfigurationInternal(ConfigMappingProvider.java:980)
	at io.smallrye.config.ConfigMappingProvider.lambda$mapConfiguration$3(ConfigMappingProvider.java:962)
	at io.smallrye.config.SecretKeys.doUnlocked(SecretKeys.java:28)
	at io.smallrye.config.ConfigMappingProvider.mapConfiguration(ConfigMappingProvider.java:962)
	at io.smallrye.config.ConfigMappings.mapConfiguration(ConfigMappings.java:91)
	at io.smallrye.config.SmallRyeConfigBuilder.build(SmallRyeConfigBuilder.java:699)
	at io.quarkus.runtime.generated.Config.readConfig(Unknown Source)
	at io.quarkus.runtime.generated.Config.createRunTimeConfig(Unknown Source)
	at io.quarkus.deployment.steps.RuntimeConfigSetup.deploy(Unknown Source)
	... 13 more
	Suppressed: java.util.concurrent.RejectedExecutionException: Executor is being shut down
		at org.jboss.threads.EnhancedQueueExecutor.rejectShutdown(EnhancedQueueExecutor.java:2138)
		... 97 more

Tracing the code I can see in the VertxVaultClient, in:

    private <T> Uni<T> exec(HttpRequest<Buffer> request, Object body, Class<T> resultClass, int expectedCode) {
        Uni<HttpResponse<Buffer>> send = body == null ? request.send() : request.sendBuffer(Buffer.buffer(requestBody(body)));

        return send.ifNoItem().after(getRequestTimeout()).failWith(TimeoutException::new)
...

Funny enough send.await().indefinitely().bodyAsString() works and returns the appropriate response from vault.
The error seems related to UniFailOnTimeout$UniFailOnTimeoutProcessor trying to schedule a task.

At this point the vault client is an instance of PrivateVertxVaultClient.

Progressing using the debugger, I was able once to get it to run, as if there was a race condition on the initialization of the executor.

Not too sure at this point if this is a vault extension issue, or a quarkus issue.
upon restart the vault extension should be provided with a non terminated executor. you opinion @radcortez
at the same time, if the config source was using a pure jdk http client, we would not be bothered by this type of issue.
what do you think @kdubb ?

see https://github.com/vsevel/reproducer_vault_devservices
@vsevel
Copy link
Contributor Author

vsevel commented Jan 26, 2024

for your information also @aloubyansky

@kdubb
Copy link
Contributor

kdubb commented Jan 26, 2024

@vsevel We can do that, once if I can resolve the issues in laid out in #215 (i.e., SSL & proxy config)

This was referenced Feb 13, 2024
Closed
Merged
@vsevel
Copy link
Contributor Author

vsevel commented Mar 26, 2024

I checked this is fixed in 4.0.0-alpha.2 see also #215

@vsevel vsevel closed this as completed Mar 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Pure JDK Vault Client for Vault Config Source vsevel/quarkus-vault
2 participants
@kdubb @vsevel