Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail to get Key Vault access through MSI #621

Open
AlexandreGarino opened this issue Mar 11, 2019 · 2 comments

Comments

Projects
None yet
2 participants
@AlexandreGarino
Copy link

commented Mar 11, 2019

Environment

  • Spring boot starter (2.1.3):

    • key vault spring boot starter (2.1.5)
  • OS Type: Windows, Linux

  • Java version:

    • 1.8

Summary

Unable to get access to secrets with MSI enabled.

Reproduce steps

Create a spring boot app from start.spring.io with Azure dependencies.

Create a Azure Key Vault:

        az keyvault create \
            --name <keyvaultname> \
            --sku standard \
            --enabled-for-deployment true \
            --enabled-for-disk-encryption true \
            --enabled-for-template-deployment true

Set a secret:

        az keyvault secret set \
            --vault-name <keyvaultname> \
            --name postgresql10-admin-user \
            --value <password>

Assign managed service identity to the web app:

        az webapp identity assign \
            --name <webappname>

Update security policy settings:

        az keyvault set-policy \
            --name <keyvaultname> \
            --object-id <principalId> \
            --secret-permissions get list

Set web app's settings:

        az webapp config appsettings set \
            --name webappname \
            --settings \
                "AZURE_KEYVAULT_URI=https://<keyvaultname>.vault.azure.net/"

Expected Results

Get secret values from the given Key Vault.

Actual Results

An exception occurs on the application startup:

11-Mar-2019 10:07:38.535 SEVERE [main] org.apache.catalina.core.ContainerBase.addChildInternal ContainerBase.addChild: start:
 org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[localhost].StandardContext[]]
        at org.apache.catalina.util.LifecycleBase.handleSubClassException(LifecycleBase.java:441)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:198)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:695)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1141)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1876)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
        at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1053)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:428)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1585)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:308)
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
        at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:424)
        at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:367)
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:972)
        at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:831)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1432)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1422)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:944)
        at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:261)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:422)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:801)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:695)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:350)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492)
Caused by: java.lang.IllegalStateException: Failed to configure KeyVault property source
        at com.microsoft.azure.keyvault.spring.KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource(KeyVaultEnvironmentPostProcessorHelper.java:76)
        at com.microsoft.azure.keyvault.spring.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:26)
        at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEnvironmentPreparedEvent(ConfigFileApplicationListener.java:177)
        at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEvent(ConfigFileApplicationListener.java:163)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127)
        at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:75)
        at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:54)
        at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:347)
        at org.springframework.boot.SpringApplication.run(SpringApplication.java:306)
        at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.run(SpringBootServletInitializer.java:157)
        at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.createRootApplicationContext(SpringBootServletInitializer.java:137)
        at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:91)
        at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:171)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5098)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        ... 38 more
Caused by: java.lang.RuntimeException: java.io.IOException: Server returned HTTP response code: 400 for URL: http://172.19.0.2:8081/msi/token?resource=https://<keyvaultname>
.vault.azure.net/&api-version=2017-09-01
        at rx.exceptions.Exceptions.propagate(Exceptions.java:57)
        at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:463)
        at rx.observables.BlockingObservable.single(BlockingObservable.java:340)
        at com.microsoft.azure.keyvault.KeyVaultClientImpl.getSecrets(KeyVaultClientImpl.java:2951)
        at com.microsoft.azure.keyvault.KeyVaultClient.listSecrets(KeyVaultClient.java:911)
        at com.microsoft.azure.keyvault.spring.KeyVaultOperation.fillSecretsHashMap(KeyVaultOperation.java:111)
        at com.microsoft.azure.keyvault.spring.KeyVaultOperation.<init>(KeyVaultOperation.java:43)
        at com.microsoft.azure.keyvault.spring.KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource(KeyVaultEnvironmentPostProcessorHelper.java:66)
        ... 55 more
Caused by: java.io.IOException: Server returned HTTP response code: 400 for URL: http://172.19.0.2:8081/msi/token?resource=https://<keyvaultname>.vault.azure.net/&api-versio
n=2017-09-01
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
        at com.microsoft.azure.credentials.AppServiceMSICredentials.getToken(AppServiceMSICredentials.java:71)
        at com.microsoft.azure.credentials.AzureTokenCredentials.getToken(AzureTokenCredentials.java:39)
        at com.microsoft.azure.credentials.AzureTokenCredentialsInterceptor.intercept(AzureTokenCredentialsInterceptor.java:36)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.rest.interceptors.BaseUrlHandler.intercept(BaseUrlHandler.java:43)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at com.microsoft.rest.interceptors.RequestIdHeaderInterceptor.intercept(RequestIdHeaderInterceptor.java:29)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
        at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
        at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
        at okhttp3.RealCall.execute(RealCall.java:77)
        at retrofit2.OkHttpCall.execute(OkHttpCall.java:180)
        at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:40)
        at retrofit2.adapter.rxjava.CallExecuteOnSubscribe.call(CallExecuteOnSubscribe.java:24)
        at rx.Observable.unsafeSubscribe(Observable.java:10327)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:48)
        at rx.internal.operators.OnSubscribeMap.call(OnSubscribeMap.java:33)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
        at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
        at rx.Observable.subscribe(Observable.java:10423)
        at rx.Observable.subscribe(Observable.java:10390)
        at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:443)
        ... 61 more
11-Mar-2019 10:07:38.554 SEVERE [main] org.apache.catalina.startup.HostConfig.deployDirectory Error deploying web application directory [/home/site/wwwroot/webapps/ROOT]
 java.lang.IllegalStateException: ContainerBase.addChild: start: org.apache.catalina.LifecycleException: Failed to start component [StandardEngine[Catalina].StandardHost[lo
calhost].StandardContext[]]
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:747)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:695)
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1141)
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1876)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
        at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1053)
        at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:428)
        at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1585)
        at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:308)
        at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123)
        at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:424)
        at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:367)
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:972)
        at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:831)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1432)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1422)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75)
        at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
        at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:944)
        at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:261)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardService.startInternal(StandardService.java:422)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:801)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.startup.Catalina.start(Catalina.java:695)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:350)
        at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492)

Workaround

Upgrade the com.microsoft.azure:azure-keyvault dependency to the latest version through Maven dependency management:

	<dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>com.microsoft.azure</groupId>
				<artifactId>azure-spring-boot-bom</artifactId>
				<version>${azure.version}</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
			<dependency>
				<groupId>com.microsoft.azure</groupId>
				<artifactId>azure-keyvault</artifactId>
				<version>1.2.0</version>
			</dependency>
		</dependencies>
	</dependencyManagement>
@sophiaso

This comment has been minimized.

Copy link
Member

commented Mar 26, 2019

@AlexandreGarino Thanks for reporting issue, could you share under which Azure service with MSI enabled did you deploy your application and you have met the failure?

@AlexandreGarino

This comment has been minimized.

Copy link
Author

commented Apr 2, 2019

@sophiaso I use a Web App running on Linux. The runtime stack is Java 8 / Tomcat 9.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.