Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Netty Dependency issues using keyvault in azure-spring-boot 2.2.4 #886

Closed
mocleiri opened this issue May 15, 2020 · 25 comments
Closed

Netty Dependency issues using keyvault in azure-spring-boot 2.2.4 #886

mocleiri opened this issue May 15, 2020 · 25 comments

Comments

@mocleiri
Copy link

mocleiri commented May 15, 2020

Environment

Spring boot starter:

  • active directory spring boot starter

  • cosmosdb spring boot starter

  • key vault spring boot starter

  • service bus spring boot starter

  • application insights spring boot starter

  • sql server database driver

  • msal identity library

  • OS Type: Linux (Azure Web App)

  • Java version:

    • 11

Summary

When starting a spring boot application using azure-spring-boot 2.2.4 it breaks with a dependency issue with netty when trying to look up the keyvault secret.

We only use the keyvault when running in Azure. Everything was fine until we deployed the application and on startup when it tried to resolve the database credentials from keyvault it broke with the below exception caused by incompatible versions of netty.

Switching back to azure-spring-boot 2.2.3 has solved the problem for now.

java.lang.NoSuchMethodError: 'boolean io.netty.handler.ssl.SslProvider.isAlpnSupported(io.netty.handler.ssl.SslProvider)'
at reactor.netty.http.client.HttpClientSecure.(HttpClientSecure.java:79) ~[reactor-netty-0.9.4.RELEASE.jar!/:0.9.4.RELEASE]

Full root cause stack trace:

ERROR org.springframework.boot.SpringApplication - Application run failed
 java.lang.NoSuchMethodError: 'boolean io.netty.handler.ssl.SslProvider.isAlpnSupported(io.netty.handler.ssl.SslProvider)'
     at reactor.netty.http.client.HttpClientSecure.(HttpClientSecure.java:79) ~[reactor-netty-0.9.4.RELEASE.jar!/:0.9.4.RELEASE]
     at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.lambda$subscribe$0(HttpClientConnect.java:297) ~[reactor-netty-0.9.4.RELEASE.jar!/:0.9.4.RELEASE]
     at reactor.core.publisher.MonoCreate.subscribe(MonoCreate.java:57) ~[reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
     at reactor.core.publisher.FluxRetryPredicate$RetryPredicateSubscriber.resubscribe(FluxRetryPredicate.java:124) ~[reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
     at reactor.core.publisher.MonoRetryPredicate.subscribeOrReturn(MonoRetryPredicate.java:51) ~[reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
     at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:48) ~[reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
     at reactor.netty.http.client.HttpClientConnect$MonoHttpConnect.subscribe(HttpClientConnect.java:322) ~[reactor-netty-0.9.4.RELEASE.jar!/:0.9.4.RELEASE]
     at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:55) ~[reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
     at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52) ~[reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
     at reactor.core.publisher.Flux.subscribe(Flux.java:8134) ~[reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
     at reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber.onComplete(FluxConcatArray.java:207) ~[reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
     at reactor.core.publisher.FluxConcatArray.subscribe(FluxConcatArray.java:80) ~[reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
     at reactor.core.publisher.FluxDefer.subscribe(FluxDefer.java:54) ~[reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
     at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:53) ~[reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
     at com.azure.core.util.paging.ContinuablePagedFluxCore.subscribe(ContinuablePagedFluxCore.java:131) ~[azure-core-1.3.0.jar!/:?]
     at com.azure.core.http.rest.PagedFluxBase.subscribe(PagedFluxBase.java:126) ~[azure-core-1.3.0.jar!/:?]
     at reactor.core.publisher.BlockingIterable.iterator(BlockingIterable.java:80) ~[reactor-core-3.3.0.RELEASE.jar!/:3.3.0.RELEASE]
     at com.azure.core.util.IterableStream.iterator(IterableStream.java:85) ~[azure-core-1.3.0.jar!/:?]
     at java.lang.Iterable.forEach(Unknown Source) ~[?:?]
     at com.microsoft.azure.keyvault.spring.KeyVaultOperation.fillSecretsList(KeyVaultOperation.java:125) ~[azure-spring-boot-2.2.4.jar!/:?]
     at com.microsoft.azure.keyvault.spring.KeyVaultOperation.(KeyVaultOperation.java:49) ~[azure-spring-boot-2.2.4.jar!/:?]
     at com.microsoft.azure.keyvault.spring.KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource(KeyVaultEnvironmentPostProcessorHelper.java:66) ~[azure-spring-boot-2.2.4.jar!/:?]
     at com.microsoft.azure.keyvault.spring.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:27) ~[azure-spring-boot-2.2.4.jar!/:?]
     at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEnvironmentPreparedEvent(ConfigFileApplicationListener.java:188) ~[spring-boot-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
     at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEvent(ConfigFileApplicationListener.java:176) ~[spring-boot-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
     at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
     at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
     at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
     at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127) ~[spring-context-5.2.5.RELEASE.jar!/:5.2.5.RELEASE]
     at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:76) ~[spring-boot-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
     at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:53) ~[spring-boot-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
     at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:345) [spring-boot-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
     at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) [spring-boot-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
     at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) [spring-boot-2.2.6.RELEASE.jar!/:2.2.6.RELEASE]
     at ca.utoronto.easi.ats.application.ApplicationConfig.main(ApplicationConfig.java:97) [classes!/:?]
     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
     at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
     at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [app.jar:?]
     at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [app.jar:?]
     at org.springframework.boot.loader.Launcher.launch(Launcher.java:51) [app.jar:?]
     at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:52) [app.jar:?]
 Wait for pid == 124 either returned successfully or was interrupted due to a signal 124

Reproduce steps

Deploy application into Azure Web App Java 11 stack.

Expected Results

Application would startup properly and resolve keyvault properties like it does in the previous version, 2.2.3.

Actual Results

failed with method not found error on acquisition of the keyvault secret due to mixed netty versions.

@saragluna
Copy link
Contributor

Thanks for reaching out and we'll look into it.

@saragluna
Copy link
Contributor

@mocleiri, I guess this could be related to upgrading cosmos SDK version in the 2.2.4 release. Could you help provide your pom file so we could locate the dependency conflicts.

@mocleiri
Copy link
Author

I couldn't upload the xml file directly so I renamed it to a txt file.

pom-xml.txt

@ftokarev
Copy link

ftokarev commented Jun 9, 2020

Fyi, I had the same issue, I solved it by specifying the version explicitly:

implementation 'io.netty:netty-all:4.1.50.Final'

After that I got java.lang.ClassNotFoundException: reactor.core.CorePublisher which I solved again by specifying reactor.core explicitly:

implementation 'io.projectreactor:reactor-core:3.3.5.RELEASE'

@jacko9et
Copy link

jacko9et commented Jul 9, 2020

Any update on this issue?

@andrewholler
Copy link

I am also facing this issue.

@jacko9et
Copy link

@mocleiri @aholler2
You can solved it by specifying the version explicitly:
implementation 'io.netty:netty-all:4.1.50.Final'
implementation 'io.projectreactor:reactor-core:3.3.5.RELEASE'

@saragluna
Copy link
Contributor

saragluna commented Sep 3, 2020

Closing this now... We've addressed this in the PR Azure/azure-sdk-for-java#14506.

@Sreini
Copy link

Sreini commented May 28, 2021

I am getting a similar error with project reactor

 ERROR o.s.boot.SpringApplication               : Application run failed

java.lang.NoSuchMethodError: 'reactor.core.publisher.Mono reactor.core.publisher.Mono.contextWrite(reactor.util.context.ContextView)'
	at com.azure.core.http.rest.RestProxy.handleRestReturnType(RestProxy.java:495) ~[azure-core-1.15.0.jar:na]
	at com.azure.core.http.rest.RestProxy.invoke(RestProxy.java:139) ~[azure-core-1.15.0.jar:na]
	at com.azure.security.keyvault.secrets.$Proxy34.getSecrets(Unknown Source) ~[na:na]
	at com.azure.security.keyvault.secrets.SecretAsyncClient.listSecretsFirstPage(SecretAsyncClient.java:786) ~[azure-security-keyvault-secrets-4.2.7.jar:na]
	at com.azure.security.keyvault.secrets.SecretAsyncClient.lambda$listPropertiesOfSecrets$64(SecretAsyncClient.java:755) ~[azure-security-keyvault-secrets-4.2.7.jar:na]
	at com.azure.core.http.rest.PagedFlux.lambda$new$2(PagedFlux.java:82) ~[azure-core-1.15.0.jar:na]
	at com.azure.core.util.paging.ContinuablePagedByIteratorBase.requestPage(ContinuablePagedByIteratorBase.java:85) ~[azure-core-1.15.0.jar:na]
	at com.azure.core.util.paging.ContinuablePagedByPageIterable$ContinuablePagedByPageIterator.<init>(ContinuablePagedByPageIterable.java:49) ~[azure-core-1.15.0.jar:na]
	at com.azure.core.util.paging.ContinuablePagedByPageIterable.iterator(ContinuablePagedByPageIterable.java:37) ~[azure-core-1.15.0.jar:na]
	at java.base/java.lang.Iterable.spliterator(Iterable.java:101) ~[na:na]
	at com.azure.spring.keyvault.KeyVaultOperation.lambda$refreshProperties$2(KeyVaultOperation.java:142) ~[azure-spring-boot-3.4.0.jar:na]
	at java.base/java.util.Optional.map(Optional.java:265) ~[na:na]
	at com.azure.spring.keyvault.KeyVaultOperation.refreshProperties(KeyVaultOperation.java:142) ~[azure-spring-boot-3.4.0.jar:na]
	at com.azure.spring.keyvault.KeyVaultOperation.<init>(KeyVaultOperation.java:80) ~[azure-spring-boot-3.4.0.jar:na]
	at com.azure.spring.keyvault.KeyVaultEnvironmentPostProcessorHelper.addKeyVaultPropertySource(KeyVaultEnvironmentPostProcessorHelper.java:101) ~[azure-spring-boot-3.4.0.jar:na]
	at com.azure.spring.keyvault.KeyVaultEnvironmentPostProcessor.postProcessEnvironment(KeyVaultEnvironmentPostProcessor.java:47) ~[azure-spring-boot-3.4.0.jar:na]
	at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEnvironmentPreparedEvent(ConfigFileApplicationListener.java:203) ~[spring-boot-2.3.9.RELEASE.jar:2.3.9.RELEASE]
	at org.springframework.boot.context.config.ConfigFileApplicationListener.onApplicationEvent(ConfigFileApplicationListener.java:191) ~[spring-boot-2.3.9.RELEASE.jar:2.3.9.RELEASE]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172) ~[spring-context-5.2.13.RELEASE.jar:5.2.13.RELEASE]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.2.13.RELEASE.jar:5.2.13.RELEASE]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.2.13.RELEASE.jar:5.2.13.RELEASE]
	at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:127) ~[spring-context-5.2.13.RELEASE.jar:5.2.13.RELEASE]
	at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:80) ~[spring-boot-2.3.9.RELEASE.jar:2.3.9.RELEASE]
	at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:53) ~[spring-boot-2.3.9.RELEASE.jar:2.3.9.RELEASE]
	at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:345) ~[spring-boot-2.3.9.RELEASE.jar:2.3.9.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) ~[spring-boot-2.3.9.RELEASE.jar:2.3.9.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1237) ~[spring-boot-2.3.9.RELEASE.jar:2.3.9.RELEASE]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1226) ~[spring-boot-2.3.9.RELEASE.jar:2.3.9.RELEASE]
	at ...]

I see the fix in the PR was only for the netty issue.

@Sreini
Copy link

Sreini commented May 28, 2021

Specifying the reactor version in the pom.xml doesn't resolve the error.

                <groupId>io.projectreactor</groupId>
                <artifactId>reactor-core</artifactId>
                <version>3.3.17.RELEASE</version>
            </dependency>

@chenrujun chenrujun self-assigned this May 31, 2021
@chenrujun
Copy link

Hi, @Sreini, could you please provide your pom.xml?

@kbking01
Copy link

kbking01 commented Jun 10, 2021

Same issue happens on boot load when both secrets and keys dependency are used together,
implementation 'com.azure:azure-security-keyvault-secrets:4.2.7'
implementation 'com.azure:azure-security-keyvault-keys:4.3.0-beta.8'

@chenrujun
Copy link

Hi, @kbking01, here is the dependency of com.azure:azure-security-keyvault-secrets:4.2.7:
image

Could you please check the reactor-core version in your classpath? Or please share your build.gradle.

@kbking01
Copy link

kbking01 commented Jun 11, 2021

Hi @chenrujun - Was using 3.3.5 previously. Changed it to below, still same issue.
io.projectreactor:reactor-core:3.4.6
io.projectreactor.netty:reactor-netty:1.0.7

KeyVaultPropertySource is kicking in from the boot starters though I manually remove the KV source from the environment defined

@chenrujun
Copy link

Hi, @kbking01 , it's a dependency version problem. Could you please share you build.gradle?

@andros0689
Copy link

Hello, @kbking01 , I'm facing the same problem trying to resolve this dependency:

java.lang.NoSuchMethodError: 'reactor.core.publisher.Mono reactor.core.publisher.Mono.contextWrite(reactor.util.context.ContextView)'
at com.azure.core.http.rest.RestProxy.handleRestReturnType(RestProxy.java:528)
at com.azure.core.http.rest.RestProxy.invoke(RestProxy.java:146)
at com.sun.proxy.$Proxy15.getSecrets(Unknown Source)
at com.azure.security.keyvault.secrets.SecretAsyncClient.listSecretsFirstPage(SecretAsyncClient.java:976)

Please, is there a solution?

Thanks

@chenrujun
Copy link

Hi, @andros0689

Thanks for reaching out.

Could you please create a new issue, including all information to reproduce the problem. Like the version of spring-boot and spring-cloud-azure-starter-keyvault-secrets.

It's better to create a simple project to reproduce the problem.

@andros0689
Copy link

andros0689 commented Sep 20, 2022

Hello @chenrujun THANKS

Actually, I just found out with my team the solution based on the BOM version (Fuse 11, java 11, Spring Boot based):

	<dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>org.jboss.redhat-fuse</groupId>
				<artifactId>fuse-springboot-bom</artifactId>
				<version>7.11.0.fuse-sb2-7_11_0-00028-redhat-00001</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
		</dependencies>
	</dependencyManagement>`

		<dependency>
			<groupId>com.azure.spring</groupId>
			<artifactId>spring-cloud-azure-starter-keyvault-secrets</artifactId>
			<version>${version.spring.cloud.azure}</version>
			<exclusions>
				<exclusion>
					<groupId>io.netty</groupId>
					<artifactId>netty-resolver-dns-native-macos</artifactId>
				</exclusion>
				<exclusion>
					<groupId>io.netty</groupId>
					<artifactId>netty-transport-native-kqueue</artifactId>
				</exclusion>
			</exclusions>
		</dependency>

Doesn't need the mac os jar from io.netty:netty-resolver-dns-native-macos and io.netty:netty-transport-native-kqueue

We were using the Fuse version 7.10.0.fuse-sb2-7_10_1-00008-redhat-00001.

Additional, exclude the class ServiceBusJmsAutoConfiguration.class in the Application.java:

@SpringBootApplication(exclude = ServiceBusJmsAutoConfiguration.class)
@ImportResource
public class Application {
    public static void main(String[] args) {
    	
    	SpringApplication.run(Application.class, args);
    }
}

@chenrujun
Copy link

Hi, @andros0689
Please update this project: https://github.com/chenrujun/azure-spring-boot-issue-886
Then I can reproduce the problem.

@andros0689
Copy link

andros0689 commented Sep 20, 2022

Hello @chenrujun

I updated the project with the property "fuse.bom.version.test" in the POM. The one is not commented reproduces the error:

java.lang.NoSuchMethodError: 'reactor.core.publisher.Mono reactor.core.publisher.Mono.contextWrite(reactor.util.context.ContextView)' at com.azure.core.http.rest.RestProxy.handleRestReturnType(RestProxy.java:528) at com.azure.core.http.rest.RestProxy.invoke(RestProxy.java:146)

The property "fuse.bom.version.test" commented (7.11.0.fuse-sb2-7_11_0-00028-redhat-00001) resolves it (Tries to resolve the DNS to the KeyVault Service).

Hope it helps to resolve the bug (The jars that points to the mac OS build).

Note: Just in case, run the project with "mvn spring-boot:run"

Additional note: The exclusion of the class "com.azure.spring.cloud.autoconfigure.jms.ServiceBusJmsAutoConfiguration" in the Application.java is because of in our project don't need it. The project required it in order to finish the compilation, nevertheless, we don't need that implementation.

Thanks a lot.

@chenrujun
Copy link

@andros0689

Current project has error when I try to run it. Could you please fix it?
Here is the screenshot about the error:
image

@andros0689
Copy link

andros0689 commented Sep 20, 2022

@chenrujun Did you pull the correct version?

image

I just reviewed and I commited the source with the correct property

:/

@chenrujun
Copy link

@andros0689

I already pulled the latest code. It has error.

See this:

https://github.com/chenrujun/azure-spring-boot-issue-886/blob/master/pom.xml#L119

image
image

@andros0689
Copy link

@chenrujun

Hello, please, verify again

@chenrujun
Copy link

<fuse.bom.version.test>7.11.0.fuse-sb2-7_11_0-00028-redhat-00001</fuse.bom.version.test>

reactor-core's version is 3.4.17

image

<fuse.bom.version.test>7.10.0.fuse-sb2-7_10_1-00008-redhat-00001</fuse.bom.version.test>

reactor-core's version is 3.3.17

image

To fix the problem of java.lang.NoSuchMethodError: 'reactor.core.publisher.Mono, reactor-core's version should be 3.4.17.

Adding the following fragment can set reactor-core's version to 3.4.17

image

2020.0.18 is decided by here: https://repo1.maven.org/maven2/io/projectreactor/reactor-bom/2020.0.18/reactor-bom-2020.0.18.pom

But it will have new problem like this:

image

Em... Let's change another way to investigate this problem:

I found this: https://github.com/jboss-fuse/redhat-fuse/blob/release-work-7.10.0.fuse-sb2-7_10_1-00/pom.xml#L66
Seems 7.10.0.fuse-sb2-7_10_1-00 is using spring-boot:2.3.12.RELEASE

But spring-cloud-azure:4.3.0 is compatible with spring-boot 2.5/2.6/2.7
Refs: https://github.com/Azure/azure-sdk-for-java/wiki/Spring-Versions-Mapping

So, spring-cloud-azure:4.3.0 is not compatible 7.10.0.fuse-sb2-7_10_1-00.

spring-cloud-azure:4.3.0 is compatible 7.11.0.fuse-sb2-7_11_0-00, because 7.11.0.fuse-sb2-7_11_0-00 is using spring-boot 2.5.13.

Refs: https://github.com/jboss-fuse/redhat-fuse/blob/release-work-7.11.0.fuse-sb2-7_11_0-00/pom.xml#L65

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants