Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Reactor/Netty dependencies issues when using KeyVault and Service Bus libraries #22242

Closed
3 tasks done
Paizo opened this issue Jun 11, 2021 · 21 comments
Closed
3 tasks done
Labels
customer-reported Issues that are reported by GitHub users external to the Azure organization. dependency-issue Issue that is caused by dependency conflicts Functions issue-addressed The Azure SDK team member assisting with this issue believes it to be addressed and ready to close. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that

Comments

@Paizo
Copy link

Paizo commented Jun 11, 2021

Describe the bug

Simple java function that get messages from a service bus queue and fetch a certificate from the keyvault.
During the keyvault initialization a NoSuchMethod error occur

Exception or Stack Trace

Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException : Result: Failure\nException: NoSuchMethodError: io.netty.handler.logging.LoggingHandler.<init>(Ljava/lang/String;Lio/netty/handler/logging/LogLevel;Lio/netty/handler/logging/ByteBufFormat;)V\nStack: java.lang.reflect.InvocationTargetException\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22)\n\tat com.microsoft.azure.functions.worker.broker.JavaMethodExecutorImpl.execute(JavaMethodExecutorImpl.java:54)\n\tat com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:57)\n\tat com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:33)\n\tat com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10)\n\tat com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:45)\n\tat com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:92)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.NoSuchMethodError: io.netty.handler.logging.LoggingHandler.<init>(Ljava/lang/String;Lio/netty/handler/logging/LogLevel;Lio/netty/handler/logging/ByteBufFormat;)V\n\tat reactor.netty.transport.logging.ReactorNettyLoggingHandler.<init>(ReactorNettyLoggingHandler.java:63)\n\tat reactor.netty.transport.logging.AdvancedByteBufFormat.toLoggingHandler(AdvancedByteBufFormat.java:109)\n\tat reactor.netty.http.client.HttpClientConfig.<clinit>(HttpClientConfig.java:635)\n\tat reactor.netty.http.client.HttpClientConnect.<init>(HttpClientConnect.java:84)\n\tat reactor.netty.http.client.HttpClient.create(HttpClient.java:391)\n\tat com.azure.core.http.netty.NettyAsyncHttpClientBuilder.build(NettyAsyncHttpClientBuilder.java:97)\n\tat com.azure.core.http.netty.NettyAsyncHttpClientProvider.createInstance(NettyAsyncHttpClientProvider.java:17)\n\tat com.azure.core.implementation.http.HttpClientProviders.createInstance(HttpClientProviders.java:58)\n\tat com.azure.core.http.HttpClient.createDefault(HttpClient.java:50)\n\tat com.azure.core.http.HttpClient.createDefault(HttpClient.java:40)\n\tat com.azure.core.http.HttpPipelineBuilder.build(HttpPipelineBuilder.java:62)\n\tat com.azure.security.keyvault.certificates.CertificateClientBuilder.buildAsyncClient(CertificateClientBuilder.java:167)\n\tat com.azure.security.keyvault.certificates.CertificateClientBuilder.buildClient(CertificateClientBuilder.java:109)

To Reproduce

create a skeleton java function application using a @ServiceBusQueueTrigger
create a keyvault with the function list/get permissions
try to fetch a certificate from the keyvault from within the function

Code Snippet

        certificateClient = new CertificateClientBuilder()
                .vaultUrl(keyVaultUrl)
                .credential(new ManagedIdentityCredentialBuilder().build())
                .buildClient();
        this.secretClient = new SecretClientBuilder()
                .vaultUrl(keyVaultUrl)
                .credential(new ManagedIdentityCredentialBuilder().build())
                .buildClient();

Expected behavior

No exceptions

Screenshots
If applicable, add screenshots to help explain your problem.

Setup (please complete the following information):

  • OS: azure linux function app service
  • IDE: IntelliJ]
  • Library/Libraries:
com.microsoft.azure.functions:azure-functions-java-library:1.4.2
com.microsoft.azure:azure-servicebus:3.6.4
com.azure:azure-security-keyvault-secrets:4.2.8
com.azure:azure-security-keyvault-certificates:4.1.8
com.azure:azure-identity:1.3.1

maven dependencies tree:

[INFO] +- com.microsoft.azure.functions:azure-functions-java-library:jar:1.4.2:compile
[INFO] +- com.microsoft.azure:azure-servicebus:jar:3.6.4:compile
[INFO] | +- org.apache.qpid:proton-j:jar:0.33.4:compile
[INFO] | +- com.microsoft.azure:qpid-proton-j-extensions:jar:1.2.3:compile
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.30:compile
[INFO] | +- org.asynchttpclient:async-http-client:jar:2.12.1:compile
[INFO] | | +- org.asynchttpclient:async-http-client-netty-utils:jar:2.12.1:compile
[INFO] | | +- io.netty:netty-codec-socks:jar:4.1.48.Final:compile
[INFO] | | +- io.netty:netty-handler-proxy:jar:4.1.48.Final:compile
[INFO] | | +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.48.Final:compile
[INFO] | | +- io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.48.Final:compile
[INFO] | | +- org.reactivestreams:reactive-streams:jar:1.0.3:compile
[INFO] | | +- com.typesafe.netty:netty-reactive-streams:jar:2.0.4:compile
[INFO] | | - com.sun.activation:javax.activation:jar:1.2.0:compile
[INFO] | - com.microsoft.azure:azure-client-authentication:jar:1.7.3:compile
[INFO] | +- com.microsoft.azure:azure-client-runtime:jar:1.7.3:compile
[INFO] | | - com.microsoft.rest:client-runtime:jar:1.7.3:compile
[INFO] | | +- com.squareup.retrofit2:retrofit:jar:2.7.2:compile
[INFO] | | +- com.squareup.okhttp3:okhttp:jar:3.12.6:compile
[INFO] | | | - com.squareup.okio:okio:jar:1.15.0:compile
[INFO] | | +- com.squareup.okhttp3:logging-interceptor:jar:3.12.2:compile
[INFO] | | +- com.squareup.okhttp3:okhttp-urlconnection:jar:3.12.2:compile
[INFO] | | +- com.squareup.retrofit2:converter-jackson:jar:2.7.2:compile
[INFO] | | +- com.fasterxml.jackson.datatype:jackson-datatype-joda:jar:2.10.1:compile
[INFO] | | +- io.reactivex:rxjava:jar:1.3.8:compile
[INFO] | | - com.squareup.retrofit2:adapter-rxjava:jar:2.7.2:compile
[INFO] | +- com.microsoft.azure:adal4j:jar:1.6.4:compile
[INFO] | | - com.google.code.gson:gson:jar:2.8.0:compile
[INFO] | - com.microsoft.azure:azure-annotations:jar:1.10.0:compile
[INFO] +- org.apache.httpcomponents.client5:httpclient5:jar:5.0.3:compile
[INFO] | +- org.apache.httpcomponents.core5:httpcore5:jar:5.0.2:compile
[INFO] | +- org.apache.httpcomponents.core5:httpcore5-h2:jar:5.0.2:compile
[INFO] | - commons-codec:commons-codec:jar:1.13:compile
[INFO] +- com.azure:azure-security-keyvault-secrets:jar:4.2.8:compile
[INFO] | +- com.azure:azure-core:jar:1.16.0:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.2:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-core:jar:2.12.2:compile
[INFO] | | +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.2:compile
[INFO] | | +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.12.2:compile
[INFO] | | +- com.fasterxml.jackson.dataformat:jackson-dataformat-xml:jar:2.12.2:compile
[INFO] | | | +- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.12.2:compile
[INFO] | | | | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile
[INFO] | | | | - jakarta.activation:jakarta.activation-api:jar:1.2.1:compile
[INFO] | | | +- org.codehaus.woodstox:stax2-api:jar:4.2.1:compile
[INFO] | | | - com.fasterxml.woodstox:woodstox-core:jar:6.2.4:compile
[INFO] | | +- io.projectreactor:reactor-core:jar:3.4.5:compile
[INFO] | | - io.netty:netty-tcnative-boringssl-static:jar:2.0.38.Final:compile
[INFO] | - com.azure:azure-core-http-netty:jar:1.9.2:compile
[INFO] | +- io.netty:netty-codec-http2:jar:4.1.63.Final:compile
[INFO] | +- io.netty:netty-transport-native-unix-common:jar:4.1.63.Final:compile
[INFO] | - io.projectreactor.netty:reactor-netty:jar:1.0.6:compile
[INFO] | +- io.projectreactor.netty:reactor-netty-core:jar:1.0.6:compile
[INFO] | | +- io.netty:netty-resolver-dns:jar:4.1.63.Final:compile
[INFO] | | | - io.netty:netty-codec-dns:jar:4.1.63.Final:compile
[INFO] | | - io.netty:netty-resolver-dns-native-macos:jar:osx-x86_64:4.1.63.Final:compile
[INFO] | +- io.projectreactor.netty:reactor-netty-http:jar:1.0.6:compile
[INFO] | - io.projectreactor.netty:reactor-netty-http-brave:jar:1.0.6:runtime
[INFO] | - io.zipkin.brave:brave-instrumentation-http:jar:5.13.3:runtime
[INFO] | - io.zipkin.brave:brave:jar:5.13.3:runtime
[INFO] | - io.zipkin.reporter2:zipkin-reporter-brave:jar:2.16.3:runtime
[INFO] | - io.zipkin.reporter2:zipkin-reporter:jar:2.16.3:runtime
[INFO] | - io.zipkin.zipkin2:zipkin:jar:2.23.2:runtime
[INFO] +- com.azure:azure-security-keyvault-certificates:jar:4.1.8:compile
[INFO] +- com.azure:azure-identity:jar:1.3.1:compile
[INFO] | +- com.microsoft.azure:msal4j:jar:1.10.0:compile
[INFO] | | - com.nimbusds:oauth2-oidc-sdk:jar:9.4:compile
[INFO] | | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
[INFO] | | +- com.nimbusds:content-type:jar:2.1:compile
[INFO] | | +- net.minidev:json-smart:jar:2.4.2:compile (version selected from constraint [1.3.2,2.4.2])
[INFO] | | | - net.minidev:accessors-smart:jar:2.4.2:compile
[INFO] | | | - org.ow2.asm:asm:jar:8.0.1:compile
[INFO] | | +- com.nimbusds:lang-tag:jar:1.5:compile
[INFO] | | - com.nimbusds:nimbus-jose-jwt:jar:9.8.1:compile
[INFO] | +- com.microsoft.azure:msal4j-persistence-extension:jar:1.1.0:compile
[INFO] | | - net.java.dev.jna:jna:jar:5.5.0:compile
[INFO] | +- net.java.dev.jna:jna-platform:jar:5.6.0:compile
[INFO] | - org.linguafranca.pwdb:KeePassJava2:jar:2.1.4:compile
[INFO] | +- org.linguafranca.pwdb:KeePassJava2-kdb:jar:2.1.4:compile
[INFO] | | +- org.linguafranca.pwdb:database:jar:2.1.4:compile
[INFO] | | | - com.madgag.spongycastle:core:jar:1.54.0.0:compile
[INFO] | | - org.jetbrains:annotations:jar:15.0:compile
[INFO] | +- org.linguafranca.pwdb:KeePassJava2-dom:jar:2.1.4:compile
[INFO] | | - org.linguafranca.pwdb:KeePassJava2-kdbx:jar:2.1.4:compile
[INFO] | +- org.linguafranca.pwdb:KeePassJava2-jaxb:jar:2.1.4:compile
[INFO] | - org.linguafranca.pwdb:KeePassJava2-simple:jar:2.1.4:compile
[INFO] | +- org.simpleframework:simple-xml:jar:2.7.1:compile
[INFO] | | +- stax:stax-api:jar:1.0.1:compile
[INFO] | | - stax:stax:jar:1.2.0:compile
[INFO] | +- org.apache.httpcomponents:httpcore:jar:4.4.5:compile
[INFO] | - com.fasterxml:aalto-xml:jar:1.0.0:compile
[INFO] +- xpp3:xpp3:jar:1.1.4c:compile
[INFO] +- org.slf4j:log4j-over-slf4j:jar:1.7.30:compile
[INFO] +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] | - ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] +- org.projectlombok:lombok:jar:1.18.20:provided
[INFO] +- org.junit.jupiter:junit-jupiter:jar:5.7.1:test
[INFO] | +- org.junit.jupiter:junit-jupiter-api:jar:5.7.1:test
[INFO] | | +- org.apiguardian:apiguardian-api:jar:1.1.0:test
[INFO] | | +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] | | - org.junit.platform:junit-platform-commons:jar:1.7.1:test
[INFO] | +- org.junit.jupiter:junit-jupiter-params:jar:5.7.1:test
[INFO] | - org.junit.jupiter:junit-jupiter-engine:jar:5.7.1:test
[INFO] | - org.junit.platform:junit-platform-engine:jar:1.7.1:test
[INFO] +- org.mockito:mockito-core:jar:3.9.0:test
[INFO] | +- net.bytebuddy:byte-buddy:jar:1.10.20:test
[INFO] | +- net.bytebuddy:byte-buddy-agent:jar:1.10.20:test
[INFO] | - org.objenesis:objenesis:jar:3.2:test
[INFO] +- org.mockito:mockito-junit-jupiter:jar:3.9.0:test
[INFO] +- org.mock-server:mockserver-netty:jar:5.11.2:test
[INFO] | +- org.mock-server:mockserver-core:jar:5.11.2:test
[INFO] | | +- com.lmax:disruptor:jar:3.4.2:test
[INFO] | | +- javax.servlet:javax.servlet-api:jar:4.0.1:test
[INFO] | | +- com.jcraft:jzlib:jar:1.1.3:test
[INFO] | | +- com.fasterxml.uuid:java-uuid-generator:jar:4.0.1:test
[INFO] | | +- org.apache.velocity:velocity-engine-scripting:jar:2.2:test
[INFO] | | +- org.apache.velocity:velocity-engine-core:jar:2.2:test
[INFO] | | +- net.javacrumbs.json-unit:json-unit-core:jar:2.19.0:test
[INFO] | | | - org.hamcrest:hamcrest-core:jar:2.2:test
[INFO] | | | - org.hamcrest:hamcrest:jar:2.2:test
[INFO] | | +- com.github.java-json-tools:json-schema-validator:jar:2.2.14:test
[INFO] | | | +- com.github.java-json-tools:jackson-coreutils-equivalence:jar:1.0:test
[INFO] | | | | - com.github.java-json-tools:jackson-coreutils:jar:2.0:test
[INFO] | | | | - com.github.java-json-tools:msg-simple:jar:1.2:test
[INFO] | | | | - com.github.java-json-tools:btf:jar:1.3:test
[INFO] | | | +- com.github.java-json-tools:json-schema-core:jar:1.2.14:test
[INFO] | | | | +- com.github.java-json-tools:uri-template:jar:0.10:test
[INFO] | | | | - org.mozilla:rhino:jar:1.7.7.2:test
[INFO] | | | +- com.sun.mail:mailapi:jar:1.6.2:test
[INFO] | | | +- joda-time:joda-time:jar:2.10.5:compile
[INFO] | | | +- com.googlecode.libphonenumber:libphonenumber:jar:8.11.1:test
[INFO] | | | - net.sf.jopt-simple:jopt-simple:jar:5.0.4:test
[INFO] | | +- com.jayway.jsonpath:json-path:jar:2.4.0:test
[INFO] | | +- io.swagger.parser.v3:swagger-parser:jar:2.0.23:test
[INFO] | | | +- io.swagger.parser.v3:swagger-parser-v2-converter:jar:2.0.23:test
[INFO] | | | | +- io.swagger:swagger-core:jar:1.6.2:test
[INFO] | | | | | +- io.swagger:swagger-models:jar:1.6.2:test
[INFO] | | | | | | - io.swagger:swagger-annotations:jar:1.6.2:test
[INFO] | | | | | - javax.validation:validation-api:jar:1.1.0.Final:test
[INFO] | | | | +- io.swagger:swagger-parser:jar:1.0.52:test
[INFO] | | | | | - org.slf4j:slf4j-ext:jar:1.7.28:test
[INFO] | | | | +- io.swagger:swagger-compat-spec-parser:jar:1.0.52:test
[INFO] | | | | | - org.apache.httpcomponents:httpclient:jar:4.5.2:test
[INFO] | | | | | - commons-logging:commons-logging:jar:1.2:test
[INFO] | | | | +- io.swagger.core.v3:swagger-models:jar:2.1.5:test
[INFO] | | | | - io.swagger.parser.v3:swagger-parser-core:jar:2.0.23:test
[INFO] | | | - io.swagger.parser.v3:swagger-parser-v3:jar:2.0.23:test
[INFO] | | | +- io.swagger.core.v3:swagger-core:jar:2.1.5:test
[INFO] | | | | +- io.swagger.core.v3:swagger-annotations:jar:2.1.5:test
[INFO] | | | | - jakarta.validation:jakarta.validation-api:jar:2.0.2:test
[INFO] | | | - com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.11.1:test
[INFO] | | | - org.yaml:snakeyaml:jar:1.26:test
[INFO] | | +- javax.xml.bind:jaxb-api:jar:2.4.0-b180830.0359:test
[INFO] | | | - javax.activation:javax.activation-api:jar:1.2.0:test
[INFO] | | +- org.xmlunit:xmlunit-core:jar:2.8.0:test
[INFO] | | +- org.xmlunit:xmlunit-placeholders:jar:2.8.0:test
[INFO] | | - org.apache.commons:commons-text:jar:1.9:test
[INFO] | +- io.netty:netty-buffer:jar:4.1.53.Final:compile
[INFO] | +- io.netty:netty-codec:jar:4.1.53.Final:compile
[INFO] | +- io.netty:netty-codec-http:jar:4.1.53.Final:compile
[INFO] | +- io.netty:netty-common:jar:4.1.53.Final:compile
[INFO] | +- io.netty:netty-handler:jar:4.1.53.Final:compile
[INFO] | | - io.netty:netty-resolver:jar:4.1.53.Final:compile
[INFO] | +- io.netty:netty-transport:jar:4.1.53.Final:compile
[INFO] | +- commons-io:commons-io:jar:2.8.0:test
[INFO] | - com.google.guava:guava:jar:30.0-android:compile
[INFO] | +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] | +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] | +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] | +- org.checkerframework:checker-compat-qual:jar:2.5.5:compile
[INFO] | +- com.google.errorprone:error_prone_annotations:jar:2.3.4:compile
[INFO] | - com.google.j2objc:j2objc-annotations:jar:1.3:compile
[INFO] +- org.mock-server:mockserver-client-java:jar:5.11.2:test
[INFO] | - org.apache.commons:commons-lang3:jar:3.11:compile
[INFO] +- org.mock-server:mockserver-junit-jupiter:jar:5.11.2:test
[INFO] +- com.github.stefanbirkner:system-lambda:jar:1.2.0:test
[INFO] - uk.org.webcompere:system-stubs-jupiter:jar:1.2.0:test
[INFO] - uk.org.webcompere:system-stubs-core:jar:1.2.0:test

Additional context

Deploy the function in a function app service, place a message in the queue, the function get triggered but fails to connect to the keyvault to fetch a certificate.

Tried to mitigate by adding reactor/netty dependencies with no success:

        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-all</artifactId>
            <version>4.1.63.Final</version>
        </dependency>

        <dependency>
            <groupId>io.projectreactor.netty</groupId>
            <artifactId>reactor-netty-http</artifactId>
            <version>3.4.5</version>
        </dependency>

result in chasing other exceptions, with no luck so far:

2021-06-10T09:10:09.804094220Z Caused by: java.lang.NoClassDefFoundError: Could not initialize class reactor.netty.http.client.HttpClientConfig
2021-06-10T09:10:09.804097920Z  at reactor.netty.http.client.HttpClientConnect.<init>(HttpClientConnect.java:84)
2021-06-10T09:10:09.804101820Z  at reactor.netty.http.client.HttpClient.create(HttpClient.java:391)
2021-06-10T09:10:09.804352328Z  at com.azure.core.http.netty.NettyAsyncHttpClientBuilder.build(NettyAsyncHttpClientBuilder.java:97)
2021-06-10T09:10:09.804373429Z  at com.azure.core.http.netty.NettyAsyncHttpClientProvider.createInstance(NettyAsyncHttpClientProvider.java:17)
2021-06-10T09:10:09.804377729Z  at com.azure.core.implementation.http.HttpClientProviders.createInstance(HttpClientProviders.java:58)
2021-06-10T09:10:09.804381429Z  at com.azure.core.http.HttpClient.createDefault(HttpClient.java:50)
2021-06-10T09:10:09.804385030Z  at com.azure.core.http.HttpClient.createDefault(HttpClient.java:40)
2021-06-10T09:10:09.804388630Z  at com.azure.core.http.HttpPipelineBuilder.build(HttpPipelineBuilder.java:62)
2021-06-10T09:10:09.804392230Z  at com.azure.security.keyvault.certificates.CertificateClientBuilder.buildAsyncClient(CertificateClientBuilder.java:167)
2021-06-10T09:10:09.804395930Z  at com.azure.security.keyvault.certificates.CertificateClientBuilder.buildClient(CertificateClientBuilder.java:109)

Information Checklist

Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

  • Bug Description Added
  • Repro Steps Added
  • Setup information Added
@ghost ghost added needs-triage This is a new issue that needs to be triaged to the appropriate team. customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Jun 11, 2021
@Paizo Paizo changed the title [BUG] reactore/netty dependencies issues when using keyvault libraries [BUG] reactor/netty dependencies issues when using keyvault libraries Jun 11, 2021
@alzimmermsft alzimmermsft added dependency-issue Issue that is caused by dependency conflicts KeyVault labels Jun 11, 2021
@ghost ghost removed the needs-triage This is a new issue that needs to be triaged to the appropriate team. label Jun 11, 2021
@alzimmermsft alzimmermsft changed the title [BUG] reactor/netty dependencies issues when using keyvault libraries [BUG] Reactor/Netty dependencies issues when using KeyVault and Service Bus libraries Jun 11, 2021
@alzimmermsft
Copy link
Member

Thank you for submitting this issue @Paizo, could you try using azure-sdk-bom 1.0.3 (guide on using it) to manage your Azure SDKs?

Looking at the dependencies you're using the Service Bus version being used is resulting in some Netty versions being resolved to a much older version, 4.1.48, compared to what is expected by the KeyVault SDKs, 4.1.63. I recommend using the azure-sdk-bom as it will manage both Azure SDK versions which have dependency alignment and the versions of the dependencies being used by the Azure SDKs.

@derekg52
Copy link

derekg52 commented Jun 11, 2021

@alzimmermsft I've had a nearly identical situation rattling around for a week or so.

[2021-06-11T19:33:28.193Z] Caused by: java.lang.NoSuchMethodError: io.netty.handler.logging.LoggingHandler.(Ljava/lang/String;Lio/netty/handler/logging/LogLevel;Lio/netty/handler/logging/ByteBufFormat;)V
[2021-06-11T19:33:28.193Z] at reactor.netty.transport.logging.ReactorNettyLoggingHandler.(ReactorNettyLoggingHandler.java:63)
[2021-06-11T19:33:28.199Z] at reactor.netty.transport.logging.AdvancedByteBufFormat.toLoggingHandler(AdvancedByteBufFormat.java:109)
[2021-06-11T19:33:28.200Z] at reactor.netty.http.client.HttpClientConfig.(HttpClientConfig.java:635)
[2021-06-11T19:33:28.201Z] at reactor.netty.http.client.HttpClientConnect.(HttpClientConnect.java:84)
[2021-06-11T19:33:28.202Z] at reactor.netty.http.client.HttpClient.create(HttpClient.java:391)
[2021-06-11T19:33:28.203Z] at com.azure.core.http.netty.NettyAsyncHttpClientBuilder.build(NettyAsyncHttpClientBuilder.java:97)
[2021-06-11T19:33:28.204Z] at com.azure.core.http.netty.NettyAsyncHttpClientProvider.createInstance(NettyAsyncHttpClientProvider.java:17)
[2021-06-11T19:33:28.205Z] at com.azure.core.implementation.http.HttpClientProviders.createInstance(HttpClientProviders.java:58)
[2021-06-11T19:33:28.206Z] at com.azure.core.http.HttpClient.createDefault(HttpClient.java:50)
[2021-06-11T19:33:28.207Z] at com.azure.core.http.HttpClient.createDefault(HttpClient.java:40)
[2021-06-11T19:33:28.213Z] at com.azure.core.http.HttpPipelineBuilder.build(HttpPipelineBuilder.java:62)
[2021-06-11T19:33:28.214Z] at com.azure.data.appconfiguration.ConfigurationClientBuilder.buildAsyncClient(ConfigurationClientBuilder.java:203)
[2021-06-11T19:33:28.215Z] at com.azure.data.appconfiguration.ConfigurationClientBuilder.buildClient(ConfigurationClientBuilder.java:124)

I tried your solution of using the 1.03 BOM and the additional recommended dependencies such as Netty but no luck. I tried different variations of what people have mentioned in the past of explicitly including Netty also, cleaned out my .m2 folder locally, etc.

Here's my dependency tree - notice Netty is 4.1.63 here:

[INFO] com.willistowerswatson:BIC_Service:jar:1.0-SNAPSHOT
[INFO] +- com.azure:azure-storage-blob:jar:12.11.1:compile
[INFO] | +- com.azure:azure-core-http-netty:jar:1.9.2:compile
[INFO] | | +- io.netty:netty-handler:jar:4.1.63.Final:compile
[INFO] | | | +- io.netty:netty-common:jar:4.1.63.Final:compile
[INFO] | | | +- io.netty:netty-resolver:jar:4.1.63.Final:compile
[INFO] | | | +- io.netty:netty-transport:jar:4.1.63.Final:compile
[INFO] | | | - io.netty:netty-codec:jar:4.1.63.Final:compile
[INFO] | | +- io.netty:netty-handler-proxy:jar:4.1.63.Final:compile
[INFO] | | | - io.netty:netty-codec-socks:jar:4.1.63.Final:compile
[INFO] | | +- io.netty:netty-buffer:jar:4.1.63.Final:compile
[INFO] | | +- io.netty:netty-codec-http:jar:4.1.63.Final:compile
[INFO] | | +- io.netty:netty-codec-http2:jar:4.1.63.Final:compile
[INFO] | | +- io.netty:netty-transport-native-unix-common:jar:4.1.63.Final:compile
[INFO] | | +- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.63.Final:compile
[INFO] | | - io.netty:netty-transport-native-kqueue:jar:osx-x86_64:4.1.63.Final:compile
[INFO] | +- com.azure:azure-storage-common:jar:12.11.1:compile
[INFO] | - com.azure:azure-storage-internal-avro:jar:12.0.4:compile
[INFO] +- com.azure:azure-cosmos:jar:4.15.0:compile
[INFO] | +- com.fasterxml.jackson.module:jackson-module-afterburner:jar:2.12.2:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.2:compile
[INFO] | +- com.fasterxml.jackson.core:jackson-core:jar:2.12.2:compile
[INFO] | +- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.12.2:compile
[INFO] | +- org.hdrhistogram:HdrHistogram:jar:2.1.12:compile
[INFO] | +- io.dropwizard.metrics:metrics-core:jar:4.1.19:compile
[INFO] | - io.micrometer:micrometer-core:jar:1.6.6:compile
[INFO] | - org.latencyutils:LatencyUtils:jar:2.0.3:runtime
[INFO] +- com.azure:azure-identity:jar:1.3.0:compile
[INFO] | +- com.microsoft.azure:msal4j:jar:1.10.0:compile
[INFO] | | - com.nimbusds:oauth2-oidc-sdk:jar:9.4:compile
[INFO] | | +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
[INFO] | | +- com.nimbusds:content-type:jar:2.1:compile
[INFO] | | +- net.minidev:json-smart:jar:2.4.2:compile (version selected from constraint [1.3.2,2.4.2])
[INFO] | | | - net.minidev:accessors-smart:jar:2.4.2:compile
[INFO] | | | - org.ow2.asm:asm:jar:8.0.1:compile
[INFO] | | +- com.nimbusds:lang-tag:jar:1.5:compile
[INFO] | | - com.nimbusds:nimbus-jose-jwt:jar:9.8.1:compile
[INFO] | +- com.microsoft.azure:msal4j-persistence-extension:jar:1.1.0:compile
[INFO] | | - net.java.dev.jna:jna:jar:5.5.0:compile
[INFO] | +- net.java.dev.jna:jna-platform:jar:5.6.0:compile
[INFO] | - org.linguafranca.pwdb:KeePassJava2:jar:2.1.4:compile
[INFO] | +- org.linguafranca.pwdb:KeePassJava2-kdb:jar:2.1.4:compile
[INFO] | | +- org.linguafranca.pwdb:database:jar:2.1.4:compile
[INFO] | | | - com.madgag.spongycastle:core:jar:1.54.0.0:compile
[INFO] | | - org.jetbrains:annotations:jar:15.0:compile
[INFO] | +- org.linguafranca.pwdb:KeePassJava2-dom:jar:2.1.4:compile
[INFO] | | - org.linguafranca.pwdb:KeePassJava2-kdbx:jar:2.1.4:compile
[INFO] | +- org.linguafranca.pwdb:KeePassJava2-jaxb:jar:2.1.4:compile
[INFO] | - org.linguafranca.pwdb:KeePassJava2-simple:jar:2.1.4:compile
[INFO] | +- org.simpleframework:simple-xml:jar:2.7.1:compile
[INFO] | | +- stax:stax-api:jar:1.0.1:compile
[INFO] | | +- stax:stax:jar:1.2.0:compile
[INFO] | | - xpp3:xpp3:jar:1.1.3.3:compile
[INFO] | +- org.apache.httpcomponents:httpcore:jar:4.4.5:compile
[INFO] | - com.fasterxml:aalto-xml:jar:1.0.0:compile
[INFO] +- com.azure:azure-security-keyvault-secrets:jar:4.2.8:compile
[INFO] +- com.azure:azure-data-appconfiguration:jar:1.1.12:compile
[INFO] +- com.azure:azure-core:jar:1.16.0:compile
[INFO] | +- com.fasterxml.jackson.dataformat:jackson-dataformat-xml:jar:2.12.2:compile
[INFO] | | +- com.fasterxml.jackson.module:jackson-module-jaxb-annotations:jar:2.12.2:compile
[INFO] | | | +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.2:compile
[INFO] | | | - jakarta.activation:jakarta.activation-api:jar:1.2.1:compile
[INFO] | | +- org.codehaus.woodstox:stax2-api:jar:4.2.1:compile
[INFO] | | - com.fasterxml.woodstox:woodstox-core:jar:6.2.4:compile
[INFO] | +- io.projectreactor:reactor-core:jar:3.4.5:compile
[INFO] | | - org.reactivestreams:reactive-streams:jar:1.0.3:compile
[INFO] | - io.netty:netty-tcnative-boringssl-static:jar:2.0.38.Final:compile
[INFO] +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.2:compile
[INFO] +- io.projectreactor.netty:reactor-netty:jar:1.0.6:compile
[INFO] | +- io.projectreactor.netty:reactor-netty-core:jar:1.0.6:compile
[INFO] | | +- io.netty:netty-resolver-dns:jar:4.1.63.Final:compile
[INFO] | | | - io.netty:netty-codec-dns:jar:4.1.63.Final:compile
[INFO] | | - io.netty:netty-resolver-dns-native-macos:jar:osx-x86_64:4.1.63.Final:compile
[INFO] | +- io.projectreactor.netty:reactor-netty-http:jar:1.0.6:compile
[INFO] | - io.projectreactor.netty:reactor-netty-http-brave:jar:1.0.6:runtime
[INFO] | - io.zipkin.brave:brave-instrumentation-http:jar:5.13.3:runtime
[INFO] | - io.zipkin.brave:brave:jar:5.13.3:runtime
[INFO] | - io.zipkin.reporter2:zipkin-reporter-brave:jar:2.16.3:runtime
[INFO] | - io.zipkin.reporter2:zipkin-reporter:jar:2.16.3:runtime
[INFO] | - io.zipkin.zipkin2:zipkin:jar:2.23.2:runtime
[INFO] +- org.slf4j:slf4j-api:jar:1.7.30:compile
[INFO] +- com.auth0:jwks-rsa:jar:0.11.0:compile
[INFO] | +- commons-codec:commons-codec:jar:1.13:compile
[INFO] | +- commons-io:commons-io:jar:2.6:runtime
[INFO] | - com.google.guava:guava:jar:27.1-jre:compile
[INFO] | +- com.google.guava:failureaccess:jar:1.0.1:compile
[INFO] | +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:compile
[INFO] | +- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] | +- org.checkerframework:checker-qual:jar:2.5.2:compile
[INFO] | +- com.google.errorprone:error_prone_annotations:jar:2.2.0:compile
[INFO] | +- com.google.j2objc:j2objc-annotations:jar:1.1:compile
[INFO] | - org.codehaus.mojo:animal-sniffer-annotations:jar:1.17:compile
[INFO] +- com.microsoft.azure.functions:azure-functions-java-library:jar:1.4.0:compile
[INFO] +- javax.servlet:javax.servlet-api:jar:4.0.1:provided
[INFO] +- com.microsoft.graph:microsoft-graph:jar:1.7.1:compile
[INFO] | +- com.sun.jersey:jersey-server:jar:1.19.4:compile
[INFO] | | - com.sun.jersey:jersey-core:jar:1.19.4:compile
[INFO] | | - javax.ws.rs:jsr311-api:jar:1.1.1:compile
[INFO] | +- com.microsoft.graph:microsoft-graph-core:jar:1.0.0:compile
[INFO] | | +- com.googlecode.json-simple:json-simple:jar:1.1:compile
[INFO] | | - com.squareup.okhttp3:okhttp:jar:4.8.1:compile
[INFO] | | +- com.squareup.okio:okio:jar:2.7.0:compile
[INFO] | | | - org.jetbrains.kotlin:kotlin-stdlib-common:jar:1.3.70:compile
[INFO] | | - org.jetbrains.kotlin:kotlin-stdlib:jar:1.3.72:compile
[INFO] | - org.apache.commons:commons-math3:jar:3.6.1:compile
[INFO] +- com.google.code.gson:gson:jar:2.8.5:compile
[INFO] +- org.json:json:jar:20090211:compile
[INFO] +- org.apache.commons:commons-lang3:jar:3.9:compile
[INFO] +- com.microsoft.azure:applicationinsights-web-auto:jar:2.5.0:compile
[INFO] +- javax.mail:mail:jar:1.5.0-b01:compile
[INFO] | - javax.activation:activation:jar:1.1:compile
[INFO] +- com.oracle:ojdbc:jar:8:system
[INFO] +- org.apache.commons:commons-dbcp2:jar:2.7.0:compile
[INFO] | +- org.apache.commons:commons-pool2:jar:2.7.0:compile
[INFO] | - commons-logging:commons-logging:jar:1.2:compile
[INFO] +- org.junit.jupiter:junit-jupiter:jar:5.4.2:test
[INFO] | +- org.junit.jupiter:junit-jupiter-api:jar:5.4.2:test
[INFO] | | +- org.apiguardian:apiguardian-api:jar:1.0.0:test
[INFO] | | +- org.opentest4j:opentest4j:jar:1.1.1:test
[INFO] | | - org.junit.platform:junit-platform-commons:jar:1.4.2:test
[INFO] | +- org.junit.jupiter:junit-jupiter-params:jar:5.4.2:test
[INFO] | - org.junit.jupiter:junit-jupiter-engine:jar:5.4.2:test
[INFO] | - org.junit.platform:junit-platform-engine:jar:1.4.2:test
[INFO] +- org.mockito:mockito-core:jar:2.23.4:test
[INFO] | +- net.bytebuddy:byte-buddy:jar:1.9.3:test
[INFO] | +- net.bytebuddy:byte-buddy-agent:jar:1.9.3:test
[INFO] | - org.objenesis:objenesis:jar:2.6:test
[INFO] - redis.clients:jedis:jar:3.2.0:compile

I had one other hunch that maybe the applicationinsights library may have been conflicting because it comes from a different non-BOM package

com.microsoft.azure applicationinsights-core 2.6.3

but that turned up nothing. I noticed that had it's own shaded version of "LoggingHandler" which seems to be the problem class right now.

@Paizo
Copy link
Author

Paizo commented Jun 14, 2021

thank you @alzimmermsft ,

I added the bom and still failing with NoSuchMethodError: io.netty.handler.logging.LoggingHandler.(Ljava/lang/String;Lio/netty/handler/logging/LogLevel;Lio/netty/handler/logging/ByteBufFormat;), pom.xml snippet:

    <azure.functions.java.library.version>1.4.2</azure.functions.java.library.version>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>com.azure</groupId>
                <artifactId>azure-sdk-bom</artifactId>
                <version>1.0.3</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>


    <dependencies>
        <dependency>
            <groupId>com.microsoft.azure.functions</groupId>
            <artifactId>azure-functions-java-library</artifactId>
            <version>${azure.functions.java.library.version}</version>
        </dependency>
        <dependency>
            <groupId>com.azure</groupId>
            <artifactId>azure-messaging-servicebus</artifactId>
        </dependency>
        <!--    http components 5    -->
        <dependency>
            <groupId>org.apache.httpcomponents.client5</groupId>
            <artifactId>httpclient5</artifactId>
            <version>5.0.3</version>
        </dependency>
        <!--   Keyvault secrets     -->
        <dependency>
            <groupId>com.azure</groupId>
            <artifactId>azure-security-keyvault-secrets</artifactId>
        </dependency>

        <!--   Keyvault Certificates  -->
        <dependency>
            <groupId>com.azure</groupId>
            <artifactId>azure-security-keyvault-certificates</artifactId>
        </dependency>
        <dependency>
            <groupId>com.azure</groupId>
            <artifactId>azure-identity</artifactId>
            <!--   Fix dependency older that 10 years      -->
            <exclusions>
                <exclusion>
                    <groupId>xpp3</groupId>
                    <artifactId>xpp3</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>xpp3</groupId>
            <artifactId>xpp3</artifactId>
            <version>1.1.4c</version>
        </dependency>
        <!--   tentative fix azure libraries NoSuchMethodError/NoClassDef  
        https://github.com/Azure/azure-sdk-for-java/issues/19041   
        https://github.com/Azure/azure-sdk-for-java/issues/22242 -->
        <dependency>
            <groupId>io.projectreactor.netty</groupId>
            <artifactId>reactor-netty</artifactId>
        </dependency>
        <dependency>
            <groupId>io.projectreactor.netty</groupId>
            <artifactId>reactor-netty-http</artifactId>
        </dependency>
        <!--   END - fix azure libraries NoSuchMethodError/NoClassDef  
        https://github.com/Azure/azure-sdk-for-java/issues/19041    
        https://github.com/Azure/azure-sdk-for-java/issues/22242 -->

        <!--    Logging    -->
        <dependency>
            <groupId>org.slf4j</groupId>
            <artifactId>log4j-over-slf4j</artifactId>
            <version>1.7.30</version>
        </dependency>
        <dependency>
            <groupId>ch.qos.logback</groupId>
            <artifactId>logback-classic</artifactId>
            <version>1.2.3</version>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <version>1.18.20</version>
            <scope>provided</scope>
        </dependency>

        <!-- Test -->
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>5.7.1</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.mockito</groupId>
            <artifactId>mockito-core</artifactId>
            <version>3.9.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.mockito</groupId>
            <artifactId>mockito-junit-jupiter</artifactId>
            <version>3.9.0</version>
            <scope>test</scope>
        </dependency>
        <!--  Integration test -->
        <dependency>
            <groupId>org.mock-server</groupId>
            <artifactId>mockserver-netty</artifactId>
            <version>5.11.2</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.mock-server</groupId>
            <artifactId>mockserver-client-java</artifactId>
            <version>5.11.2</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.mock-server</groupId>
            <artifactId>mockserver-junit-jupiter</artifactId>
            <version>5.11.2</version>
            <scope>test</scope>
        </dependency>
        <!--  testing code which uses java.lang.System (ie: System.getenv()) -->
        <dependency>
            <groupId>com.github.stefanbirkner</groupId>
            <artifactId>system-lambda</artifactId>
            <version>1.2.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>uk.org.webcompere</groupId>
            <artifactId>system-stubs-jupiter</artifactId>
            <version>1.2.0</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

logs:

6f0f4a663f4f','level':'Error','levelId':4,'processId':18,'exceptionDetails':'Microsoft.Azure.WebJobs.Host.FunctionInvocationException : Exception while executing function: Functions.ArchiveMessageFunction ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException : Result: Failure\nException: NoSuchMethodError: io.netty.handler.logging.LoggingHandler.<init>(Ljava/lang/String;Lio/netty/handler/logging/LogLevel;Lio/netty/handler/logging/ByteBufFormat;)V\nStack: java.lang.reflect.InvocationTargetException\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22)\n\tat com.microsoft.azure.functions.worker.broker.JavaMethodExecutorImpl.execute(JavaMethodExecutorImpl.java:54)\n\tat com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:57)\n\tat com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:33)\n\tat com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10)\n\tat com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:45)\n\tat com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:92)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.NoSuchMethodError: io.netty.handler.logging.LoggingHandler.<init>(Ljava/lang/String;Lio/netty/handler/logging/LogLevel;Lio/netty/handler/logging/ByteBufFormat;)V\n\tat reactor.netty.transport.logging.ReactorNettyLoggingHandler.<init>(ReactorNettyLoggingHandler.java:63)\n\tat reactor.netty.transport.logging.AdvancedByteBufFormat.toLoggingHandler(AdvancedByteBufFormat.java:109)\n\tat reactor.netty.http.client.HttpClientConfig.<clinit>(HttpClientConfig.java:635)\n\tat reactor.netty.http.client.HttpClientConnect.<init>(HttpClientConnect.java:84)\n\tat reactor.netty.http.client.HttpClient.create(HttpClient.java:391)\n\tat com.azure.core.http.netty.NettyAsyncHttpClientBuilder.build(NettyAsyncHttpClientBuilder.java:97)\n\tat com.azure.core.http.netty.NettyAsyncHttpClientProvider.createInstance(NettyAsyncHttpClientProvider.java:17)\n\tat com.azure.core.implementation.http.HttpClientProviders.createInstance(HttpClientProviders.java:58)\n\tat com.azure.core.http.HttpClient.createDefault(HttpClient.java:50)\n\tat com.azure.core.http.HttpClient.createDefault(HttpClient.java:40)\n\tat com.azure.core.http.HttpPipelineBuilder.build(HttpPipelineBuilder.java:62)\n\tat com.azure.security.keyvault.certificates.CertificateClientBuilder.buildAsyncClient(CertificateClientBuilder.java:167)\n\tat com.azure.security.keyvault.certificates.CertificateClientBuilder.buildClient(CertificateClientBuilder.java:109)\n\tat com.abnamro.ibii.repository.AzureCertificateRepository.<init>

if I add the netty-all dependency then the exception is different, NoClassDefFoundError reactor.netty.http.client.HttpClientConfig:

        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-all</artifactId>
        </dependency>

Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException : Result: Failure\nException: NoClassDefFoundError: Could not initialize class reactor.netty.http.client.HttpClientConfig\nStack: java.lang.reflect.InvocationTargetException\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat com.microsoft.azure.functions.worker.broker.JavaMethodInvokeInfo.invoke(JavaMethodInvokeInfo.java:22)\n\tat com.microsoft.azure.functions.worker.broker.JavaMethodExecutorImpl.execute(JavaMethodExecutorImpl.java:54)\n\tat com.microsoft.azure.functions.worker.broker.JavaFunctionBroker.invokeMethod(JavaFunctionBroker.java:57)\n\tat com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:33)\n\tat com.microsoft.azure.functions.worker.handler.InvocationRequestHandler.execute(InvocationRequestHandler.java:10)\n\tat com.microsoft.azure.functions.worker.handler.MessageHandler.handle(MessageHandler.java:45)\n\tat com.microsoft.azure.functions.worker.JavaWorkerClient$StreamingMessagePeer.lambda$onNext$0(JavaWorkerClient.java:92)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\nCaused by: java.lang.NoClassDefFoundError: Could not initialize class reactor.netty.http.client.HttpClientConfig\n\tat reactor.netty.http.client.HttpClientConnect.<init>(HttpClientConnect.java:84)\n\tat reactor.netty.http.client.HttpClient.create(HttpClient.java:391)\n\tat com.azure.core.http.netty.NettyAsyncHttpClientBuilder.build(NettyAsyncHttpClientBuilder.java:97)\n\tat com.azure.core.http.netty.NettyAsyncHttpClientProvider.createInstance(NettyAsyncHttpClientProvider.java:17)\n\tat com.azure.core.implementation.http.HttpClientProviders.createInstance(HttpClientProviders.java:58)\n\tat com.azure.core.http.HttpClient.createDefault(HttpClient.java:50)\n\tat com.azure.core.http.HttpClient.createDefault(HttpClient.java:40)\n\tat com.azure.core.http.HttpPipelineBuilder.build(HttpPipelineBuilder.java:62)\n\tat com.azure.security.keyvault.certificates.CertificateClientBuilder.buildAsyncClient(CertificateClientBuilder.java:167)\n\tat com.azure.security.keyvault.certificates.CertificateClientBuilder.buildClient(CertificateClientBuilder.java:109)\n\tat com.abnamro.ibii.repository.AzureCertificateRepository.<init>

@Paizo
Copy link
Author

Paizo commented Jun 14, 2021

gave it a try without the netty-all and added all the netty/reactor dependencies I can see in the dependency:tree like so:

        <dependency>
            <groupId>io.projectreactor</groupId>
            <artifactId>reactor-core</artifactId>
        </dependency>
        <dependency>
            <groupId>io.projectreactor.netty</groupId>
            <artifactId>reactor-netty</artifactId>
        </dependency>
        <dependency>
            <groupId>io.projectreactor.netty</groupId>
            <artifactId>reactor-netty-core</artifactId>
        </dependency>
        <dependency>
            <groupId>io.projectreactor.netty</groupId>
            <artifactId>reactor-netty-http</artifactId>
        </dependency>
        <dependency>
            <groupId>io.projectreactor.netty</groupId>
            <artifactId>reactor-netty-http-brave</artifactId>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-codec-socks</artifactId>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-handler-proxy</artifactId>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-transport-native-epoll</artifactId>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-transport-native-kqueue</artifactId>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-tcnative-boringssl-static</artifactId>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-codec-http2</artifactId>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-transport-native-unix-common</artifactId>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-resolver-dns</artifactId>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-codec-dns</artifactId>
        </dependency>
        <dependency>
            <groupId>io.netty</groupId>
            <artifactId>netty-codec-http2</artifactId>
        </dependency>

the error is still the same: java.lang.NoClassDefFoundError: Could not initialize class reactor.netty.http.client.HttpClientConfig

@derekg52
Copy link

For my scenario, I reverted back to a pom from a few months ago to take a fresh look. What I noticed is when I first added the Cosmos artifact (with a defined version like 4.15), that's when I saw these errors again. After deleting my local m2 and starting with an old version (4.1) in this case, it worked OK.

What I noticed was Cosmos required a newer netty than my other azure dependencies and from viewing the dependency tree, bumped up netty from 4.1.42 to 4.1.49.

For the moment, I think I can work with this but I have some more analysis to do to see where it finally breaks, and what version of netty that corresponds to.

@derekg52
Copy link

I spent a few more hours wrestling with various dependency inclusions of parts of netty and reactor netty with no luck.

What worked for me was following the guide of switching from using Netty to OkHTTP per (https://docs.microsoft.com/en-us/azure/developer/java/sdk/http-client-pipeline)

It immediately worked with no fuss, I'm going to continue exploring this route. Hopefully this information helps someone else out.

@alzimmermsft
Copy link
Member

Thank you for the additional information @Paizo and @derekg52. For these applications, are they running locally or in Azure Functions? Would it be possible to do a quick run locally to determine if the environment has any impact on dependency resolution. For example, is a custom class loader being used which resolves the Netty dependencies to another version compared to what Maven dependency:tree is reporting as the resolution version.

@derekg52
Copy link

Hey @alzimmermsft - I'm attempting to get my function running again locally though in my Azure dev environment it runs but failed when I moved the artifact over to test. I noticed that in my case, I think Cosmos requires Netty and using OkHTTP doesn't seem to work when it's directed to. Interestingly enough, I was able to run some of the Java mains through eclipse OK though it fails when invoking the API endpoint through the function.

What I have to have printed out at this point it looks like is comparing what the classpath eclipse is running with versus the tree from maven when running the function. I'll get back to you.

@Paizo
Copy link
Author

Paizo commented Jun 14, 2021

thanks @alzimmermsft and @derekg52 ,

tested both locally and on azure function, still have to give a try to okhttp.
This function has an infinite retry and locally I can clearly see it failing for NoSuchMethodError: io.netty.handler.logging.LoggingHandler at the first invocation and the subsequent retries with NoClassDefFoundError: Could not initialize class reactor.netty.http.client.HttpClientConfig

@derekg52
Copy link

@Paizo yep that's where I am, endless loops chasing netty messages even when I try to exclude them to use the okhttp implementation:

[2021-06-14T23:47:06.817Z] Caused by: java.lang.NoSuchMethodError: io.netty.handler.logging.LoggingHandler.(Ljava/lang/String;Lio/netty/handler/logging/LogLevel;Lio/netty/handler/logging/ByteBufFormat;)V
[2021-06-14T23:47:06.818Z] at reactor.netty.transport.logging.ReactorNettyLoggingHandler.(ReactorNettyLoggingHandler.java:63)
[2021-06-14T23:47:06.820Z] at reactor.netty.transport.logging.AdvancedByteBufFormat.toLoggingHandler(AdvancedByteBufFormat.java:109)
[2021-06-14T23:47:06.823Z] at reactor.netty.http.client.HttpClientConfig.(HttpClientConfig.java:635)
[2021-06-14T23:47:06.824Z] at reactor.netty.http.client.HttpClientConnect.(HttpClientConnect.java:84)
[2021-06-14T23:47:06.826Z] at reactor.netty.http.client.HttpClient.create(HttpClient.java:391)
[2021-06-14T23:47:06.827Z] at com.azure.core.http.netty.NettyAsyncHttpClientBuilder.build(NettyAsyncHttpClientBuilder.java:97)
[2021-06-14T23:47:06.828Z] at com.azure.core.http.netty.NettyAsyncHttpClientProvider.createInstance(NettyAsyncHttpClientProvider.java:17)
[2021-06-14T23:47:06.830Z] at com.azure.core.implementation.http.HttpClientProviders.createInstance(HttpClientProviders.java:58)
[2021-06-14T23:47:06.831Z] at com.azure.core.http.HttpClient.createDefault(HttpClient.java:50)
[2021-06-14T23:47:06.835Z] at com.azure.core.http.HttpClient.createDefault(HttpClient.java:40)
[2021-06-14T23:47:06.836Z] at com.azure.core.http.HttpPipelineBuilder.build(HttpPipelineBuilder.java:62)
[2021-06-14T23:47:06.837Z] at com.azure.data.appconfiguration.ConfigurationClientBuilder.buildAsyncClient(ConfigurationClientBuilder.java:203)

I even stood up a new function in VS code and slowly dropped my pieces into it with the same results. I was thinking maybe something from my original pom was setup wrong or something along those lines but it's been consistent whether it was VS code or my Eclipse one.

@alzimmermsft
Copy link
Member

Hi @Paizo and @derekg52, for using OkHttp as the HTTP networking layer instead of Netty, do all of the included Azure SDKs include the azure-core-http-netty exclusion? Based on your dependency trees:

@Paizo

<dependencies>
    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-security-keyvault-secrets</artifactId>
        <version>4.2.8</version>
        <exclusions>
            <exclusion>
                <groupId>com.azure</groupId>
                <artifactId>azure-core-http-netty</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-security-keyvault-certificates</artifactId>
        <version>4.1.8</version>
        <exclusions>
            <exclusion>
                <groupId>com.azure</groupId>
                <artifactId>azure-core-http-netty</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-identity</artifactId>
        <version>1.3.1</version>
        <exclusions>
            <exclusion>
                <groupId>com.azure</groupId>
                <artifactId>azure-core-http-netty</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
</dependencies>

@derekg52

<dependencies>
    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-storage-blob</artifactId>
        <version>12.11.1</version>
        <exclusions>
            <exclusion>
                <groupId>com.azure</groupId>
                <artifactId>azure-core-http-netty</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-cosmos</artifactId>
        <version>4.15.0</version>
        <exclusions>
            <exclusion>
                <groupId>com.azure</groupId>
                <artifactId>azure-core-http-netty</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-identity</artifactId>
        <version>1.3.0</version>
        <exclusions>
            <exclusion>
                <groupId>com.azure</groupId>
                <artifactId>azure-core-http-netty</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-security-keyvault-secrets</artifactId>
        <version>4.2.8</version>
        <exclusions>
            <exclusion>
                <groupId>com.azure</groupId>
                <artifactId>azure-core-http-netty</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>com.azure</groupId>
        <artifactId>azure-data-appconfiguration</artifactId>
        <version>1.1.12</version>
        <exclusions>
            <exclusion>
                <groupId>com.azure</groupId>
                <artifactId>azure-core-http-netty</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
</dependencies>

@alzimmermsft
Copy link
Member

Hi @Paizo and @derekg52, which version of Java is being used by Azure Functions? If you are using Java 8 could you run your application with the FUNCTIONS_WORKER_JAVA_LOAD_APP_LIBS environment variable set to true.

@derekg52
Copy link

derekg52 commented Jun 15, 2021

@alzimmermsft DING DING DING! That was it, thank you a million times, who knew one little setting like that would do it? I'm going to do some more unit testing on my end but I was able to use the BOM definition and also it uses netty still (I removed all the exclusions and the okhttp implementation).

@derekg52
Copy link

Just to put one more stake in this, I checked my configuration and my dev environment function in Azure is Java 11 and the test one (the one that didn't work), was set to Java 8.

@Paizo
Copy link
Author

Paizo commented Jun 15, 2021

yep, that was it. Thank you

Currently using java 1.8 both locally and on azure; now with the bom and no extra libraries such as netty the flag solve the issue.
Is this documented somewhere? Are there other similar flags we need to pay attention?

@alzimmermsft
Copy link
Member

At this time it isn't documented anywhere aside from source code in the azure-functions-java-library library. I'll follow-up to getting this documented in a better location.

@alzimmermsft alzimmermsft added the issue-addressed The Azure SDK team member assisting with this issue believes it to be addressed and ready to close. label Jul 30, 2021
@ghost
Copy link

ghost commented Jul 30, 2021

Hi @Paizo. Thank you for opening this issue and giving us the opportunity to assist. We believe that this has been addressed. If you feel that further discussion is needed, please add a comment with the text “/unresolve” to remove the “issue-addressed” label and continue the conversation.

@ghost
Copy link

ghost commented Aug 6, 2021

Hi @Paizo, since you haven’t asked that we “/unresolve” the issue, we’ll close this out. If you believe further discussion is needed, please add a comment “/unresolve” to reopen the issue.

@AlejandroBlanco
Copy link

@alzimmermsft DING DING DING! That was it, thank you a million times, who knew one little setting like that would do it? I'm going to do some more unit testing on my end but I was able to use the BOM definition and also it uses netty still (I removed all the exclusions and the okhttp implementation).

Can you elaborate how you are using that app setting. I have added it in my local.settings.json like: "FUNCTIONS_WORKER_JAVA_LOAD_APP_LIBS": true and I have added it too in my pom like:

            <appSettings>
                <property>
                    <name>FUNCTIONS_EXTENSION_VERSION</name>
                    <value>~3</value>
                </property>
                <property>
                    <name>FUNCTIONS_WORKER_JAVA_LOAD_APP_LIBS</name>
                    <value>1</value>
                </property>
            </appSettings>

under azure functions maven plugin.

Still, i have no results...

@Paizo
Copy link
Author

Paizo commented Aug 18, 2021

@AlejandroBlanco
I'm using this in local.settings.json

{
  "IsEncrypted": false,
  "Values": {
    ...
    "FUNCTIONS_WORKER_RUNTIME": "java",
    "JAVA_HOME": "/Library/Java/JavaVirtualMachines/jdk1.8.0_271.jdk/Contents/Home",
    "FUNCTIONS_WORKER_JAVA_LOAD_APP_LIBS": "true",
    ... 
  }
}

no changes in the pom.xml but I pass them as app settings during deployment:
image

@derekg52
Copy link

@AlejandroBlanco
I'm using this in local.settings.json

{
  "IsEncrypted": false,
  "Values": {
    ...
    "FUNCTIONS_WORKER_RUNTIME": "java",
    "JAVA_HOME": "/Library/Java/JavaVirtualMachines/jdk1.8.0_271.jdk/Contents/Home",
    "FUNCTIONS_WORKER_JAVA_LOAD_APP_LIBS": "true",
    ... 
  }
}

no changes in the pom.xml but I pass them as app settings during deployment:
image

@AlejandroBlanco - the change I made was from going from Java 8 to 11. That was what made it magically work for me. I didn't try the FUNCTIONS_WORKER_JAVA_LOAD_APP_LIBS setting, although from the comments from @alzimmermsft that should work as well if you keep your stack on Java 8.

@github-actions github-actions bot locked and limited conversation to collaborators Apr 12, 2023
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
customer-reported Issues that are reported by GitHub users external to the Azure organization. dependency-issue Issue that is caused by dependency conflicts Functions issue-addressed The Azure SDK team member assisting with this issue believes it to be addressed and ready to close. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Projects
None yet
Development

No branches or pull requests

4 participants