Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] exception is reported for azure-messaging-servicebus sdk component. ERROR Schedulers: Scheduler worker in group main failed with an uncaught exception #13328

Closed
2 tasks done
MironAtHome opened this issue Jul 20, 2020 · 10 comments
Closed
2 tasks done

[BUG] exception is reported for azure-messaging-servicebus sdk component. ERROR Schedulers: Scheduler worker in group main failed with an uncaught exception #13328

MironAtHome opened this issue Jul 20, 2020 · 10 comments
Assignees
@hemanttanwar
Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Bus ServiceBusTrack2 This label is to manage work item for track 2.
Milestone
[2020] December

Comments

@MironAtHome
Copy link

Describe the bug
Implementing async subscriber to receive messages, near copy from readme example, and running from custom thread from inside Microsoft Azure Workspace Spark Synapse, raises uncaught exception. The exception is raised at the time of setting up receiver. Further on, no messages can be received from the queue and a different exception is raised.

Exception or Stack Trace
Exception #1: 20/07/16 02:10:42 ERROR Schedulers: Scheduler worker in group main failed with an uncaught exception
java.lang.NoSuchMethodError: reactor.core.scheduler.Schedulers.boundedElastic()Lreactor/core/scheduler/Scheduler;
at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.onNext(ServiceBusReceiveLinkProcessor.java:210)
at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.onNext(ServiceBusReceiveLinkProcessor.java:41)
at reactor.core.publisher.FluxRepeatPredicate$RepeatPredicateSubscriber.onNext(FluxRepeatPredicate.java:78)
at reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onNext(FluxPeekFuseable.java:204)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1510)
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1510)
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241)
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1510)
at reactor.core.publisher.MonoIgnoreThen$ThenAcceptInner.onNext(MonoIgnoreThen.java:296)
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1510)
at reactor.core.publisher.MonoIgnoreThen$ThenAcceptInner.onNext(MonoIgnoreThen.java:296)
at reactor.core.publisher.MonoCreate$DefaultMonoSink.success(MonoCreate.java:156)
at com.azure.core.amqp.implementation.ReactorSession.lambda$createConsumer$18(ReactorSession.java:402)
at com.azure.core.amqp.implementation.handler.DispatchHandler.onTimerTask(DispatchHandler.java:36)
at com.azure.core.amqp.implementation.ReactorDispatcher$WorkScheduler.run(ReactorDispatcher.java:130)
at org.apache.qpid.proton.reactor.impl.SelectableImpl.readable(SelectableImpl.java:118)
at org.apache.qpid.proton.reactor.impl.IOHandler.handleQuiesced(IOHandler.java:61)
at org.apache.qpid.proton.reactor.impl.IOHandler.onUnhandled(IOHandler.java:390)
at com.azure.core.amqp.implementation.handler.CustomIOHandler.onUnhandled(CustomIOHandler.java:39)
at org.apache.qpid.proton.engine.BaseHandler.onReactorQuiesced(BaseHandler.java:87)
at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:206)
at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324)
at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:292)
at com.azure.core.amqp.implementation.ReactorExecutor.run(ReactorExecutor.java:82)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:50)
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:27)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Exception #2
Exception in thread "Thread-44" java.util.concurrent.RejectedExecutionException: ReactorDispatcher instance is closed.
at com.azure.core.amqp.implementation.ReactorDispatcher.throwIfSchedulerError(ReactorDispatcher.java:93)
at com.azure.core.amqp.implementation.ReactorDispatcher.invoke(ReactorDispatcher.java:68)
at com.azure.core.amqp.implementation.ReactorReceiver.dispose(ReactorReceiver.java:169)
at com.azure.messaging.servicebus.implementation.ServiceBusReactorReceiver.dispose(ServiceBusReactorReceiver.java:177)
at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.checkAndSetTerminated(ServiceBusReceiveLinkProcessor.java:531)
at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.drainQueue(ServiceBusReceiveLinkProcessor.java:460)
at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.drain(ServiceBusReceiveLinkProcessor.java:450)
at com.azure.messaging.servicebus.implementation.ServiceBusReceiveLinkProcessor.dispose(ServiceBusReceiveLinkProcessor.java:355)
at com.azure.messaging.servicebus.ServiceBusAsyncConsumer.close(ServiceBusAsyncConsumer.java:80)
at com.azure.messaging.servicebus.ServiceBusReceiverAsyncClient.close(ServiceBusReceiverAsyncClient.java:1145)
at com.microsoft.azure.poc.StreamingPipeline.run(ExampleStream.scala:191)
at java.lang.Thread.run(Thread.java:748)

To Reproduce
Please find below is a sanitized version of IntelliJ IDEA project ZIP archive with code and build script prepared. Loading same from the context of Microsoft Azure Workspace Spark Synapse as Scala Job invariable runs into this exception
Project.zip

Code Snippet
please see ZIP archive above

Expected behavior
Message received from Service Bus with no issues

Screenshots
Not applicable

Setup (please complete the following information):
Not applicable to the extent I can follow

Additional context
Similar issue was reported for Microsoft Azure Event Hub Java API/SDK. Based on my understanding of developer dialogs logged in the issue, it appears that the trouble is somewhere between code as such and ability of the code to perform at high throughput.
Please find is the issue link: Azure/azure-event-hubs-spark#363

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

  • [x ] Bug Description Added
  • Repro Steps Added
  • Setup information Added
@ghost ghost added needs-triage This is a new issue that needs to be triaged to the appropriate team. customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that labels Jul 20, 2020
@joshfree joshfree added bug This issue requires a change to an existing behavior in the product in order to be resolved. Client This issue points to a problem in the data-plane of the library. Service Bus labels Jul 20, 2020
@ghost ghost removed the needs-triage This is a new issue that needs to be triaged to the appropriate team. label Jul 20, 2020
@joshfree
Copy link
Member

@hemanttanwar PTAL

@MironAtHome
Copy link
Author

Is PTAL stands for (P)lease (T)ake a (L)ook?
I mean, I put together a very long and nice report. Can't you at least try to pretend, out of politeness, if not par for par, than at least a minimal amount of typing?
So that I get an impression of commitment...

@hemanttanwar hemanttanwar mentioned this issue Aug 17, 2020
Closed
3 tasks
@hemanttanwar hemanttanwar added the ServiceBusTrack2 This label is to manage work item for track 2. label Aug 17, 2020
@conniey conniey added this to the [2020] September milestone Aug 17, 2020
@vinishgeorge
Copy link

+1

@ranjankumar03
Copy link

I am facing same issue .

ERROR 5860 --- [ctor-http-nio-2] reactor.core.publisher.Operators : Operator called default onErrorDropped

java.lang.NoSuchMethodError: reactor.core.scheduler.Schedulers.boundedElastic()Lreactor/core/scheduler/Scheduler;
at com.azure.messaging.eventhubs.PartitionPumpManager.startPartitionPump(PartitionPumpManager.java:180) ~[azure-messaging-eventhubs-5.2.0.jar:na]
at com.azure.messaging.eventhubs.PartitionBasedLoadBalancer.lambda$claimOwnership$23(PartitionBasedLoadBalancer.java:426) ~[azure-messaging-eventhubs-5.2.0.jar:na]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[na:1.8.0_144]
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) ~[na:1.8.0_144]
at com.azure.messaging.eventhubs.PartitionBasedLoadBalancer.lambda$claimOwnership$24(PartitionBasedLoadBalancer.java:426) ~[azure-messaging-eventhubs-5.2.0.jar:na]
at reactor.core.publisher.LambdaMonoSubscriber.onNext(LambdaMonoSubscriber.java:137) ~[reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1515) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:241) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:121) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1515) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:145) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFilter$FilterSubscriber.onComplete(FluxFilter.java:160) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.checkTerminated(FluxFlatMap.java:794) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.drainLoop(FluxFlatMap.java:560) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.drain(FluxFlatMap.java:540) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]
at reactor.core.publisher.FluxFlatMap$FlatMapMain.onComplete(FluxFlatMap.java:426) [reactor-core-3.2.10.RELEASE.jar:3.2.10.RELEASE]

@UmeshRajani
Copy link

Facing same issue, I spent almost 5 hours to resolve and found here that it will be resolved in December milestone, hope so.

@hemanttanwar
Copy link
Contributor

@ranjankumar03
But looking at the error, here is my analysis.

it seems like the setup in which ServiceBus is running is using older version of "projectreactor". The exception trace shows that it uses reactor-core-3.2.10.RELEASE.jar.

You can see that in this version Schedulers.boundedElastic() does not exists, Thus you are getting java.lang.NoSuchMethodError.
3.2.10.RELEASE API Reference: https://projectreactor.io/docs/core/3.2.10.RELEASE/api/reactor/core/scheduler/Schedulers.html

The Service Bus 7.x.x version need version "io.projectreacto:reactor-core:3.3.11.RELEASE".

We configure it here : https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/core/azure-core/pom.xml#L87

Can you ensure that this version is used ?

@hemanttanwar
Copy link
Contributor

@MironAtHome
I have not worked in "Microsoft Azure Workspace Spark Synapse", So I did not setup exactly same Synapse Workspace to replicate this. I will do some tutorials on how to set it up.

Thank you for sending Project.zip Appreciate it.

But I looked at your Stream_jar.xml and it looks like your setup is using "io.projectreactor:reactor-core:3.3.7.RELEASE:jar" ? Is it possible that your setup is using older version of "io.projectreacto:reactor-core" ?
The cause for 'java.lang.NoSuchMethodError' is normally, if your setup is not using correct version of library.

The Service Bus 7.x.x version need version "io.projectreacto:reactor-core:3.3.11.RELEASE".
We configure it here : https://github.com/Azure/azure-sdk-for-java/blob/master/sdk/core/azure-core/pom.xml#L87

Also since your program is failing at ' exception is raised at the time of setting up receiver', it can not be performance related issue. To eliminate the performance issue: May be you can try reading just one message from queue and see if you can read it.

@UmeshRajani
Copy link

@hemanttanwar

Thanks Hemant, it's working perfect as with spring boot it was taking older version of reactor so explicitly added io.projectreacto:reactor-core:3.3.11.RELEASE in pom and works with Service Bus 7.0.0

@hemanttanwar
Copy link
Contributor

@UmeshRajani Happy to see it resolved for you.

@conniey
Copy link
Member

conniey commented Dec 4, 2020

Glad this is resolved for you. NoSuchMethodErrors are often caused by mismatched dependencies. When I encounter these, I check the dependency tree. It'll show you all dependencies and how it decided to resolve the one causing your error.

https://maven.apache.org/plugins/maven-dependency-plugin/examples/resolving-conflicts-using-the-dependency-tree.html

Ex: mvn dependency:tree -Dverbose -Dincludes=reactor*

@conniey conniey closed this as completed Dec 4, 2020
@github-actions github-actions bot locked and limited conversation to collaborators Apr 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@hemanttanwar hemanttanwar

Labels
bug This issue requires a change to an existing behavior in the product in order to be resolved. Client This issue points to a problem in the data-plane of the library. customer-reported Issues that are reported by GitHub users external to the Azure organization. question The issue doesn't require a change to the product in order to be resolved. Most issues start as that Service Bus ServiceBusTrack2 This label is to manage work item for track 2.
Projects
None yet
Milestone
[2020] December
Development

No branches or pull requests
8 participants
@joshfree @hemanttanwar @ranjankumar03 @conniey @UmeshRajani @ramya-rao-a @MironAtHome @vinishgeorge