Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Service Bus] Fix MaxListenersExceeded for management client #11738

Conversation

HarshaNalluru
Copy link
Member

Issue

#11736

Cause

Consider renewlock attempts being made in parallel for multiple messages, and the management link is not initialized.

  • _init is called from the _makeManagementRequest method multiple times since the AMQP link is not open.
  • Link gets initialized for one of those _init calls.
  • Upon initializing, each of the paths get to add their own listeners to "sender_error" and "receiver_error" events to the same link.
  • If there are too many listeners, MaxListenersExceeded gets thrown.

Fix

Updating the "adding listeners" code inside _init under managementClient with the if-checks to add the listeners only if they are not present already.
There are other ways to solve the issue, but this is probably the simplest and least disruptive.

ReceiverEvents.receiverError
);
if (senderErrorListenerCount && senderErrorListenerCount < 1) {
this.link!.sender.on(SenderEvents.senderError, (context: EventContext) => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are options that you can pass to initLink for callbacks that are intended to be only set once - if the ones you're setting aren't there we should add them and register them there.

That should eliminate the need to do this kind of checking.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did look into that. There are complications, however.

"rhea" doesn't allow setting only the "onError" handler in the options if it is not accompanied by an "onMessage" handler.
image

Our onMessage handler("message" event) is set at core-amqp and we are trying to pass the onError handler in the options.
If we want to pass both onError and onMessage through core-amqp, we'd need to update the core-amqp's logic drastically, which would also impact event-hubs I guess.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have the lock to protect you here so you can do one-time init-only modification to the receiver.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, updated createRheaLink instead to add the listener and thus, this would happen as a one-time init-only modification.

…to harshan/sb/issue/renewlock-listener-exceeded-warning
…king for the count before adding the listener
@HarshaNalluru
Copy link
Member Author

/azp run js - servicebus - tests

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@@ -13,9 +13,12 @@

- `NamespaceProperties` interface property "messageSku" type changed from "string" to string literal type "Basic" | "Premium" | "Standard". [PR 11810](https://github.com/Azure/azure-sdk-for-js/pull/11810)

- Internal improvement - For the operations depending on `$management` link such as peek or lock renewals, the listeners for the "sender_error" and "receiver_error" events were added to the link for each new request made before the link is initialized. This has been improved such that the listeners are reused.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be worth mentioning the error message that would have popped up (for people searching).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mentioned

Copy link
Member

@richardpark-msft richardpark-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor changelog suggestion but otherwise looks good.

I'm interested if you have a way of testing this - I believe we did, on occasion, see this warning in our tests. Perhaps it's something we can pick up in our stress tests (ie, settle many messages outside of the receiver link).

@HarshaNalluru
Copy link
Member Author

Perhaps it's something we can pick up in our stress tests (ie, settle many messages outside of the receiver link).

Sounds good as a stress scenario, I'll add to the list, however I don't think there are any good ways to test other than grep-ing the output for the warning.

@HarshaNalluru HarshaNalluru merged commit 383b4b0 into Azure:master Oct 19, 2020
openapi-sdkautomation bot pushed a commit to AzureSDKAutomation/azure-sdk-for-js that referenced this pull request Dec 4, 2020
Portal revision API (Azure#11738)

* Portal revision API

* fixed spec errors

* fixed references

* fixed return code on create and added long running operation properties

* fixed create response

* fixed comments

* small fix
@HarshaNalluru HarshaNalluru deleted the harshan/sb/issue/renewlock-listener-exceeded-warning branch August 3, 2021 07:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants