Fix: edgeAgent creates rogue ModuleClients in case of error #5332
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…multiple clients trying to connect with same identity
nyanzebra
previously approved these changes
Aug 5, 2021
varunpuranik
reviewed
Aug 8, 2021
| // swallowing intentionally | ||
| } | ||
|
|
||
| throw; |
There was a problem hiding this comment.
Where would this get caught/handled?
There was a problem hiding this comment.
the main idea of the rethrow was to not change the existing error handling, but
- this is called from an OnClosed() event handler, and it catches and swallows the exception
- it is called by 'GetTwinFromIotHub' which catches it and either retries or just swallows the exception (retries once)
- called from the ModuleClient ctor - there the idea I suppose that the module client subscribes to events (method handler and desired properties) plus when the connection gets closed, it reopens. I think this latter logic might be broken as I see, because started offline, the method will throw an exception and I don't see how it would automatically create a new connection anytime soon (it will when some periodic operation, like twin pull requests a new connection) - but again, with this change the logic stays the same.
added a ticket for this last question: (10555951)
varunpuranik
reviewed
Aug 8, 2021
varunpuranik
approved these changes
Aug 10, 2021
vipeller
added a commit
to vipeller/iotedge
that referenced
this pull request
Aug 11, 2021
vipeller
added a commit
to vipeller/iotedge
that referenced
this pull request
Aug 11, 2021
damonbarry
pushed a commit
to damonbarry/iotedge
that referenced
this pull request
Apr 15, 2022
causing multiple ModuleClients trying to connect with same identity. When edgeAgent creates a new instance of module client, it sets up subscriptions. If that failed (e.g in case of network problem), then the newly created ModuleClient was left there without referencing it and nothing closed it afterwards. Later edgeAgent needed to create a new module client again (as the previous attempt failed). Thus at least two active ModuleClient was created with the same connection string, both trying to connect iothub. When this happened, they started to fight over the connection - iothub always disconnected the previously connected ModuleClient. Because edgeAgent pulls a twin at every connection, the connection fight mentioned above caused pulling several twins in every second.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
causing multiple ModuleClients trying to connect with same identity.
When edgeAgent creates a new instance of module client, it sets up subscriptions. If that failed (e.g in case of network problem), then the newly created ModuleClient was left there without referencing it and nothing closed it afterwards. Later edgeAgent needed to create a new module client again (as the previous attempt failed). Thus at least two active ModuleClient was created with the same connection string, both trying to connect iothub. When this happened, they started to fight over the connection - iothub always disconnected the previously connected ModuleClient.
Because edgeAgent pulls a twin at every connection, the connection fight mentioned above caused pulling several twins in every second.