Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix a problem where the RabbitMQPublisher could deadlock #43

Merged
merged 4 commits into from Jan 20, 2022

Conversation

t-persson
Copy link
Collaborator

Applicable Issues

Description of the Change

There have been multiple problems with this code. One problem is that it would send events from two different threads. The main thread and the thread that resends nacked events. This could cause the RabbitMQ server to disconnect the client.
Secondly since the send_event method would block and wait for the client to reconnect in case of disconnects, the resend_nacked_events method could block the IOLoop which would prevent the client from reconnecting, thus causing a deadlock

Alternate Designs

I thought about having a queue of events instead which send_event and _resend_nacked_events would populate and having a separate thread go through this queue and send events.
This was a part of the previous design of this library and it had multiple problems due to the complexity that threading adds when using pika.

Benefits

Deadlocks are dangerous and we don't want to have to reconnect because we behave badly.

Possible Drawbacks

This adds more complexity and overhead.

Sign-off

Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or

(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or

(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.

(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

Signed-off-by: Tobias Persson tobias.persson@axis.com

There have been multiple problems with this code.
One problem is that it would send events from two different
threads. The main thread and the thread that resends nacked
events. This could cause the RabbitMQ server to disconnect
the client.
Secondly since the send_event method would block and wait
for the client to reconnect in case of disconnects, the
resend_nacked_events method could block the IOLoop which
would prevent the client from reconnecting, thus causing
a deadlock
@t-persson t-persson requested a review from a team as a code owner January 17, 2022 07:29
@t-persson t-persson requested review from fredjn and removed request for a team January 17, 2022 07:29
Copy link
Collaborator Author

@t-persson t-persson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Must protect class attributes within send_event as well.

Added a lot of debug information as well which will be active
until we've verified this PR
@t-persson
Copy link
Collaborator Author

Copyright notice needs an update of the year

@t-persson t-persson merged commit b4fe63e into eiffel-community:main Jan 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants