Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Redis broker] message_refc dips into negative numbers on ConnectionError #262

Closed

Conversation

benekastah
Copy link
Contributor

@benekastah benekastah commented Jan 9, 2020

This can cause prefetching too many messages. The reason is that when we
try to call consumer.ack or consumer.nack, and a ConnectionError
happens, we will retry. But each time we retry, we decrement message_refc
again. Solved this by using a set to track queued messages, so calling ack
or nack on the same message doesn't mess up our accounting.

benekastah added 3 commits Jan 9, 2020
This can cause prefetching too many messages. The reason is that when we
try to call `consumer.ack` or `consumer.nack`, and a `ConnectionError`
happens, we will retry. But each time we retry, we decrement `message_refc`
again. Solved this by using a set to track queued messages, so calling `ack`
or `nack` on the same message doesn't mess up our accounting.
self.misses = 0

@property
def message_refc(self):
Copy link
Contributor Author

@benekastah benekastah Jan 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This property is so I can use the same test code with this version of the file and with the previous version of the file.

@benekastah benekastah changed the title message_refc dips into negative numbers on ConnectionError [Redis broker] message_refc dips into negative numbers on ConnectionError Jan 9, 2020
return MessageProxy(message)
except IndexError:
# If there are fewer messages currently being
# processed than we're allowed to prefetch,
# prefetch up to that number of messages.
messages = []
num_messages = len(self.queued_message_ids)
if self.message_refc < self.prefetch:
Copy link
Owner

@Bogdanp Bogdanp Jan 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like this is a bug since message_refc has been removed. Did you mean to use num_messages here?

@Bogdanp
Copy link
Owner

@Bogdanp Bogdanp commented Jan 20, 2020

I made a couple changes here and merged this into master with rebase. Thanks for the fix!

@Bogdanp Bogdanp closed this Jan 20, 2020
@Bogdanp Bogdanp added this to the v1.8.0 milestone Jan 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants