Fix/redis flushall issue#13
Merged
Merged
Conversation
When Redis is flushed at runtime, stream consumer groups are deleted. xreadgroup then throws NOGROUP on every iteration, causing the consumer loop to spin indefinitely without processing any tasks. Catch NOGROUP in _try_read_stream and call setup_consumer_groups() to recreate the groups immediately. The Reconciler then republishes any tasks that were enqueued before the groups were restored. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If a task completes mid-execution while the consumer group has been deleted (e.g. Redis FLUSHALL), xack throws NOGROUP. This caused the success-path exception handler to log a false "Task failed", apply incorrect backoff, and surface an unhandled asyncio exception. After a flush the message is already gone from the PEL, so there is nothing to acknowledge. Catch NOGROUP in _ack_message and log a warning instead of propagating the error. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🐛 Consumer group loss after
FLUSHALL:FLUSHALLat runtime deletes all stream consumer groups. The Coordinator recreates streams viaXADD, butxreadgroupkept throwingNOGROUPon every iteration. CatchNOGROUPin_try_read_streamand callsetup_consumer_groups()to recover; the Reconciler republishes any tasks queued before the group was restored.🐛 False "Task failed" after flush: When a task finishes while the consumer group is gone,
xackthrowsNOGROUP. The generic handler in_execute_and_ackwas catching this, logging "Task failed", and applying backoff on a task that actually succeeded. CatchNOGROUPin_ack_messageand log a warning instead — the message is already gone from the PEL.