-
Notifications
You must be signed in to change notification settings - Fork 11
fix(worker): Fix infinite retry loop in bundle analysis processor #631
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
- Change retry_num from self.attempts to self.request.retries to ensure consistent retry counting - Remove max_retries parameter from LockManager.locked() call (no longer accepted) - Simplify exception handler to use centralized self.retry() which properly checks max retries This fixes the looping retry issue seen in task logs where tasks would retry indefinitely due to inconsistent retry counting between self.attempts and self.request.retries.
- Introduced a check for max retries in the retry logic to avoid infinite loops. - Updated exception handling to raise MaxRetriesExceededError when max retries are exceeded. - Enhanced unit tests to verify correct behavior when max retries are reached and ensure proper retry logic.
…lts on max retries exceeded - Removed MaxRetriesExceededError raising and instead return previous_result when max retries are exceeded. - Enhanced logging to capture relevant details when max retries are exceeded. - Updated unit tests to verify that previous results are returned, preserving chain behavior and preventing infinite retry loops.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #631 +/- ##
==========================================
+ Coverage 93.89% 93.90% +0.01%
==========================================
Files 1286 1286
Lines 46807 46802 -5
Branches 1517 1517
==========================================
Hits 43951 43951
+ Misses 2547 2542 -5
Partials 309 309
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
- Introduced integration tests for the complete bundle analysis process, verifying the execution of the processor and notify tasks. - Ensured correct handling of database state and previous results during task execution. - Added tests to check behavior when maximum retries are exceeded, confirming that the chain does not crash and returns previous results.
Description
Fixes the infinite retry loop bug in
BundleAnalysisProcessorTaskthat was causing tasks to retry indefinitely. This is a minimal atomic change extracted from PR #608.Root Cause:
The task was using
self.attemptsinstead ofself.request.retrieswhen callingLockManager.locked(). Theself.attemptsproperty can be inconsistent withself.request.retriesdue to visibility timeout re-deliveries, causing the max retries check to fail and allowing infinite retries.Changes:
retry_num=self.attemptstoretry_num=self.request.retriesinLockManager.locked()callmax_retriesparameter (no longer accepted byLockManager)self.retry()method which properly checks max retriesImpact:
self.request.retriesBaseCodecovTask.retry()Related:
Note
Resolves runaway retries in bundle analysis processing and hardens retry behavior.
bundle_analysis_processor.py, passretry_num=self.request.retriestoLockManager.locked()and handleLockRetryby checkingself.request.retries >= max_retries; on exceed, returnprevious_result, otherwise callself.retry()with backoffprevious_resultwhen lock retries are exhausted; remove legacysafe_retryusage in favor of centralizedself.retry()test_bundle_analysis_e2e.py)Retrybelow max, returningprevious_resultat/above max) and error-state persistence on processing failures (test_bundle_analysis_processor_task.py)Written by Cursor Bugbot for commit 78a77b5. This will update automatically on new commits. Configure here.