fix(sync): clear exhausted jobs blocked by durable read failures - #2712
Merged
Conversation
Stop the self-heal sweep from re-logging the same exhausted calendarListSync row every cycle when the connection already carries lastReadFailureAt. Co-authored-by: Tyler Dane <tyler-dane@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stops the sync self-heal sweep from re-logging the same exhausted
calendarListSyncjob every ~10 minutes when the connection already carries a durable provider refusal.PostHog cluster (7d):
6a775383…/ connection6a653974…ProviderCalendarError(discoveryFailed): … The user must be signed up for Google Calendar.Root cause of remaining noise: #2696 drops durable discovery failures going forward, but a pre-deploy exhausted failed row kept its coalescing key and made self-heal
logger.erroron every sweep. Events resources on the same connection already hadlastReadFailureAtfrom incrementalPull drops.Fix: when listing exhausted jobs, auto-clear any whose connection already has a durable
lastReadFailureAtmarker (same effect asmanage-failed-jobs clear), log once at warn, and only error for remaining exhausted rows that still need an operator.Simplicity
One extra resource lookup on the existing exhausted sample path — no new job states, no new failure classes. Reuses
jobs.remove/listByConnection/ the health marker already stamped by readFailed and discoveryFailed drops.Automated validation
bun test:sync -- packages/sync/src/domain/failed-job-requeue.service.db.test.ts→ 7 passbun run lint→ no new issues in touched files (pre-existing warnings only)Independent review
Diff-first pass: auto-clear is gated on an existing durable marker (not on job kind alone), so genuine exhausted transient failures still alert. Cleared jobs free the coalescing key so rediscovery can enqueue a fresh calendarListSync that #2696 will drop. No further highs found.
Test plan
lastReadFailureAton the connection is cleared and removed from error countrequeuedCountcases still passRemaining risk / ops
6a775383…and silence019fdc65-…. Resolve both PostHog issues once quiet.6a653974…still cannot list Google calendars — enable Google Calendar or disconnect in Compass so rediscovery can succeed.