Skip to content

fix(sync): clear exhausted jobs blocked by durable read failures - #2712

Merged
cursor[bot] merged 1 commit into
mainfrom
cursor/posthog-error-investigation-7cc1
Aug 9, 2026
Merged

fix(sync): clear exhausted jobs blocked by durable read failures#2712
cursor[bot] merged 1 commit into
mainfrom
cursor/posthog-error-investigation-7cc1

Conversation

@cursor

@cursor cursor Bot commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Stops the sync self-heal sweep from re-logging the same exhausted calendarListSync job every ~10 minutes when the connection already carries a durable provider refusal.

PostHog cluster (7d):

  • Sync self-heal sweep exhausted budget — 118 occ, still firing through 2026-08-09 12:52Z for job 6a775383… / connection 6a653974…
  • Companion Sync job engine failed — calendarListSync half quiet after attempt 20 at 02:54Z; root cause ProviderCalendarError(discoveryFailed): … The user must be signed up for Google Calendar.

Root cause of remaining noise: #2696 drops durable discovery failures going forward, but a pre-deploy exhausted failed row kept its coalescing key and made self-heal logger.error on every sweep. Events resources on the same connection already had lastReadFailureAt from incrementalPull drops.

Fix: when listing exhausted jobs, auto-clear any whose connection already has a durable lastReadFailureAt marker (same effect as manage-failed-jobs clear), log once at warn, and only error for remaining exhausted rows that still need an operator.

Simplicity

One extra resource lookup on the existing exhausted sample path — no new job states, no new failure classes. Reuses jobs.remove / listByConnection / the health marker already stamped by readFailed and discoveryFailed drops.

Automated validation

  • bun test:sync -- packages/sync/src/domain/failed-job-requeue.service.db.test.ts → 7 pass
  • bun run lint → no new issues in touched files (pre-existing warnings only)

Independent review

Diff-first pass: auto-clear is gated on an existing durable marker (not on job kind alone), so genuine exhausted transient failures still alert. Cleared jobs free the coalescing key so rediscovery can enqueue a fresh calendarListSync that #2696 will drop. No further highs found.

Test plan

  • Exhausted job with lastReadFailureAt on the connection is cleared and removed from error count
  • Exhausted job without a durable marker still reports as needing operator attention
  • Existing requeue / cooldown / missing-requeuedCount cases still pass

Remaining risk / ops

  • After deploy, the next self-heal sweep should clear stuck job 6a775383… and silence 019fdc65-…. Resolve both PostHog issues once quiet.
  • Account 6a653974… still cannot list Google calendars — enable Google Calendar or disconnect in Compass so rediscovery can succeed.
Open in Web View Automation 

Stop the self-heal sweep from re-logging the same exhausted calendarListSync
row every cycle when the connection already carries lastReadFailureAt.

Co-authored-by: Tyler Dane <tyler-dane@users.noreply.github.com>
@cursor
cursor Bot marked this pull request as ready for review August 9, 2026 21:49
@cursor
cursor Bot merged commit 24f89f5 into main Aug 9, 2026
20 checks passed
@cursor
cursor Bot deleted the cursor/posthog-error-investigation-7cc1 branch August 9, 2026 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant