Skip to content

fix: stop idempotent job insert storms - #3059

Merged
chubes4 merged 1 commit into
mainfrom
fix-3058-idempotent-job-storm
Aug 6, 2026
Merged

fix: stop idempotent job insert storms#3059
chubes4 merged 1 commit into
mainfrom
fix-3058-idempotent-job-storm

Conversation

@chubes4

@chubes4 chubes4 commented Aug 6, 2026

Copy link
Copy Markdown
Member

Summary

  • bound job labels to the owned varchar(255) schema so oversized batch titles remain insertable
  • recover duplicate races and lock/deadlock contention with bounded attempts while preserving the original write error
  • rate-limit identical persistent error logs, emit an unthrottled metrics hook, and avoid exposing raw idempotency keys
  • add focused regression coverage for malformed labels, race winners, transient contention, persistent failures, and log amplification

Production mechanism

Read-only Events diagnostics found one durable worklist item retried continuously with a 295-character title. Calling WordPress's wpdb::process_fields() against the live jobs schema returned false specifically for that label before SQL execution. The recovery lookup then reset wpdb->last_error, so every failure was logged with an empty database error and the worklist returned the item to ready for another attempt.

The existing unique idx_idempotency_key index is present and healthy; Action Scheduler retention pressure remains separately tracked in #2792 and #2793.

Verification

  • php tests/jobs-idempotent-insert-recovery-smoke.php (21 assertions)
  • Homeboy changed-file lint: PHPCS passed, PHPStan level 7 passed
  • php tests/prefix-policy-audit.php (11 passed)
  • Homeboy architecture audit: no introduced findings
  • git diff --check

Homeboy's selected WP smoke wrapper could not launch because the host WP Codebox cache is missing packages/cli/dist/index.js; the selected standalone test passes directly.

Closes #3058

@chubes4
chubes4 merged commit f034056 into main Aug 6, 2026
9 of 13 checks passed
@chubes4
chubes4 deleted the fix-3058-idempotent-job-storm branch August 6, 2026 16:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: diagnose and stop Events idempotent job insert failure storm

1 participant