Skip to content

Conversation

@troyciesco
Copy link
Contributor

@troyciesco troyciesco commented Nov 25, 2025

ref https://linear.app/ghost/issue/NY-801

  • Randomizes when the member welcome email job runs. Still every 5 minutes, but starts on a different minute/second
  • This will help avoid a "thundering herd" problem on app servers running multiple Ghost instances, and is the same pattern we use for other jobs like email analytics

below we can see what it looks like when the jobs get scheduled randomly. 4th row is site A, 3rd row is site B. i happened to get the 0 minute for both on the first try lol, so i restarted each site. Then 2nd row is site A, 1st row is site B, and you can see each time they're all getting scheduled differently
Screenshot 2025-12-02 at 4 02 12 PM

Then in this image we can see site A and B running every 5 minutes, but at different times: 01,06,11 and 02,07,12 minutes respectively, as expected.
Screenshot 2025-12-02 at 4 16 29 PM


Note

Randomizes the non-instant member welcome email job to run every 5 minutes at a randomized second and minute offset, retaining the instant test schedule.

  • Backend · Jobs
    • Member welcome emails (core/server/services/member-welcome-emails/jobs/index.js):
      • Replace fixed cron with randomized schedule for non-instant mode: compute s = Math.floor(Math.random() * 60) and m = Math.floor(Math.random() * 5), then use ${s} ${m}/5 * * * *.
      • Keep instant test schedule as */3 * * * * *.

Written by Cursor Bugbot for commit c246614. This will update automatically on new commits. Configure here.

@coderabbitai
Copy link
Contributor

coderabbitai bot commented Nov 25, 2025

Walkthrough

The change updates cron scheduling in the member welcome emails job. For non-instant sends it now generates a randomized schedule by choosing seconds (0–59) and a random 5-minute offset (0–4) and constructing the cron expression as ${s} ${m}/5 * * * *. Instant sends continue using the previous */3 * * * * * schedule. No other control flow, error handling, or exported interfaces were modified; the change is limited to cron expression construction and accompanying comments.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

  • Verify the randomized s and m values produce valid cron expressions and are correctly interpolated.
  • Confirm ${m}/5 yields the intended cadence across instances (minute-offset behavior).
  • Check whether tests need updates or additions for non-deterministic scheduling.
  • Review the single modified file (ghost/core/core/server/services/member-welcome-emails/jobs/index.js) and the explanatory comments for clarity.

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately describes the main change: randomizing the member welcome email job's run schedule.
Description check ✅ Passed The description is directly related to the changeset, explaining the randomization rationale, implementation details, and including verification screenshots.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch spike_no-thundering-herd-251125

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 3e44f48 and c246614.

📒 Files selected for processing (1)
  • ghost/core/core/server/services/member-welcome-emails/jobs/index.js (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • ghost/core/core/server/services/member-welcome-emails/jobs/index.js
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (8)
  • GitHub Check: Ghost-CLI tests
  • GitHub Check: Acceptance tests (Node 22.18.0, mysql8)
  • GitHub Check: Legacy tests (Node 22.18.0, mysql8)
  • GitHub Check: Legacy tests (Node 22.18.0, sqlite3)
  • GitHub Check: Unit tests (Node 22.18.0)
  • GitHub Check: Lint
  • GitHub Check: Cursor Bugbot
  • GitHub Check: Build & Push

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@troyciesco troyciesco force-pushed the spike_smaller-welcome-email-job-251124 branch 3 times, most recently from 9ef0f1a to 9be656b Compare December 1, 2025 17:29
@troyciesco troyciesco force-pushed the spike_smaller-welcome-email-job-251124 branch 3 times, most recently from 10adf64 to 90cab1d Compare December 2, 2025 20:13
@troyciesco troyciesco force-pushed the spike_no-thundering-herd-251125 branch from d60e62e to cb6b272 Compare December 2, 2025 20:14
@troyciesco troyciesco force-pushed the spike_smaller-welcome-email-job-251124 branch from 90cab1d to 53250ba Compare December 2, 2025 22:07
@troyciesco troyciesco force-pushed the spike_no-thundering-herd-251125 branch from cb6b272 to c8d4e08 Compare December 2, 2025 22:08
@github-actions
Copy link
Contributor

github-actions bot commented Dec 2, 2025

E2E Tests Failed

To view the Playwright test report locally, run:

REPORT_DIR=$(mktemp -d) && gh run download 19875111154 -n playwright-report -D "$REPORT_DIR" && npx playwright show-report "$REPORT_DIR"

troyciesco added a commit that referenced this pull request Dec 2, 2025
… outbox (#25521)

ref https://linear.app/ghost/issue/NY-805

- this makes it so the welcome emails job works similar to the email
analytics job, in that it just emits a `StartMemberWelcomeEmailJobEvent`
- The `MemberWelcomeEmailsService` subscribes to that event and then
processes the outbox
- This had a positive improvement on CPU utilization, most
visible/useful for app servers running many Ghost instances;
#25534 will further improve it by
randomizing when the job starts for each instance
Base automatically changed from spike_smaller-welcome-email-job-251124 to main December 2, 2025 23:14
@troyciesco troyciesco force-pushed the spike_no-thundering-herd-251125 branch from c8d4e08 to 3e44f48 Compare December 2, 2025 23:17
@troyciesco troyciesco marked this pull request as ready for review December 2, 2025 23:21
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
ghost/core/core/server/services/member-welcome-emails/jobs/index.js (1)

20-26: Randomized cron schedule achieves desired staggering; consider minor scope/comment tweaks

The randomization logic looks sound: s in [0,59] and m in [0,4] produces 5-minute schedules with per-instance offsets while preserving the 6-field cron format. This should effectively reduce synchronized spikes between instances.

Two small, optional nits:

  • You could move the s/m calculations into the non‑instant branch to avoid computing them when testEmailSendInstantly is true:
-        // use a random seconds value to avoid spikes to the database on the minute
-        const s = Math.floor(Math.random() * 60); // 0-59
-        // run every 5 minutes, on 1,6,11..., 2,7,12..., 3,8,13..., etc
-        const m = Math.floor(Math.random() * 5); // 0-4
-
-        const cronSchedule = testEmailSendInstantly ? '*/3 * * * * *' : `${s} ${m}/5 * * * *`;
+        let cronSchedule;
+        if (testEmailSendInstantly) {
+            cronSchedule = '*/3 * * * * *';
+        } else {
+            // use a random seconds value to avoid spikes to the database on the minute
+            const s = Math.floor(Math.random() * 60); // 0-59
+            // run every 5 minutes with a randomized minute offset (0–4) per instance
+            const m = Math.floor(Math.random() * 5); // 0-4
+            cronSchedule = `${s} ${m}/5 * * * *`;
+        }
  • The comment currently examples 1,6,11..., 2,7,12..., 3,8,13...; since m can also be 0 or 4, you might generalize the wording (e.g. “random 0–4 minute offset, so instances run at different X, X+5, X+10, ... minutes”) to avoid confusion for future readers.
📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 0e08ace and 3e44f48.

📒 Files selected for processing (1)
  • ghost/core/core/server/services/member-welcome-emails/jobs/index.js (1 hunks)
🧰 Additional context used
🧠 Learnings (3)
📓 Common learnings
Learnt from: troyciesco
Repo: TryGhost/Ghost PR: 25288
File: ghost/core/core/server/services/member-welcome-emails/jobs/lib/process-entries.js:46-64
Timestamp: 2025-11-10T23:10:17.470Z
Learning: In ghost/core/core/server/services/member-welcome-emails/jobs/lib/process-entries.js and the outbox processing flow, entries are marked as PROCESSING before being processed. If a failure occurs after email send but before deletion, the entry remains stuck in PROCESSING state (not reprocessed). This intentional design prevents duplicate emails. Handling stuck PROCESSING entries is planned for a separate PR.
📚 Learning: 2025-10-30T17:13:26.190Z
Learnt from: sam-lord
Repo: TryGhost/Ghost PR: 25303
File: ghost/core/core/server/services/email-service/BatchSendingService.js:19-19
Timestamp: 2025-10-30T17:13:26.190Z
Learning: In ghost/core/core/server/services/email-service/BatchSendingService.js and similar files in the Ghost codebase, prefer using `{...options}` spread syntax without explicit guards like `...(options || {})` when spreading potentially undefined objects, as the maintainer prefers cleaner syntax over defensive patterns when the behavior is safe.

Applied to files:

  • ghost/core/core/server/services/member-welcome-emails/jobs/index.js
📚 Learning: 2025-11-10T23:10:17.470Z
Learnt from: troyciesco
Repo: TryGhost/Ghost PR: 25288
File: ghost/core/core/server/services/member-welcome-emails/jobs/lib/process-entries.js:46-64
Timestamp: 2025-11-10T23:10:17.470Z
Learning: In ghost/core/core/server/services/member-welcome-emails/jobs/lib/process-entries.js and the outbox processing flow, entries are marked as PROCESSING before being processed. If a failure occurs after email send but before deletion, the entry remains stuck in PROCESSING state (not reprocessed). This intentional design prevents duplicate emails. Handling stuck PROCESSING entries is planned for a separate PR.

Applied to files:

  • ghost/core/core/server/services/member-welcome-emails/jobs/index.js
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (7)
  • GitHub Check: E2E Tests (Ember 1/2)
  • GitHub Check: E2E Tests (Ember 2/2)
  • GitHub Check: [Optional] E2E Tests (React 2/2)
  • GitHub Check: [Optional] E2E Tests (React 1/2)
  • GitHub Check: Ghost-CLI tests
  • GitHub Check: Acceptance tests (Node 22.18.0, mysql8)
  • GitHub Check: Unit tests (Node 22.18.0)

@troyciesco troyciesco force-pushed the spike_no-thundering-herd-251125 branch from 3e44f48 to c246614 Compare December 3, 2025 18:59
@troyciesco troyciesco merged commit fbaba51 into main Dec 3, 2025
38 checks passed
@troyciesco troyciesco deleted the spike_no-thundering-herd-251125 branch December 3, 2025 19:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants