Skip to content

Added indexed availability for deferred outbox work - #30691

Closed
jonatansberg wants to merge 1 commit into
jonatan-ber-3936-bounded-mailgun-fetchingfrom
jonatan-ber-3936-durable-suppression-cleanup
Closed

Added indexed availability for deferred outbox work#30691
jonatansberg wants to merge 1 commit into
jonatan-ber-3936-bounded-mailgun-fetchingfrom
jonatan-ber-3936-durable-suppression-cleanup

Conversation

@jonatansberg

Copy link
Copy Markdown
Member

ref https://linear.app/ghost/issue/BER-3936/raise-mailgun-event-fetch-throughput-concurrent-paging-logs-api

Remote suppression cleanup needs durable retry scheduling so Mailgun delays do not hold up event ingestion. This is the schema foundation: it adds a nullable outbox.available_at timestamp and an index for selecting due work within an event type and status.

outbox
  event_type + status + available_at + id → bounded selection of due work
  existing payload / status / created-at index → retained

Existing rows receive NULL and their payloads are preserved. The migration uses idempotent nontransactional helpers and removes the dependent index before dropping the column on rollback. It adds no producer or worker; the following PR will commit local unsubscribe/complaint safety state with cleanup intents and process them with retries, lease recovery and shutdown draining. Keeping this foundation separate lets us review the database change on its own.

Stacked on #30690 as the next part of the ingestion performance stack. The schema has no logical dependency on prefetch or counter modes. Adding the index can still take time on an existing outbox; no claim is made from the small test database about production DDL duration.

Validation: schema integrity, Core/test types and focused lint pass. Manual MySQL CLI initialization, rollback to 6.63 and forward migration to 6.64 preserved an existing payload and verified the nullable field and index order; the temporary database was removed. SQLite up/up/down/down/up and all four migration integration tests pass. Five independent review passes found no high-confidence issues.

The full unit run is not green: unrelated untracked suites fail and the runner reports source-map parsing errors. A separate tracked-only run reported 8,687 passing tests but still exited with 64 source-map errors. The migration-specific checks above pass. The earlier full repository check also stops on 452 unrelated formatting files.

ref https://linear.app/ghost/issue/BER-3936/raise-mailgun-event-fetch-throughput-concurrent-paging-logs-api

Let durable suppression cleanup select due retries without scanning deferred work, while preserving existing outbox rows and indexes.
@coderabbitai

coderabbitai Bot commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the migration [pull request] Includes migration for review label Sep 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

It looks like this PR contains a migration 👀
Here's the checklist for reviewing migrations:

General requirements

  • ⚠️ Tested performance on staging database servers, as performance on local machines is not comparable to a production environment
  • Satisfies idempotency requirement (both up() and down())
  • Does not reference models
  • Filename is in the correct format (and correctly ordered)
  • Targets the next minor version
  • All code paths have appropriate log messages
  • Uses the correct utils
  • Contains a minimal changeset
  • Does not mix DDL/DML operations

Schema changes

  • Both schema change and related migration have been implemented
  • For index changes: has been performance tested for large tables
  • For new tables/columns: fields use the appropriate predefined field lengths
  • For new tables/columns: field names follow the appropriate conventions
  • Does not drop a non-alpha table outside of a major version

Data changes

  • Mass updates/inserts are batched appropriately
  • Does not loop over large tables/datasets
  • Defends against missing or invalid data
  • For settings updates: follows the appropriate guidelines

@nx-cloud

nx-cloud Bot commented Sep 10, 2026

Copy link
Copy Markdown

🤖 Nx Cloud AI Fix

Ensure the fix-ci command is configured to always run in your CI pipeline to get automatic fixes in future runs. For more information, please see https://nx.dev/ci/features/self-healing-ci


View your CI Pipeline Execution ↗ for commit 78ed019

Command Status Duration Result
nx run ghost:test:ci:integration ✅ Succeeded 4m 43s View ↗
nx run ghost:test:integration ✅ Succeeded 3m 42s View ↗
nx run ghost:test:legacy ✅ Succeeded 3m 19s View ↗
nx run ghost:test:ci:e2e ✅ Succeeded 4m 16s View ↗
nx run ghost:test:e2e ✅ Succeeded 2m 59s View ↗
nx run-many -t test:types -p ghost ✅ Succeeded 5s View ↗
nx run-many --target=build --projects=tag:publi... ✅ Succeeded <1s View ↗
nx run-many -t test:unit -p ghost ✅ Succeeded 36s View ↗
Additional runs (4) ✅ Succeeded ... View ↗

💡 Verify your cache is correct by running tasks in a sandbox. Read docs ↗


☁️ Nx Cloud last updated this comment at 2026-09-10 17:46:08 UTC

@codecov

codecov Bot commented Sep 10, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 67.72%. Comparing base (4d53c90) to head (78ed019).

Additional details and impacted files
@@                              Coverage Diff                              @@
##           jonatan-ber-3936-bounded-mailgun-fetching   #30691      +/-   ##
=============================================================================
+ Coverage                                      67.70%   67.72%   +0.01%     
=============================================================================
  Files                                           1674     1674              
  Lines                                          60586    60586              
  Branches                                       10499    10499              
=============================================================================
+ Hits                                           41017    41029      +12     
+ Misses                                         17230    17222       -8     
+ Partials                                        2339     2335       -4     
Flag Coverage Δ
e2e-tests 70.46% <ø> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

migration [pull request] Includes migration for review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant