Batch concurrent write checkpoint requests

[rkistner]

We've seen some cases of users calling the write-checkpoint2.json API at 500 requests/s or more in peak periods. This easily exceeds the default router handler's concurrency limit of 10 and the queue limit of 20, especially if other db load is slowing down these requests. That results in high rates of 429 errors sent to clients. Clients automatically retry, but we'd rather avoid those completely.

This optimization notes that we can optimize this significantly under load:

1. If we have many concurrent write checkpoint requests, we only need to get a single "replication head" on the source database for all of them. The only condition here is that it needs to start after the last call was received - we cannot add serve new requests from an existing in-flight replication head request.
2. On the storage database, we can similarly optimize this to do a single batched database call, instead of individual calls for each request.

These optimizations apply only if we get concurrent requests. Sequential requests behave the same as before, with no additional delay. Right now we support up to 2 in-flight batches - any new requests will be queued and form part of a new batch.

This now also places checkpoint requests on a separate fastify queue, with up to 50 requests processed concurrently, and an additional 100 requests queued. Since these requests typically have low overhead, and increased concurrency improves the efficiency of batching, this should not add significant overhead. The split is specifically to prevent the increased limits from affecting other APIs such as the relatively expensive diagnostic APIs.

Benchmarks

Test case: Request 200x concurrent write-checkpoints for individual clients. Test with various fastify concurrency settings and batching settings. Added artificial MongoDB latency affecting both the source db and storage db, resulting in a single API call taking around 1s:

db.adminCommand({
     configureFailPoint: "failCommand",
     mode: "alwaysOn",
     data: {
       failCommands: ["findAndModify"],
       blockConnection: true,
       blockTimeMS: 500
     }
   })

The first case here ran on the main branch, the rest on this branch. For all of these I increased the queue size to avoid 429 errors, for a fair comparison.

Not taken into account here:

1. Latency of individual requests
2. Source database and replication overhead from creating many individual checkpoint requests when not using batching.

Fastify concurrency	Concurrent requests	Batching mode	Duration
10	200	Main branch	22.2s
10	200	No batching	21.5s
10	200	Max 2x batches	25.4s
50	200	No batching	11.2s
50	200	Max 3x batches	5.2s
50	200	Max 2x batches	5.4s
50	200	Max 1x batch	6.2s
100	200	No batching	13.8s
100	200	Max 3x batches	2.6s
100	200	Max 2x batches	2.6s
200	200	Max 2x batches	1.6s

It is clear from the results that increasing the fastify concurrency increases the throughput. But when not using batching, this is limited by the database connection pool size. Batching removes that bottleneck, reducing both the number of database commands and database connections in use. This would be even more relevant when there is high sync load, which typically co-occurs with high write checkpoint rates.

Note that the 200x requests here is significantly higher more than the 500/s observed in production: The 500/s is the total for the instance, spread out over 20+ processes, so around 25/s per process.

AI usage

Manually planned the implementation; used Codex gpt-5.5 to implement. Reviewed with Claude Opus 4.8.

[changeset-bot]

🦋 Changeset detected

Latest commit: f08d200

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 13 packages

Name	Type
@powersync/service-module-postgres-storage	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-core-tests	Patch
@powersync/service-module-postgres	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-core	Patch
@powersync/service-schema	Patch
@powersync/service-module-convex	Patch
@powersync/service-module-mssql	Patch
@powersync/service-module-mysql	Patch
@powersync/service-image	Patch
@powersync/service-module-core	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 36d8df44c2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[stevensJourney]

The logic here looks sound to me. Happy with these changes.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f08d2008cc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".