Improve compacting & checksum pre-calculations

[rkistner]

Checksum pre-calculations

We occasionally see this error in the checksum pre-calculation step after the initial snapshot completed: [Processing Sync Rules] [PSYNC_S2403] Query timed out while reading checksums. The pre-calculation roughly worked as follows:

1. Fetch batch of 5000 buckets that need a pre-calculation.
2. Split into smaller batches of 200 at a time to do the checksum queries.
3. Calculate checksums over a max of 50 000 operations at a time.
4. Once all 5000 buckets have the checksums computed, persist it.

In certain scenarios under high load, especially if the documents themselves are large, a checksum query can timeout. And if any of the smaller batches time out, the entire larger batch is restarted.

This improves the process in two ways:

1. Start by only fetching batches of 200 at a time.
2. Check the estimated size of the buckets. If the buckets are large, limit the calculation to a smaller batch to only contain around 50 000 operations at a time.

This is not a guaranteed fix, but should reduce the timeouts.

Compacting buckets

#375 introduced a change to only compact buckets with changes. However, the change in logic had another side-effect: If buckets are modified while the compact process is running, they could be repeatedly re-compacted, resulting in the compact process never finishing.

This changes the logic to:

1. Only compact buckets with a minimum of 10 new operations since the last compact.
2. Avoid re-compacting the 100 most recently processed buckets.

This logic may also need further tweaks over time as we get more real-world usage data.

[changeset-bot]

🦋 Changeset detected

Latest commit: add62d4

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 11 packages

Name	Type
@powersync/service-module-mongodb-storage	Patch
@powersync/service-core-tests	Patch
@powersync/service-core	Patch
@powersync/service-schema	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres	Patch
@powersync/service-image	Patch
@powersync/service-module-postgres-storage	Patch
@powersync/service-module-core	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[stevensJourney]

The improvements here are very nice! The implementation here looks good to me.