fix(sync-service): drop subquery shapes on restart instead of restoring them - #4735
Merged
Conversation
✅ Deploy Preview for electric-next ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4735 +/- ##
==========================================
- Coverage 60.06% 60.01% -0.06%
==========================================
Files 397 397
Lines 43766 43766
Branches 12587 12586 -1
==========================================
- Hits 26290 26267 -23
- Misses 17395 17418 +23
Partials 81 81
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
robacourt
force-pushed
the
rob/drop-subqueries-on-restart
branch
from
July 27, 2026 14:43
785891f to
d675fa3
Compare
robacourt
requested review from
KyleAMathews,
alco,
balegas,
icehaunter,
kevin-dp,
magnetised,
msfstef,
paulharter,
samwillis and
thruflo
as code owners
July 27, 2026 15:06
robacourt
force-pushed
the
rob/drop-subqueries-on-restart
branch
from
July 27, 2026 15:10
4149742 to
6edc3e2
Compare
robacourt
force-pushed
the
rob/drop-subqueries-on-restart
branch
from
July 27, 2026 15:45
548df00 to
45a2291
Compare
robacourt
force-pushed
the
rob/drop-subqueries-on-restart
branch
from
July 27, 2026 16:03
45a2291 to
a2b11a5
Compare
This was referenced Jul 28, 2026
alco
approved these changes
Aug 4, 2026
Contributor
|
This PR has been released! 🚀 The following packages include changes from this PR:
Thanks for contributing to Electric! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Correctly restoring a subquery shape's on-disk view together with its dependency materializer across a server restart is not yet implemented. Until it is, a restored subquery shape can diverge, producing incorrect results after the server comes back up.
So for now this drops every shape involved in a subquery on restart — the outer shape and all of its dependency materializers — and lets clients re-request them from scratch.
How it works
Both restore paths in the shape subsystem read the same source of truth,
ShapeStatus:ShapeLogCollector.restore_shapesrebuilds the routing indexes (starts first)ShapeCache.wait_for_restorestarts consumers (starts later)So the subquery shapes are removed once, from that single source, before either path reads it:
ShapeStatus.prune_subquery_shapes/1deletes the transitive subquery hierarchy —[handle | shape_dependencies_handles]for every shape with a non-emptyshape_dependencies(nested subqueries are covered automatically: an intermediate dependency that itself has a subquery matches the same filter and contributes its own dependencies) — from shape metadata + on-disk storage.ShapeLogCollector.restore_shapes— the first restore path to readShapeStatus, and one that runs on every:one_for_allshape-subsystem restart. By the time routing is built or consumers start, the shapes are already gone, so no restore path can reinstate them.The old eager subquery-consumer start in
ShapeCache.wait_for_restoreis removed (there is nothing to eagerly start once these shapes are dropped). The lazyrestore_shape_and_dependencies/3path (used bystart_consumer_for_handleand fresh creation) is untouched.Test Plan
oracle_property_test_with_restartswhich restarts the server every 3 batches and checks consistency is maintained after: graceful shutdown and restore and brutal shutdown and recovery.Notes
Supersedes the restore path added in #4666 / #4715 for the subquery-restart case.
Generated with Claude Code