Skip to content

fix(sync-service): drop subquery shapes on restart instead of restoring them - #4735

Merged
robacourt merged 2 commits into
mainfrom
rob/drop-subqueries-on-restart
Aug 4, 2026
Merged

fix(sync-service): drop subquery shapes on restart instead of restoring them#4735
robacourt merged 2 commits into
mainfrom
rob/drop-subqueries-on-restart

Conversation

@robacourt

@robacourt robacourt commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Correctly restoring a subquery shape's on-disk view together with its dependency materializer across a server restart is not yet implemented. Until it is, a restored subquery shape can diverge, producing incorrect results after the server comes back up.

So for now this drops every shape involved in a subquery on restart — the outer shape and all of its dependency materializers — and lets clients re-request them from scratch.

How it works

Both restore paths in the shape subsystem read the same source of truth, ShapeStatus:

  • ShapeLogCollector.restore_shapes rebuilds the routing indexes (starts first)
  • ShapeCache.wait_for_restore starts consumers (starts later)

So the subquery shapes are removed once, from that single source, before either path reads it:

  • New ShapeStatus.prune_subquery_shapes/1 deletes the transitive subquery hierarchy — [handle | shape_dependencies_handles] for every shape with a non-empty shape_dependencies (nested subqueries are covered automatically: an intermediate dependency that itself has a subquery matches the same filter and contributes its own dependencies) — from shape metadata + on-disk storage.
  • It is called at the very start of ShapeLogCollector.restore_shapes — the first restore path to read ShapeStatus, and one that runs on every :one_for_all shape-subsystem restart. By the time routing is built or consumers start, the shapes are already gone, so no restore path can reinstate them.
  • Because no consumer or routing entry exists yet at that point, pruning is a plain metadata + storage delete — no consumer to stop, no routing entry to clear, and no async routing-removal race.

The old eager subquery-consumer start in ShapeCache.wait_for_restore is removed (there is nothing to eagerly start once these shapes are dropped). The lazy restore_shape_and_dependencies/3 path (used by start_consumer_for_handle and fresh creation) is untouched.

Test Plan

  • This PR adds a new CI github action oracle_property_test_with_restarts which restarts the server every 3 batches and checks consistency is maintained after: graceful shutdown and restore and brutal shutdown and recovery.
Screenshot 2026-07-27 at 21 48 44

Notes

Supersedes the restore path added in #4666 / #4715 for the subquery-restart case.


Generated with Claude Code

@netlify

netlify Bot commented Jul 21, 2026

Copy link
Copy Markdown

Deploy Preview for electric-next ready!

Name Link
🔨 Latest commit d675fa3
🔍 Latest deploy log https://app.netlify.com/projects/electric-next/deploys/6a676eade76def0008c33f64
😎 Deploy Preview https://deploy-preview-4735--electric-next.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@codecov

codecov Bot commented Jul 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.01%. Comparing base (9175891) to head (a2b11a5).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4735      +/-   ##
==========================================
- Coverage   60.06%   60.01%   -0.06%     
==========================================
  Files         397      397              
  Lines       43766    43766              
  Branches    12587    12586       -1     
==========================================
- Hits        26290    26267      -23     
- Misses      17395    17418      +23     
  Partials       81       81              
Flag Coverage Δ
packages/agents 72.64% <ø> (ø)
packages/agents-mcp 77.70% <ø> (ø)
packages/agents-mobile 80.67% <ø> (ø)
packages/agents-runtime 83.72% <ø> (-0.05%) ⬇️
packages/agents-server 75.47% <ø> (-0.19%) ⬇️
packages/agents-server-ui 8.32% <ø> (ø)
packages/electric-ax 51.06% <ø> (ø)
packages/experimental 87.73% <ø> (ø)
packages/react-hooks 86.48% <ø> (ø)
packages/start 82.83% <ø> (ø)
packages/typescript-client 91.89% <ø> (ø)
packages/y-electric 56.05% <ø> (ø)
typescript 60.01% <ø> (-0.06%) ⬇️
unit-tests 60.01% <ø> (-0.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@robacourt
robacourt force-pushed the rob/drop-subqueries-on-restart branch from 548df00 to 45a2291 Compare July 27, 2026 15:45
@robacourt
robacourt merged commit 9f84e60 into main Aug 4, 2026
73 of 74 checks passed
@robacourt
robacourt deleted the rob/drop-subqueries-on-restart branch August 4, 2026 08:46
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

This PR has been released! 🚀

The following packages include changes from this PR:

  • @core/sync-service@1.7.10

Thanks for contributing to Electric!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants