Skip to content

DRAFT: simple subqueries with DNF#4051

Draft
robacourt wants to merge 59 commits intomainfrom
rob/simple-subqueries-with-dnf
Draft

DRAFT: simple subqueries with DNF#4051
robacourt wants to merge 59 commits intomainfrom
rob/simple-subqueries-with-dnf

Conversation

@robacourt
Copy link
Copy Markdown
Contributor

No description provided.

@codecov
Copy link
Copy Markdown

codecov bot commented Mar 24, 2026

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
2952 1 2951 1
View the top 3 failed test(s) by shortest run time
Elixir.Electric.ShapeCacheTest::test after restart restarted subquery shape reseeds the subquery index after restart
Stack Traces | 0.0681s run time
26) test after restart restarted subquery shape reseeds the subquery index after restart (Electric.ShapeCacheTest)
     test/electric/shape_cache_test.exs:1172
     Expected truthy, got false
     code: assert SubqueryIndex.has_positions?(index_after, shape_handle)
     arguments:

         # 1
         #Reference<0.545337320.3064856578.71777>

         # 2
         "50617496-1774543966838479"

     stacktrace:
       test/electric/shape_cache_test.exs:1194: (test)
Elixir.Electric.Plug.RouterTest::test /v1/shapes - subqueries move-in into move-out into move-in of the same parent collapses queued oscillations
Stack Traces | 1.17s run time
67) test /v1/shapes - subqueries move-in into move-out into move-in of the same parent collapses queued oscillations (Electric.Plug.RouterTest)
     .../electric/plug/router_test.exs:3002
     match (=) failed
     The following variables were pinned:
       tag2 = "6bebeb5229cafd582613af0e41ae7ce5"
       tag3 = "d5fb9f1b896ece086a1a1debb5f93991"
     code:  assert {_req, 200,
             [
               %{"headers" => %{"event" => "move-in"}},
               %{
                 "headers" => %{"operation" => "insert", "is_move_in" => true, "tags" => [^tag2]},
                 "value" => %{"id" => "2", "parent_id" => "2", "value" => "20"}
               },
               %{"headers" => %{"control" => "snapshot-end"}},
               %{"headers" => %{"event" => "move-out", "patterns" => [%{"pos" => 0, "value" => ^tag2}]}},
               %{"headers" => %{"event" => "move-in"}},
               %{
                 "headers" => %{"operation" => "insert", "is_move_in" => true, "tags" => [^tag3]},
                 "value" => %{"id" => "3", "parent_id" => "3", "value" => "30"}
               },
               %{"headers" => %{"control" => "snapshot-end"}},
               up_to_date_ctl()
             ]} = shape_req(req, ctx.opts)
     left:  {_req, 200,
             [
               %{"headers" => %{"event" => "move-in"}},
               %{
                 "headers" => %{"operation" => "insert", "is_move_in" => true, "tags" => [^tag2]},
                 "value" => %{"id" => "2", "parent_id" => "2", "value" => "20"}
               },
               %{"headers" => %{"control" => "snapshot-end"}},
               %{"headers" => %{"event" => "move-out", "patterns" => [%{"pos" => 0, "value" => ^tag2}]}},
               %{"headers" => %{"event" => "move-in"}},
               %{
                 "headers" => %{"operation" => "insert", "is_move_in" => true, "tags" => [^tag3]},
                 "value" => %{"id" => "3", "parent_id" => "3", "value" => "30"}
               },
               %{"headers" => %{"control" => "snapshot-end"}},
               up_to_date_ctl()
             ]}
     right: {%{handle: "99950534-1774543997008362", offset: "1_3", table: "child", where: "parent_id in (SELECT id FROM parent WHERE value = 1)", live: false}, 200, [%{"headers" => %{"event" => "move-in", "patterns" => [%{"pos" => 0, "value" => "6bebeb5229cafd582613af0e41ae7ce5"}]}}, %{"headers" => %{"active_conditions" => [true], "is_move_in" => true, "operation" => "insert", "relation" => ["public", "child"], "tags" => ["6bebeb5229cafd582613af0e41ae7ce5"]}, "key" => "\"public\".\"child\"/\"2\"", "value" => %{"id" => "2", "parent_id" => "2", "value" => "20"}}, %{"headers" => %{"control" => "snapshot-end"}}, %{"headers" => %{"event" => "move-out", "patterns" => [%{"pos" => 0, "value" => "6bebeb5229cafd582613af0e41ae7ce5"}]}}, %{"headers" => %{"control" => "up-to-date", "global_last_seen_lsn" => "402594224"}}]}
     stacktrace:
       .../electric/plug/router_test.exs:3036: (test)
Elixir.Electric.Plug.RouterTest::test /v1/shapes - subqueries allows 3 level subquery in where clauses
Stack Traces | 4.17s run time
1) test /v1/shapes - subqueries allows 3 level subquery in where clauses (Electric.Plug.RouterTest)
     .../electric/plug/router_test.exs:2486
     match (=) failed
     code:  assert {req, 200,
             [
               %{"headers" => %{"event" => "move-in"}},
               %{
                 "value" => %{"id" => "2", "value" => "20"},
                 "headers" => %{"operation" => "insert", "is_move_in" => true, "tags" => [tag]}
               },
               %{"headers" => %{"control" => "snapshot-end"}},
               up_to_date_ctl()
             ]} = Task.await(task)
     left:  {req, 200,
             [
               %{"headers" => %{"event" => "move-in"}},
               %{
                 "value" => %{"id" => "2", "value" => "20"},
                 "headers" => %{"operation" => "insert", "is_move_in" => true, "tags" => [tag]}
               },
               %{"headers" => %{"control" => "snapshot-end"}},
               up_to_date_ctl()
             ]}
     right: {%{handle: "108347933-1774544054266082", offset: "1000465064_0", table: "child", where: "parent_id in (SELECT id FROM parent WHERE grandparent_id in (SELECT id FROM grandparent WHERE value = 10))", live: true}, 200, [%{"headers" => %{"control" => "up-to-date", "global_last_seen_lsn" => "1000465200"}}]}
     stacktrace:
       .../electric/plug/router_test.exs:2517: (test)

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@github-actions
Copy link
Copy Markdown
Contributor

🏋️ Load test triggered for 38e4688. Results will be posted here when complete.

@github-actions
Copy link
Copy Markdown
Contributor

Benchmark results, triggered for 38e46

  • controlled load test failed

controlled load test results

@netlify
Copy link
Copy Markdown

netlify bot commented Mar 25, 2026

Deploy Preview for electric-next ready!

Name Link
🔨 Latest commit a1e0fc1
🔍 Latest deploy log https://app.netlify.com/projects/electric-next/deploys/69c528e10fa8da00088f62cf
😎 Deploy Preview https://deploy-preview-4051--electric-next.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@robacourt robacourt force-pushed the rob/simple-subqueries-with-dnf branch 2 times, most recently from c7da8bb to f33c781 Compare March 25, 2026 14:38
@github-actions
Copy link
Copy Markdown
Contributor

🏋️ Load test triggered for f33c781. Results will be posted here when complete.

@github-actions
Copy link
Copy Markdown
Contributor

🏋️ Load test triggered for 6715aea. Results will be posted here when complete.

@github-actions
Copy link
Copy Markdown
Contributor

Benchmark results, triggered for f33c7

  • controlled load test completed

controlled load test results

@github-actions
Copy link
Copy Markdown
Contributor

Benchmark results, triggered for 6715a

  • controlled load test completed

controlled load test results

@robacourt robacourt force-pushed the rob/simple-subqueries-with-dnf branch from 6715aea to 6669eb8 Compare March 26, 2026 12:38
robacourt and others added 14 commits March 26, 2026 12:38
…link-values cache and inverted index (#3937)"

This reverts commit 8fe0a37.
Add active_conditions support to the sync protocol as a
backward-compatible change, preparing for OR/NOT in WHERE clauses.

Elixir client (from #3791):
- Tags become {position, hash} tuples with slash-delimited wire format
- active_conditions tracking and DNF visibility evaluation
- disjunct_positions derived once per shape, shared across keys

Server (minimal changes for simple case):
- Add active_conditions field to NewRecord/UpdatedRecord/DeletedRecord
- Include active_conditions in JSON headers when present
- Compute active_conditions: [true, ...] for shapes with subqueries
- Include active_conditions in snapshot SQL queries
- Read Electric-Protocol-Version header from HTTP requests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
robacourt and others added 26 commits March 26, 2026 12:38
Add a new ETS-backed reverse index module (SubqueryIndex) that stores
per-position metadata and shape-handle polarity for subquery filtering.
Wire it into Filter.add_shape/remove_shape to register/unregister shapes
with compiled DnfPlan positions. Shapes start in a fallback set until
their consumer seeds dynamic membership.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add subquery_member_from_index/2 to WhereClause that creates a
subquery_member? callback backed by the SubqueryIndex ETS table.
This enables filter-side exact verification without loading full
MapSet views, replacing the old refs_fun-backed path.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Make SubqueryIndex ETS table :public and discoverable via persistent_term
so consumers can write membership entries. Consumer seeds initial views
into the index during initialize_subquery_runtime and marks shapes ready.
Dynamic updates are applied by diffing routing views before/after each
subquery state transition, with conservative projections during buffering
(union for positive, intersection for negated dependencies).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the unconditional subquery_shape_ids_for_table union in
shapes_affected_by_record with reverse-index candidate lookup plus
exact WHERE clause verification. Candidate shapes are verified
against the full predicate using the SubqueryIndex-backed
subquery_member? callback. Fallback shapes (not yet seeded by their
consumer) pass through without verification for safety.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…Stage 6)

Remove refs_fun from Filter struct and WhereCondition - subquery
evaluation now uses SubqueryIndex-backed callbacks exclusively.
Replace the unconditional subquery_shape_ids_for_table union with
reverse-index candidate lookup in shapes_affected_by_record.
Re-enable the previously skipped OR+subquery test which now works
correctly with seeded index membership. Update existing tests to
clarify they test fallback (unseeded) behavior.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ng move-in

The subquery buffering state machine can accumulate transactions
indefinitely while waiting for a move-in query to complete. Add a
configurable limit (default 1000) that emits a :shutdown action when
exceeded, terminating the shape and triggering a 409 must-refetch for
clients.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@robacourt robacourt force-pushed the rob/simple-subqueries-with-dnf branch from 6669eb8 to a1e0fc1 Compare March 26, 2026 12:38
@robacourt robacourt force-pushed the rob/simple-subqueries-with-dnf branch from 226f9b3 to 203f4fc Compare March 26, 2026 16:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants