feat: add support for subqueries behind a feature flag

[icehaunter]

This PR allows initial, under-optimised implementation of where clauses in the form of column IN (SELECT id FROM parent WHERE ...)

It has some limitations:

• Inner select can only have PKs as selected columns
• Composite PKs are not supported
• Runtime evaluation is unindexed, so it'll be fairly slow on a lot of dependent shapes
• Any change inside the inner select (move-in/out) will immediately cause all dependent shapes to be dropped
• Inner select is done as an (unexposed) shape, which is materialized. This materialization is in memory for now, so it's not memory-efficient
• Only 1 inner select is supported for now

This PR is mergable because it doesn't change current behaviour, but adds a feature flag as ENV variable that allows creations of these shapes, to allow us to iterate in main codebase and make this reviewable piece by piece

[netlify]

✅ Deploy Preview for electric-next ready!

Name	Link
🔨 Latest commit	ac089ee
🔍 Latest deploy log	https://app.netlify.com/projects/electric-next/deploys/68a446e6f1dc300008a17ed4
😎 Deploy Preview	https://deploy-preview-3005--electric-next.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[codecov]

Codecov Report

❌ Patch coverage is 73.96226% with 69 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.48%. Comparing base (a7acded) to head (760fa88).
⚠️ Report is 7 commits behind head on main.

Files with missing lines	Patch %	Lines
packages/sync-service/lib/electric/shapes/shape.ex	41.37%	34 Missing ⚠️
...nc-service/lib/electric/replication/eval/parser.ex	75.00%	13 Missing ⚠️
...kages/sync-service/lib/electric/shapes/consumer.ex	80.00%	6 Missing ⚠️
...rvice/lib/electric/shapes/consumer/materializer.ex	88.67%	6 Missing ⚠️
packages/sync-service/lib/electric/config.ex	0.00%	5 Missing ⚠️
...nc-service/lib/electric/replication/eval/walker.ex	33.33%	2 Missing ⚠️
...s/sync-service/lib/electric/shapes/where_clause.ex	66.66%	1 Missing ⚠️
packages/sync-service/lib/electric/utils.ex	75.00%	1 Missing ⚠️
packages/sync-service/lib/pg_interop/sublink.ex	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3005      +/-   ##
==========================================
- Coverage   78.70%   77.48%   -1.22%     
==========================================
  Files         163      167       +4     
  Lines        8476     8725     +249     
  Branches      282      283       +1     
==========================================
+ Hits         6671     6761      +90     
- Misses       1803     1962     +159     
  Partials        2        2              

Flag	Coverage Δ
elixir	76.30% <73.96%> (-1.37%)	⬇️
elixir-client	74.05% <ø> (+0.53%)	⬆️
packages/experimental	87.73% <ø> (ø)
packages/react-hooks	86.30% <ø> (ø)
packages/typescript-client	93.16% <ø> (ø)
packages/y-electric	55.12% <ø> (ø)
postgres-140000	76.24% <73.96%> (-1.66%)	⬇️
postgres-150000	76.15% <73.96%> (-1.75%)	⬇️
postgres-170000	76.16% <73.96%> (-1.60%)	⬇️
sync-service	76.54% <73.96%> (-1.60%)	⬇️
typescript	85.23% <ø> (ø)
unit-tests	77.48% <73.96%> (-1.22%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[msfstef]

Nice work, love that it's fairly contained!

I've left some comments mostly about 1) some lookups that I think might be avoidable and 2) the Shape.new code becoming very hard to follow and understand.

Other limitations are known, I did want to highlight the blocking of ShapeCache for materializing subqueries which feels like it should be noted even for the feature flag.

[msfstef]

Do we need pids_by_shape_handle? Would it not be enough to just use Consumer.name(stack_id, shape_handle) instead and avoid maintaining another lookup?

[KyleAMathews]

ChatGPT Pro's review https://chatgpt.com/share/68a4839c-5294-800c-9d71-b79d0e231ae3

[robacourt]

benchmark this

[github-actions]

Benchmark results, triggered for 760fa

• write fanout completed

• unrelated shapes one client latency completed

• many shapes one client latency completed

• concurrent shape creation completed

• diverse shape fanout completed