Skip to content

Conversation

@robacourt
Copy link
Contributor

@robacourt robacourt commented Dec 3, 2025

Summary

Migrates the Filter module and its namespace (WhereCondition, EqualityIndex, InclusionIndex) from
Elixir maps to ETS tables to reduce garbage collection pressure with large numbers of shapes
(200K+).

Problem

With 200K shapes being added and removed frequently, the map-based implementation causes large GC
delays due to copying immutable map data structures on every add/remove operation.

Solution

Store filter data in ETS tables (outside the process heap) instead of nested Elixir maps:

  • 5 private ETS tables per filter: shapes_table, tables_table, where_cond_table, eq_index_table,
    incl_index_table
  • Same algorithmic complexity: O(1) for equality lookups, O(tree depth) for inclusion index
  • API unchanged: All existing tests pass without modification

Performance Results (200K shapes)

Scenario ETS Implementation Original (maps) Target
Equality index lookup 5.75µs 4.41µs <100µs ✅
Multiple matches (20) 13.82µs 9.07µs <100µs ✅
Post-churn performance 9.49µs 4.69µs <100µs ✅

The ETS implementation meets the <100µs target for affected_shapes with 200K equality-indexed
shapes.

Key Changes

  1. filter.ex: Changed struct to hold ETS table references; operations mutate ETS in-place
  2. where_condition.ex: WhereConditions now identified by refs and stored in ETS
  3. index.ex: Changed from protocol dispatch to direct function dispatch
  4. equality_index.ex: Stores entries in ETS with keys {where_cond_id, field, value}
  5. inclusion_index.ex: Stores tree nodes in ETS with keys {where_cond_id, field, path}
Screenshot 2025-12-04 at 17 06 47

@codecov
Copy link

codecov bot commented Dec 3, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.21%. Comparing base (d8588c9) to head (4b8ae25).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #3547   +/-   ##
=======================================
  Coverage   75.21%   75.21%           
=======================================
  Files          51       51           
  Lines        2744     2744           
  Branches      409      405    -4     
=======================================
  Hits         2064     2064           
  Misses        678      678           
  Partials        2        2           
Flag Coverage Δ
electric-telemetry 22.71% <ø> (ø)
elixir 57.38% <ø> (ø)
elixir-client 73.94% <ø> (ø)
packages/experimental 87.73% <ø> (ø)
packages/react-hooks 86.48% <ø> (ø)
packages/typescript-client 93.08% <ø> (ø)
packages/y-electric 55.12% <ø> (ø)
typescript 87.46% <ø> (ø)
unit-tests 75.21% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@robacourt
Copy link
Contributor Author

benchmark this

Copy link
Contributor

@magnetised magnetised left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great stuff!

I'm trusting the tests to validate the functionality -- haven't reviewed the algorithm itself. Can we cull the value-less comments and docs though pls.

@msfstef
Copy link
Contributor

msfstef commented Dec 3, 2025

This looks pretty good but a bit hard to review - in the meantime might be worth running a benchmark on it as well?

@robacourt
Copy link
Contributor Author

benchmark this

@github-actions
Copy link
Contributor

github-actions bot commented Dec 4, 2025

Benchmark results, triggered for 7fc5c

  • write fanout completed

write fanout results

  • unrelated shapes one client latency completed

unrelated shapes one client latency results

  • many shapes one client latency completed

many shapes one client latency results

  • concurrent shape creation completed

concurrent shape creation results

  • diverse shape fanout completed

diverse shape fanout results

@netlify
Copy link

netlify bot commented Dec 8, 2025

Deploy Preview for electric-next ready!

Name Link
🔨 Latest commit 1ec78ae
🔍 Latest deploy log https://app.netlify.com/projects/electric-next/deploys/69370b216da66b00086b4ef7
😎 Deploy Preview https://deploy-preview-3547--electric-next.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@robacourt robacourt marked this pull request as ready for review December 9, 2025 12:02
Copy link
Contributor

@msfstef msfstef left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks sane!

@KyleAMathews
Copy link
Contributor

Just reading through the PR body again. It's probably worth calling out that the micro benchmarks are slower but when running, it's faster due to no/less gc pauses.

@robacourt robacourt merged commit ebdc25d into main Dec 9, 2025
47 checks passed
@robacourt robacourt deleted the rob/ets-filter branch December 9, 2025 17:50
@github-actions
Copy link
Contributor

This PR has been released! 🚀

The following packages include changes from this PR:

Thanks for contributing to Electric!

KyleAMathews pushed a commit that referenced this pull request Dec 11, 2025
…e comments

The release notification comments were missing package data because
backticks in the body text (e.g., `@electric-sql/client@1.2.8`) were
being interpreted by the shell as command substitution. This caused
the package names to be replaced with empty strings.

Fix by using gh's --body-file - flag to pass the body via stdin,
avoiding shell interpretation entirely.

Fixes comments like #3547 (comment)
KyleAMathews added a commit that referenced this pull request Dec 11, 2025
The release notification comments were missing package data because
backticks in the body text (e.g., `@electric-sql/client@1.2.8`) were
being interpreted by the shell as command substitution. This caused the
package names to be replaced with empty strings.

Fix by using gh's --body-file - flag to pass the body via stdin,
avoiding shell interpretation entirely.

Fixes comments like
#3547 (comment)

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants