Skip to content

Materializer crashes when UpdatedRecord changes a primary key column #3948

@alco

Description

@alco

Bug Description

When a row's primary key column is updated in the database, the Materializer crashes with a KeyError because it tries to look up the new key (post-update) in an index that only contains the old key (pre-update).

Reproduction

Given a table where a column is part of the primary key (e.g., task_user_acl with PK (organization_id, subject_user_id, task_id)):

UPDATE task_user_acl SET task_id = 'new-task-id' WHERE task_id = 'old-task-id';

This causes the Materializer to crash:

** (KeyError) key "\"public\".\"task_user_acl\"/\"org-092\"/\"user-0078\"/\"org-100-task-1109\"" not found in:

%{
  "\"public\".\"task_user_acl\"/\"org-092\"/\"user-0078\"/\"org-092-task-0013\"" => "org-092-task-0013",
  ...
}

    :erlang.map_get("...new_key...", %{"...old_key..." => ...})
    (electric) lib/electric/shapes/consumer/materializer.ex:466: anonymous fn/3 in Electric.Shapes.Consumer.Materializer.apply_changes/2

Root Cause

In materializer.ex, the apply_changes/2 function handles UpdatedRecord by extracting only key (the new key after the update):

%Changes.UpdatedRecord{
  key: key,
  record: record,
  ...
},

Then it uses key to look up the existing value:

old_value = Map.fetch!(index, key)

But when a PK column changes, key != old_key. The index contains entries keyed by old_key, so the lookup fails.

Fix

The fix requires:

  1. Extract old_key from UpdatedRecord in addition to key
  2. Handle the case where old_key is nil (PK didn't change) by defaulting to key
  3. Use old_key when looking up/removing from the index
  4. Use old_key when removing from tag indices
  5. Use key (new key) when inserting into the index and adding to tag indices
 %Changes.UpdatedRecord{
   key: key,
+  old_key: old_key,
   record: record,
   move_tags: move_tags,
   removed_move_tags: removed_move_tags
 },
 {{index, tag_indices}, counts_and_events} ->
+  old_key = old_key || key
   ...
   tag_indices =
     tag_indices
-    |> remove_row_from_tag_indices(key, removed_move_tags)
+    |> remove_row_from_tag_indices(old_key, removed_move_tags)
     |> add_row_to_tag_indices(key, move_tags)

   if columns_present do
     {value, original_string} = cast!(record, state)
-    old_value = Map.fetch!(index, key)
+    {old_value, index} = Map.pop!(index, old_key)
     index = Map.put(index, key, value)

Context

This affects shapes that use subqueries with ACL tables where the ACL table has a composite primary key that includes a foreign key column. When that FK column is updated, the shape's materializer crashes.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions