Skip to content

chore(db): remove deprecated entity schema from memory_links - #3177

Merged
nicoloboschi merged 1 commit into
mainfrom
chore/drop-memory-links-entity-schema
Aug 4, 2026
Merged

chore(db): remove deprecated entity schema from memory_links#3177
nicoloboschi merged 1 commit into
mainfrom
chore/drop-memory-links-entity-schema

Conversation

@nicoloboschi

Copy link
Copy Markdown
Collaborator

Summary

Closes #3005.

Entity edges are no longer materialized in memory_links. Retain stores memory-to-entity associations in unit_entities, and both read paths derive entity edges from that table on demand — the /graph endpoint from shared unit_entities rows, and recall via the unit_entities self-join. Migration e9b2c7d1f3a4 deleted the stored entity rows, and current writers only ever pass entity_id = NULL, so the entity-specific schema on memory_links was dead weight that made the table look like it still supported a second, materialized source of truth for entity edges.

This removes that schema and the write-path plumbing behind it.

Schema changes (new migration, PG + Oracle)

  • Defensively delete any residual link_type = 'entity' rows.
  • Drop the entity_id column and its FK to entities.
  • Drop the entity index (idx_memory_links_entity / idx_ml_entity).
  • Recreate the link_type CHECK without 'entity' (temporal, semantic, and the legacy causal values remain valid).
  • Collapse the function-based idx_memory_links_unique to (from_unit_id, to_unit_id, link_type) — non-entity rows already deduplicate on those three columns via the old COALESCE(entity_id, nil) key, so effective uniqueness is preserved.

Production-safe by construction

memory_links and its entity index can be very large on established banks, so the migration avoids long exclusive locks:

  • the residual delete is chunked with per-batch commits;
  • every index is swapped CONCURRENTLY (PG) / ONLINE (Oracle), building the new unique index under a temporary name and renaming so duplicate protection is never dropped;
  • the new CHECK is added NOT VALID then VALIDATEd separately, so writers are never blocked on a full-table scan.

Downgrade restores the former schema shape but, as documented in the migration, cannot reconstruct the historical entity rows e9b2c7d1f3a4 already deleted.

Application code

  • Remove MemoryLink.entity_id / MemoryLink.entity and Entity.memory_links; update the docstring and CHECK metadata.
  • Simplify internal link tuples from (from, to, link_type, weight, entity_id) to (from, to, link_type, weight) across the temporal, semantic, causal, and graph-maintenance producers.
  • Remove _NIL_ENTITY_UUID and the nil_entity_uuid DataAccessOps parameter; drop the entity_id column/placeholder from the PG and Oracle bulk inserts (conflict target is now (from_unit_id, to_unit_id, link_type)).
  • Drop the dead entity_id tiebreaker from the chunk-storage lock ordering and a stale runtime comment.
  • The graph API keeps returning dynamically derived entity edges from unit_entities — unchanged.

Tests

  • New test_migration_drop_memory_links_entity.py (dedicated pg0 instance): asserts the column/index/entity CHECK value are gone at head, the unique index is three columns, and the downgrade → re-upgrade round-trip restores and re-drops the shape.
  • Updated test_link_utils.py and test_document_transfer.py for four-element tuples / no entity_id column.

Non-goals

Unchanged: the unit_entities model, link_type = 'entity' in graph/API response types, entity resolution / co-occurrence / orphan cleanup, and temporal/semantic/causal link generation.

@nicoloboschi
nicoloboschi force-pushed the chore/drop-memory-links-entity-schema branch from 02cbfb0 to cab9145 Compare August 4, 2026 15:07
Entity edges are no longer materialized in memory_links. Retain stores
memory-to-entity associations in unit_entities, and both read paths derive
entity edges from that table on demand — the /graph endpoint from shared
unit_entities rows and recall via the unit_entities self-join. Migration
e9b2c7d1f3a4 deleted the stored entity rows and current writers only ever
pass entity_id = NULL, leaving the entity-specific schema on memory_links
as dead weight.

New migration (PG + Oracle) drops the entity_id column and its FK, the
entity index, 'entity' from the link_type CHECK, and the entity_id term in
the function-based unique index (which collapses to
(from_unit_id, to_unit_id, link_type)). It is written to avoid long locks
on large tables: the residual delete is chunked with per-batch commits,
indexes are swapped CONCURRENTLY, and the new CHECK is added NOT VALID then
validated separately.

Application code drops _NIL_ENTITY_UUID and the nil_entity_uuid DataAccessOps
parameter, simplifies internal link tuples to four elements
(from, to, link_type, weight), and removes the entity_id column/placeholder
from the PG and Oracle bulk inserts and the chunk-storage lock ordering.
The graph API keeps returning dynamically derived entity edges.
@nicoloboschi
nicoloboschi force-pushed the chore/drop-memory-links-entity-schema branch from cab9145 to 437f6f1 Compare August 4, 2026 15:14
@nicoloboschi
nicoloboschi merged commit 5f8a030 into main Aug 4, 2026
104 checks passed
nicoloboschi added a commit that referenced this pull request Aug 7, 2026
…#3177) (#3244)

* Revert "chore(db): remove deprecated entity schema from memory_links (#3177)"

This reverts commit 5f8a030.

* fix(migrations): re-parent f2a6d8c4b1e9 onto e4a7c1b9d2f6 after removing c1e7a9d3f5b2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

chore(db): remove deprecated entity schema from memory_links

1 participant