Skip to content
This repository was archived by the owner on May 13, 2026. It is now read-only.

feat(fingerprints): wire generic ingestion into the fingerprint pipeline - #770

Merged
shiba4life merged 1 commit into
mainfrom
worktree-wire-fingerprints
Apr 30, 2026
Merged

feat(fingerprints): wire generic ingestion into the fingerprint pipeline#770
shiba4life merged 1 commit into
mainfrom
worktree-wire-fingerprints

Conversation

@shiba4life

Copy link
Copy Markdown
Collaborator

Summary

Every record written through IngestionService now passes through the text extractor, so emails and phone numbers in normal user-data ingestion (Notes, Contacts, smart-folder, file-upload, Apple imports) become Fingerprint / Mention / Edge records — and contact-shaped sources auto-form tentative Personas. Before this change, only the dedicated /api/fingerprints/* endpoints fed the identity graph; the rest of ingestion was disconnected from it.

The change lands as five phases in one PR:

  • Phase 0auto_propose::maybe_spawn_persona_sweep helper centralizes the post-ingest sweep spawn across ingest_text_signals_batch, ingest_photo_faces_batch, and import_contacts (the last of which previously never triggered the sweep — bug fix).
  • Phase 1SignalBinding enum on the text extractor. Strong emits StrongMatch (0.95) edges; CoOccurrence (default) preserves existing 0.3 behavior. Shared-inbox demotion (info@, support@, …) still applies in both modes — a contact card listing info@acme.com lands at 0.475, below the 0.85 sweep floor by design.
  • Phase 2schema_policy::binding_for_schema allowlists Contacts / AddressBook / AppleContacts / CalendarEvent / AppleCalendarEvent / EmailHeader for Strong binding. Everything else stays on CoOccurrence.
  • Phase 2.5SYSTEM_DESCRIPTIVE_NAMES mirrors the 12 fingerprint subsystem schemas locally; a unit test pins it to the upstream PHASE_1_DESCRIPTIVE_NAMES so it cannot drift.
  • Phase 3ingestion::fingerprint_hook::run_after_batch runs the extractor over every non-system mutation after mutate_batch succeeds, then spawns the persona sweep if anything yielded. Wired into both execute_mutations_with_tracking and the per-item decomposition.rs path.

Invariants

  • Best-effort: per-record extraction failures log at warn and never propagate. The user-data write already succeeded before the hook runs.
  • Recursion-safe: writes on the fingerprint subsystem's own schemas are skipped via is_system_descriptive_schema, so a Mention insert does not trigger another extraction pass.
  • Idempotent: ingest_text_signals keys its Mention + ExtractionStatus on (source_schema, source_key, "text_regex"), so re-ingesting the same record converges to the same row.

Out of scope

Photo face extraction in the generic ingest path (would need ONNX inline), name extraction (P2-6 NER), per-schema identity_assertion config field, and Apple-import side-call cleanup at apple_import.rs:904 (now duplicate work via the hook — follow-up PR).

Test plan

  • cargo clippy --workspace --all-targets -- -D warnings (clean)
  • cargo test -p fold_db_node --lib (859 passed, 0 failed)
  • cargo test --workspace --all-targets -- --skip cli_does_not_auto_spawn_daemon_when_port_unset (all green; the skipped test is a pre-existing flake on main)
  • New unit tests cover: SignalBinding::Strong emits StrongMatch at 0.95; CoOccurrence keeps 0.3; shared-inbox demotion in both modes; binding_for_schema allowlist + case-insensitivity; SYSTEM_DESCRIPTIVE_NAMESPHASE_1_DESCRIPTIVE_NAMES parity; collect_string_fields walks nested objects/arrays and skips non-strings; source_key_for uses KeyValue Display + falls back to uuid.
  • Manual end-to-end demo: paste sample_data/contacts/address_book.json into the Ingestion tab → wait ~30s → tentative personas appear in People → Personas. (Will run before merge.)

🤖 Generated with Claude Code

Every record written through `IngestionService` now passes through the
text extractor, so emails and phone numbers in user-data ingestion
(Notes, Contacts, smart-folder, file-upload, Apple imports) become
Fingerprint/Mention/Edge rows and contact-shaped sources auto-form
tentative Personas.

Phase 0 — `maybe_spawn_persona_sweep` helper centralizes the spawn
pattern across `ingest_text_signals_batch`, `ingest_photo_faces_batch`,
and now `import_contacts` (which previously never triggered the sweep).

Phase 1 — `SignalBinding` enum on the text extractor. `Strong` emits
StrongMatch (0.95) edges; `CoOccurrence` (default) preserves existing
0.3 behavior. Shared-inbox demotion still applies in both modes — a
contact card listing `info@acme.com` lands at 0.475, below the 0.85
auto-sweep floor by design.

Phase 2 — `schema_policy::binding_for_schema` allowlists Contacts /
AddressBook / AppleContacts / CalendarEvent / AppleCalendarEvent /
EmailHeader for Strong binding. Everything else stays on CoOccurrence.

Phase 2.5 — `SYSTEM_DESCRIPTIVE_NAMES` mirrors the 12 fingerprint
subsystem schemas locally with a unit test pinning it to the upstream
`PHASE_1_DESCRIPTIVE_NAMES` so it cannot drift.

Phase 3 — `ingestion::fingerprint_hook::run_after_batch` runs the
extractor over every non-system mutation after `mutate_batch` succeeds,
then spawns the persona sweep if anything yielded. Wired into both
`execute_mutations_with_tracking` and the per-item path in
`decomposition.rs`. Best-effort and recursion-safe: extraction failure
is logged but never breaks user-data ingestion, and writes on system
schemas are skipped so the hook never recurses into its own writes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@shiba4life
shiba4life enabled auto-merge April 30, 2026 21:40
@shiba4life
shiba4life added this pull request to the merge queue Apr 30, 2026
Merged via the queue into main with commit 26dcb77 Apr 30, 2026
10 checks passed
@shiba4life
shiba4life deleted the worktree-wire-fingerprints branch April 30, 2026 22:04
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant