This repository was archived by the owner on May 13, 2026. It is now read-only.
feat(fingerprints): wire generic ingestion into the fingerprint pipeline - #770
Merged
Conversation
Every record written through `IngestionService` now passes through the text extractor, so emails and phone numbers in user-data ingestion (Notes, Contacts, smart-folder, file-upload, Apple imports) become Fingerprint/Mention/Edge rows and contact-shaped sources auto-form tentative Personas. Phase 0 — `maybe_spawn_persona_sweep` helper centralizes the spawn pattern across `ingest_text_signals_batch`, `ingest_photo_faces_batch`, and now `import_contacts` (which previously never triggered the sweep). Phase 1 — `SignalBinding` enum on the text extractor. `Strong` emits StrongMatch (0.95) edges; `CoOccurrence` (default) preserves existing 0.3 behavior. Shared-inbox demotion still applies in both modes — a contact card listing `info@acme.com` lands at 0.475, below the 0.85 auto-sweep floor by design. Phase 2 — `schema_policy::binding_for_schema` allowlists Contacts / AddressBook / AppleContacts / CalendarEvent / AppleCalendarEvent / EmailHeader for Strong binding. Everything else stays on CoOccurrence. Phase 2.5 — `SYSTEM_DESCRIPTIVE_NAMES` mirrors the 12 fingerprint subsystem schemas locally with a unit test pinning it to the upstream `PHASE_1_DESCRIPTIVE_NAMES` so it cannot drift. Phase 3 — `ingestion::fingerprint_hook::run_after_batch` runs the extractor over every non-system mutation after `mutate_batch` succeeds, then spawns the persona sweep if anything yielded. Wired into both `execute_mutations_with_tracking` and the per-item path in `decomposition.rs`. Best-effort and recursion-safe: extraction failure is logged but never breaks user-data ingestion, and writes on system schemas are skipped so the hook never recurses into its own writes. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every record written through
IngestionServicenow passes through the text extractor, so emails and phone numbers in normal user-data ingestion (Notes, Contacts, smart-folder, file-upload, Apple imports) becomeFingerprint/Mention/Edgerecords — and contact-shaped sources auto-form tentative Personas. Before this change, only the dedicated/api/fingerprints/*endpoints fed the identity graph; the rest of ingestion was disconnected from it.The change lands as five phases in one PR:
auto_propose::maybe_spawn_persona_sweephelper centralizes the post-ingest sweep spawn acrossingest_text_signals_batch,ingest_photo_faces_batch, andimport_contacts(the last of which previously never triggered the sweep — bug fix).SignalBindingenum on the text extractor.StrongemitsStrongMatch(0.95) edges;CoOccurrence(default) preserves existing 0.3 behavior. Shared-inbox demotion (info@,support@, …) still applies in both modes — a contact card listinginfo@acme.comlands at 0.475, below the 0.85 sweep floor by design.schema_policy::binding_for_schemaallowlistsContacts/AddressBook/AppleContacts/CalendarEvent/AppleCalendarEvent/EmailHeaderfor Strong binding. Everything else stays on CoOccurrence.SYSTEM_DESCRIPTIVE_NAMESmirrors the 12 fingerprint subsystem schemas locally; a unit test pins it to the upstreamPHASE_1_DESCRIPTIVE_NAMESso it cannot drift.ingestion::fingerprint_hook::run_after_batchruns the extractor over every non-system mutation aftermutate_batchsucceeds, then spawns the persona sweep if anything yielded. Wired into bothexecute_mutations_with_trackingand the per-itemdecomposition.rspath.Invariants
warnand never propagate. The user-data write already succeeded before the hook runs.is_system_descriptive_schema, so aMentioninsert does not trigger another extraction pass.ingest_text_signalskeys its Mention + ExtractionStatus on(source_schema, source_key, "text_regex"), so re-ingesting the same record converges to the same row.Out of scope
Photo face extraction in the generic ingest path (would need ONNX inline), name extraction (P2-6 NER), per-schema
identity_assertionconfig field, and Apple-import side-call cleanup atapple_import.rs:904(now duplicate work via the hook — follow-up PR).Test plan
cargo clippy --workspace --all-targets -- -D warnings(clean)cargo test -p fold_db_node --lib(859 passed, 0 failed)cargo test --workspace --all-targets -- --skip cli_does_not_auto_spawn_daemon_when_port_unset(all green; the skipped test is a pre-existing flake onmain)SignalBinding::Strongemits StrongMatch at 0.95;CoOccurrencekeeps 0.3; shared-inbox demotion in both modes;binding_for_schemaallowlist + case-insensitivity;SYSTEM_DESCRIPTIVE_NAMES↔PHASE_1_DESCRIPTIVE_NAMESparity;collect_string_fieldswalks nested objects/arrays and skips non-strings;source_key_foruses KeyValue Display + falls back to uuid.sample_data/contacts/address_book.jsoninto the Ingestion tab → wait ~30s → tentative personas appear in People → Personas. (Will run before merge.)🤖 Generated with Claude Code