Skip to content

Phase 3 Collation Entity Resolution colent

Jump to bottom
elb-pr edited this page Apr 7, 2026 · 2 revisions

Phase 3 — Collation & Entity Resolution (colent)

Phase 3 transforms raw, vetted intelligence into a structured relational database. It serves as the "central nervous system" of the analytical effort, moving from unstructured collection to a machine-readable format governed by the POLE (Person, Object, Location, Event) ontology.

Overview of the Collation Pipeline

The collation process is divided into three primary steps:

  1. Step 6: Structured Extraction & Schema Mapping: Defining the ontology and parsing data.
  2. Step 7: Centralized Database Management: Executing the Entity Resolution (ER) pipeline using the Fellegi-Sunter framework.
  3. Step 8: Consolidated Profiling: Synthesizing resolved entities into Subject and Corporate profiles.

Collation Data Flow

The following diagram illustrates how raw intelligence is processed into the CSDb (Claudian Sleuth Database) via the extraction and resolution pipeline.

Collation Pipeline Data Flow

graph TD
    subgraph "Natural Language Space"
        A["Unstructured Text (Witness Statements, Web Scrapes)"]
        B["Semi-structured Data (Social Media, PDF Forms)"]
    end

    subgraph "Code Entity Space (Processing)"
        C["entity_extractor.py"]
        D["NER (spaCy / Transformers)"]
        E["Coreference Resolution"]
        F["Deterministic Matcher"]
        G["Probabilistic Matcher (Fellegi-Sunter)"]
    end

    subgraph "CSDb (Persistent Storage)"
        H[("Entities Table")]
        I[("Relationships Table")]
        J[("Source Attribution Metadata")]
    end

    A --> C
    B --> C
    C --> D
    D --> E
    E --> F
    F -- "No Match" --> G
    F -- "Exact Match" --> H
    G -- "Confirmed" --> H
    H -- "Typed Edges" --> I
    C -- "Provenance" --> J
Loading

Step 6: Structured Extraction and Schema Mapping

Before extraction begins, the investigator must freeze the operational ontology to prevent schema drift.

The POLE Model

The standard framework categorizes all intelligence into four canonical entities:

Extraction Procedures

Extraction is handled by entity_extractor.py skills/claude-sleuth/references/colent/t6.2.md:3.

  1. Classification: Data is classified as Structured (CSV), Semi-structured (Web/PDF), or Unstructured (Narrative).
  2. Named Entity Recognition (NER): Extracts names, organizations, locations, and IDs skills/claude-sleuth/references/colent/t6.2.md:12.
  3. Coreference Resolution: Links multiple references (e.g., "John Smith" and "he") before record creation to prevent duplication skills/claude-sleuth/references/colent/t6.2.md:14.

Step 7: Entity Resolution (ER) & Register Management

Entity Resolution is the process of determining when fragmented records refer to the same real-world entity skills/claude-sleuth/references/colent/resolution.md:16.

Resolution Stages

The pipeline executes in four sequential stages:

  1. Deterministic Pass: Exact matches on unique identifiers (Passport, VIN, Company Reg) skills/claude-sleuth/references/colent/t7.1.md:10. Deterministic matching takes absolute precedence over probabilistic methods skills/claude-sleuth/references/colent/t6.3.md:18.
  2. Near-Deterministic Pass: Matches on combinations of high-reliability fields like Full Name + DOB + Address skills/claude-sleuth/references/colent/t7.1.md:11.
  3. Probabilistic Pass (Fellegi-Sunter): Applies weighted field comparisons using fuzzy string matching (Jaro-Winkler, Levenshtein) skills/claude-sleuth/references/colent/t7.1.md:12.
  4. Human Review: Records scoring between the auto-link and non-match thresholds are presented for manual adjudication skills/claude-sleuth/references/colent/t7.1.md:13.

Universal Metadata Tagging

Every record in the CSDb must maintain strict provenance skills/claude-sleuth/references/colent/resolution.md:18.

  • Source Attribution: Field-level attribution to the Source Register.
  • Temporal Metadata: dateObserved (real-world time) and dateCollected (investigation time).
  • Confidence Scoring: High (Deterministic), Medium (Probabilistic), or Low (Ambiguous).

Entity Resolution Logic

graph TD
    subgraph "CSDb MCP Tools"
        T1["add_entity"]
        T2["resolve_entities"]
    end

    subgraph "Resolution Logic (t7.1/t7.2)"
        R1{"Deterministic Key?"}
        R2["Fellegi-Sunter Weights"]
        R3{"Score > Threshold?"}
        R4["Manual Review Card"]
    end

    T1 --> R1
    R1 -- "Yes" --> T2
    R1 -- "No" --> R2
    R2 --> R3
    R3 -- "Auto-Merge" --> T2
    R3 -- "Uncertain" --> R4
    R4 -- "Analyst Confirm" --> T2
Loading

Step 8: Consolidated Profiling & Specialized Research

After resolution, entities are synthesized into high-level profiles to support decision-making.

Subject and Corporate Profiles

  • Subject Profiles: Focus on identity, employment history, digital footprints (social media IDs, not just names), and financial summaries.
  • Corporate Profiles: Map ownership chains to the Ultimate Beneficial Owner (UBO) and explicitly check for Circular Ownership (e.g., A owns B owns C owns A), which flags potential money laundering.

Specialized Research

  • Genealogical Research: Uses the Genealogical Proof Standard (GPS) to map kinship-based control structures.
  • Cultural Context: Maps formal and informal power structures (clans, patronage networks) when crossing cultural boundaries.

Quality and Integrity Checkpoints

To maintain analytical rigor, the following audits are mandatory:

Overview

Investigation Phases

CLI Tools

Scripts

Templates

CSDb

Project

Clone this wiki locally