Skip to content
BranchesTags

Repository files navigation

Layer10 — Grounded Long-Term Memory System

A production-quality grounded memory system over the Enron Email Dataset. Extracts entities, claims, and evidence from 64,675 emails(truncated kaggle enron dataest); stores them in a Neo4j knowledge graph with full temporal tracking; retrieves grounded, cited answers to natural-language questions; and visualizes the memory graph interactively.

Demo video (sorry audio got cut off first i showed the neo4j web interface and then custom UI KG) - https://www.loom.com/share/a8514b0d352d4eb7b6a8a55b05f57d79

image image

Download the emails.csv dataset form kaggle - https://www.kaggle.com/datasets/wcukierski/enron-email-dataset/data

Quick Start

# 1. Clone / unzip the project
cd layer10

# 2. Create virtual environment and install
uv sync          # or: python -m venv .venv && pip install -e .

# 3. Set up environment variables
cp .env.example .env    # then fill in HF_TOKEN (free HuggingFace token)

# 4. Start Neo4j
docker compose up -d neo4j
# Wait ~30s for Neo4j to become ready

# 5. Run the visualization (uses the pre-built database)
python -m layer10.step7_viz.run --open
# Opens http://localhost:8000 in your browser

The pre-built SQLite database (data/db/layer10_eighth.db) and a pre-built frontend (src/layer10/step7_viz/frontend/dist/) are included. You can explore the visualization immediately after loading data into Neo4j (Step 5 below).

Prerequisites

Requirement Version Notes
Python 3.11+
uv any pip install uv or curl -LsSf https://astral.sh/uv/install.sh | sh
Docker Desktop any Must be running for Neo4j
Node.js 18+ Only needed to rebuild the frontend
HuggingFace token free From https://huggingface.co/settings/tokens

Repository Layout

layer10/
├── src/layer10/
│   ├── step1_ingestion/     Email parsing and SQLite storage
│   ├── step2_schema/        Pydantic models + DDL
│   ├── step3_extraction/    LLM-based entity/claim extraction
│   ├── step4_dedup/         Deduplication and entity canonicalization
│   ├── step5_graph/         Neo4j loader, vector index, hybrid search
│   ├── step6_retrieval/     Query pipeline: RRF + reranking + grounding
│   └── step7_viz/           FastAPI + React/D3 visualization
├── data/
│   └── db/
│       └── layer10_eighth.db   Active SQLite database (1/8th dataset)
├── outputs/
│   ├── context_packs/          Example retrieval results (JSON)
│   └── graph_export/           Graph snapshot (JSON)
├── tests/                  36 unit tests
├── docker-compose.yml      Neo4j 5.20 service definition
├── pyproject.toml
└── TECHNICAL_REPORT.md     Full design document

End-to-End Pipeline

Step 0: Dataset

The pipeline expects emails.csv in the project root. The file is the standard Enron Email Dataset (1.4 GB, 517,401 rows, columns: file, message).

A pre-ingested 1/8th subset is already in data/db/layer10_eighth.db. You can skip to Step 5 to load the graph and start the visualization.

Step 1: Ingestion

Parses RFC 2822 email messages, stores artifacts in SQLite.

# Ingest 1/8th of the dataset (recommended for demo)
python -m layer10.step1_ingestion.run --fraction 0.125

# Ingest full dataset
python -m layer10.step1_ingestion.run

# Options
--db-path data/db/layer10.db   # output database path
--fraction 0.125               # fraction of emails to ingest
--batch-size 1000              # write batch size

Output: artifacts table with parsed headers, deduplication-ready content hash, and cleaned body text.

Step 2: Schema

No CLI step required. The schema is applied automatically during Step 1.

To inspect the schema: data/db_schema.sql

Step 3: LLM Extraction

Runs Qwen2.5-7B-Instruct (HuggingFace free tier) over sampled active artifacts to extract entities and claims.

# Set your HuggingFace token
export HF_TOKEN=hf_...   # or add to .env

# Extract from 200 artifacts (default)
PYTHONIOENCODING=utf-8 python -m layer10.step3_extraction.run

# Options
--batch-size 200          # number of artifacts to process
--db-path data/db/layer10_eighth.db
--skip-extracted          # skip artifacts already processed

Rate limits: The HuggingFace free tier allows ~30 requests/min. Extraction of 200 artifacts takes ~10–15 minutes. The pipeline saves progress after each artifact and is safe to interrupt and resume.

Expected success rate: ~30–40% with Qwen 7B free tier (validation failures, format errors, hallucinated excerpts are rejected).

Step 4: Deduplication and Canonicalization

Three sub-stages: artifact dedup, entity canonicalization, claim dedup.

# Full dedup pipeline
PYTHONIOENCODING=utf-8 python -m layer10.step4_dedup.run

# Individual stages
PYTHONIOENCODING=utf-8 python -m layer10.step4_dedup.run --stage artifact
PYTHONIOENCODING=utf-8 python -m layer10.step4_dedup.run --stage entity
PYTHONIOENCODING=utf-8 python -m layer10.step4_dedup.run --stage claim
  • Artifact dedup (Phase 1): Marks duplicate artifacts by SHA-256 content hash. Phase 2: MinHash LSH for near-duplicates (Jaccard ≥ 0.85, 5-shingle, 128 permutations).
  • Entity canonicalization: Merges person entities by email address and name similarity. Backfills confidence scores from raw_entities. Records all merges in entity_merge_log.
  • Claim dedup: Groups semantically similar claims (same subject+object+type); keeps the most-confident canonical claim; marks superseded claims.

Step 5: Memory Graph (Neo4j)

Loads the SQLite data into Neo4j. Requires Docker Desktop running.

# Start Neo4j (first time takes ~30s to initialize)
docker compose up -d neo4j

# Wait for Neo4j to be ready
# Check: curl http://localhost:7474 (should return JSON)

# Load everything into Neo4j
PYTHONIOENCODING=utf-8 python -m layer10.step5_graph.run

# Options
--neo4j-uri bolt://localhost:7687
--neo4j-user neo4j
--neo4j-password layer10pass
--db-path data/db/layer10_eighth.db
--skip-embeddings   # skip vector embedding (faster, disables vector search)

Neo4j browser: http://localhost:7474 (user: neo4j, password: layer10pass)

Loaded graph (from layer10_eighth.db):

  • 64,675 Artifact nodes
  • 114 Entity nodes (52 Person, 8 Organization, 54 Topic)
  • 52 Claim nodes
  • 1,343 Evidence nodes
  • Relationships: HAS_EVIDENCE, FROM_ARTIFACT, SUBJECT, OBJECT, ALIAS_OF, REPLIES_TO

Step 6: Retrieval & Grounding

Query the memory graph with natural language. Returns a ContextPack with cited evidence.

# Run demo queries
PYTHONIOENCODING=utf-8 python -m layer10.step6_retrieval.run --demo

# Single query
PYTHONIOENCODING=utf-8 python -m layer10.step6_retrieval.run \
  --query "Who does Phillip Allen report to?"

# Query with JSON output
PYTHONIOENCODING=utf-8 python -m layer10.step6_retrieval.run \
  --query "What decisions were made about gas trading?" \
  --json-out outputs/context_packs/result.json \
  --top-k 5

Example output files: outputs/context_packs/example_context_packs.json (7 pre-generated examples).

Step 7: Visualization

Interactive knowledge graph UI.

# Build frontend (only needed once, or after frontend changes)
cd src/layer10/step7_viz/frontend
npm install
npm run build
cd ../../../..   # back to project root

# Start the visualization server
python -m layer10.step7_viz.run
# Open http://localhost:8000

# Development mode (Vite HMR + auto-open browser)
python -m layer10.step7_viz.run --dev --open

UI Features:

  • Graph canvas: D3 force-directed graph. Click = inspect entity. Double-click = neighbourhood view. Drag = pin node.
  • Filter bar: Filter by entity type, confidence threshold, date range. Toggle current/historical claims.
  • Evidence panel: Scrollable list of claims + evidence cards with source excerpt highlighted in original email body.
  • Merge inspector: View entity alias merges with reasons and confidence scores.
  • Search (Ask): Semantic search using the Step 6 retrieval pipeline. Returns cited evidence with conflict detection.
  • Timeline scrubber: Monthly email volume sparkline with dual date-range sliders.
  • Stats bar: Live counts of artifacts, entities, claims, and conflicts.

Running Tests

pytest tests/ -v
# Expected: 36 passed

Environment Variables

Create a .env file in the project root:

# Required for LLM extraction (Step 3)
HF_TOKEN=hf_your_token_here

# Optional overrides (defaults shown)
NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=layer10pass
SQLITE_DB=data/db/layer10_eighth.db

Get a free HuggingFace token at: https://huggingface.co/settings/tokens

Outputs

Pre-generated outputs are included in the repository:

File Description
outputs/context_packs/example_context_packs.json 7 example retrieval results covering identity, hierarchy, decisions, projects
outputs/graph_export/graph_snapshot.json Graph statistics and node/edge counts
data/db/layer10_eighth.db SQLite database with all pipeline outputs

Docker (Neo4j)

# docker-compose.yml — excerpt
services:
  neo4j:
    image: neo4j:5.20
    ports:
      - "7474:7474"   # HTTP browser
      - "7687:7687"   # Bolt protocol
    environment:
      NEO4J_AUTH: neo4j/layer10pass
      NEO4J_PLUGINS: '["apoc", "graph-data-science"]'
# Start
docker compose up -d neo4j

# Stop (preserves data in Docker volume)
docker compose stop neo4j

# Full reset (deletes all graph data)
docker compose down -v neo4j

Technical Report

See TECHNICAL_REPORT.md for the full design document covering:

  1. Ontology design and schema
  2. Extraction contract and LLM prompting strategy
  3. Deduplication strategy (artifact + entity + claim levels)
  4. Memory graph design decisions
  5. Retrieval and grounding pipeline
  6. Visualization architecture
  7. Layer10 adaptation considerations (multi-source, incremental updates, permissions, scaling)

Architecture Summary

emails.csv
    │
    ▼ Step 1 — Ingestion
SQLite: artifacts
    │
    ▼ Step 3 — LLM Extraction (Qwen 7B, HuggingFace)
SQLite: raw_entities, raw_claims, evidence_pointers
    │
    ▼ Step 4 — Deduplication
SQLite: person_entities, org_entities, topic_entities,
        canonical_claims, entity_merge_log
    │
    ▼ Step 5 — Memory Graph
Neo4j: (:Person|Org|Topic|Artifact|Claim|Evidence)
       + HNSW vector index (all-MiniLM-L6-v2, 384-dim)
    │
    ├──▶ Step 6 — Retrieval
    │    VectorChannel + BM25Channel + GraphChannel
    │    → RRF fusion → CrossEncoder reranking → ContextPack
    │
    └──▶ Step 7 — Visualization
         FastAPI + React/D3 → http://localhost:8000

Known Limitations

  • Low extraction yield: ~30% success rate with Qwen 7B free tier due to JSON formatting errors and hallucinated excerpts (rejected by grounding verifier). A larger model or fine-tuned extractor would improve this substantially.
  • Small extraction sample: Only 200 of 36,793 active artifacts were processed for LLM extraction in the demo database. The pipeline is designed to scale to the full corpus with sufficient API quota.
  • SQLite concurrency: The pipeline uses SQLite, which limits write concurrency. For production, PostgreSQL would be preferred.
  • HuggingFace rate limits: Free tier limits ~30 requests/min. Burst extraction runs may hit 429 errors; the pipeline retries with exponential backoff but may stall.

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages