feat: real-time indexing hooks — per-message prompt/response pairing

[EtanHey]

Summary

• Real-time BrainLayer indexing via Claude Code hooks (Stop + UserPromptSubmit + PostCompact)
• Pairs user prompts with assistant responses using in-memory correlation
• Content-hash dedup (session_id + SHA256) prevents duplicate indexing during batch→realtime migration
• Chapter boundaries from PostCompact events for session structure
• Fallback queue when DB is locked/unavailable

Based on Research 10 architecture.

Components

File	What
src/brainlayer/hooks/indexer.py	RealtimeIndexer class: prompt capture, response pairing, dedup, chapters, stale cleanup
hooks/brainbar-stop-index.py	Stop hook: reads last_assistant_message, pairs with pending prompt, stores to production DB
hooks/brainbar-prompt-capture.py	UserPromptSubmit hook: captures user prompt to pending file
hooks/brainbar-postcompact.py	PostCompact hook: creates chapter boundary in chapters table
tests/test_realtime_hooks.py	14 tests covering all flows

Architecture

UserPromptSubmit → write prompt to ~/.brainlayer/pending/{session_id}.txt
Stop → read pending prompt + last_assistant_message → pair → INSERT OR IGNORE into chunks (source='realtime')
PostCompact → compact_summary → INSERT into chapters table

Test plan

• 14 unit tests pass (prompt capture, pairing, dedup, chapters, stale cleanup, queue fallback)
• Manual test against production DB (312K chunks) — chunk indexed with source='realtime'
• Fixed NOT NULL constraint on source_file (silent INSERT OR IGNORE failure)
• Live Claude Code session test (Stop hook fires automatically)
• Verify dedup during parallel batch+realtime operation

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Introduced real-time conversation indexing that automatically captures and stores prompts and responses.
  • Added session management with chapter boundaries for organizing conversation content.
  • Implemented content deduplication to avoid storing duplicate entries.
  • Added automatic fallback mechanism for data persistence when the database is unavailable.

• Tests

  • Added comprehensive test suite for real-time indexing functionality, including edge cases and error handling scenarios.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: b0d089d1-753b-4251-b0b8-5bb19ddfc096

📥 Commits

Reviewing files that changed from the base of the PR and between 6fc6fea and 2773ab5.

📒 Files selected for processing (6)

• hooks/brainbar-postcompact.py
• hooks/brainbar-prompt-capture.py
• hooks/brainbar-stop-index.py
• src/brainlayer/hooks/__init__.py
• src/brainlayer/hooks/indexer.py
• tests/test_realtime_hooks.py

---

📝 Walkthrough

Walkthrough

Introduces a comprehensive real-time indexing system for capturing user prompts, pairing them with assistant responses, and persisting indexed content to a SQLite database with automatic fallback to JSONL file queueing. Includes three hook scripts, a core RealtimeIndexer class, and test coverage for various indexing scenarios.

Changes

Cohort / File(s)	Summary
Hook Scripts hooks/brainbar-postcompact.py, hooks/brainbar-prompt-capture.py, hooks/brainbar-stop-index.py	Three standalone hook scripts that capture prompts, index responses paired with prompts, and record chapter boundaries. Each script reads JSON payloads from stdin, validates required fields, interacts with SQLite and/or the filesystem, and includes silent error handling to avoid blocking callers.
Core Indexer Module src/brainlayer/hooks/indexer.py	Introduces RealtimeIndexer class that orchestrates real-time indexing of user prompts and assistant responses, manages chapter boundaries, handles SQLite persistence with automatic JSONL fallback queueing, and supports stale prompt cleanup. Provides public methods for capturing prompts, indexing responses, recording chapters, and cleanup.
Test Suite tests/test_realtime_hooks.py	Comprehensive test coverage for RealtimeIndexer functionality, including prompt capture, response pairing, content deduplication, chapter boundaries, stale prompt cleanup, project extraction, and fallback queue behavior under various edge cases.

Sequence Diagram

sequenceDiagram
    participant User as User
    participant Prompt as Prompt Capture<br/>(Hook)
    participant Response as Response Index<br/>(Hook)
    participant Indexer as RealtimeIndexer
    participant SQLite as SQLite DB
    participant Queue as JSONL Queue
    
    User->>Prompt: Send prompt payload (session_id)
    Prompt->>Indexer: capture_prompt()
    Indexer->>Indexer: Store prompt in memory
    
    User->>Response: Send response payload (session_id, message)
    Response->>Indexer: index_response()
    Indexer->>Indexer: Retrieve pending prompt<br/>Pair with response<br/>Generate content hash & chunk_id
    
    alt DB Available
        Indexer->>SQLite: Insert chunk record
        SQLite-->>Indexer: Success
        Indexer->>Indexer: Clear pending prompt
    else DB Unavailable
        Indexer->>Queue: Write JSONL entry
        Queue-->>Indexer: Queued
    end
    
    Indexer-->>Response: Return chunk_id
    Response-->>User: Confirm indexing

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• feat: Phase 5 — search/store hardening #26: Adds hooks that persist data to BrainLayer SQLite DB with JSONL fallback queueing when DB is unavailable, implementing similar persistence patterns as this PR.

Poem

🐰 A hook here, a prompt there,
Responses paired with utmost care,
SQLite stores what we hold dear,
Queue steps in when DB's not near!
Chapters bound, dedup'd right—
BrainLayer's indexing burns so bright! ✨

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/realtime-indexing-hooks

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[greptile-apps]

Your free trial has ended. If you'd like to continue receiving code reviews, you can add a payment method here.

[EtanHey]

@coderabbitai review

[coderabbitai]

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.