Pipeline

Pipeline

DeepLore Enhanced uses a sequential pipeline to determine which vault entries get injected into each generation. This page explains every stage.

Pipeline Flow

onGenerate(chat)
  │
  ├─ ensureIndexFresh()              Refresh from Obsidian if cache expired
  │
  ├─ matchEntries(chat)              Stage 1: Keyword scan (broad pre-filter)
  │    ├─ buildScanText(chat, depth)   Concatenate recent messages
  │    ├─ keyword matching             Check each entry's keys against scan text
  │    ├─ per-entry scanDepth          Override scan depth for specific entries
  │    ├─ recursive scanning           Scan matched entry content for more triggers
  │    └─ Active Character Boost       Auto-match active character's entry
  │
  ├─ buildCandidateManifest()         Build compact manifest from keyword matches
  │
  ├─ aiSearch(chat, manifest)         Stage 2: AI selects best from candidates
  │    ├─ check cache                   Reuse if chat+manifest hash matches
  │    ├─ build AI context              Recent chat + manifest + header
  │    ├─ call AI (profile or proxy)    Send to configured AI connection
  │    └─ parse response                Extract JSON array of selections
  │
  ├─ applyGating(entries)             Apply requires/excludes rules
  │    └─ iterative resolution          Cascade removals through dependencies
  │
  ├─ formatAndGroup(entries)          Budget limits + injection grouping
  │    ├─ sort by priority              Lower number = higher priority
  │    ├─ apply entry limit             Max entries cap (if not unlimited)
  │    ├─ apply token budget            Max tokens cap (if not unlimited)
  │    ├─ apply injection template      Format with {{title}} and {{content}}
  │    └─ group by injection position   Separate groups for before/after/in_chat
  │
  └─ setExtensionPrompt()            Inject each group into SillyTavern context

Three Pipeline Modes

Two-Stage (keywords → AI)

The default and recommended mode. Keywords run first to narrow down candidates, then the AI picks the best from those candidates.

Keywords match 20 entries  →  AI selects 8 most relevant  →  Inject 8 + constants

Error fallback: If AI fails, the keyword results are used directly. Empty fallback: If AI returns [], only constants are injected.

AI Only (full vault)

Skips keyword matching entirely. The entire vault manifest is sent to the AI.

Full vault (100 entries)  →  AI selects 10 most relevant  →  Inject 10 + constants

More thorough but uses more tokens per call. Best for vaults where keywords are sparse or unreliable.

Error fallback: If AI fails, the full vault is used sorted by priority.

Keywords Only (AI disabled)

When AI Search is disabled. Pure keyword matching.

Keywords match 12 entries  →  Inject 12 + constants

No API calls, no latency. Good for simple setups or when you want full control via keywords.

Stage Details

Index Refresh

Before matching, the pipeline checks if the cached vault index is stale (based on Cache TTL). If expired, it re-fetches all #lorebook entries from Obsidian via the server plugin, parses frontmatter, extracts wikilinks, and builds the VaultEntry[] array.

Keyword Matching

1. Build scan text: Concatenate the last N messages (Scan Depth setting, default 4)
2. Check each entry's keys: For each entry with keys, test if any keyword appears in the scan text
   • Respects Case Sensitive and Match Whole Words settings
   • Entries with per-entry scanDepth use their own message window
   • If refine_keys is set, at least one refine key must also match (AND filter)
3. Warmup check: If entry has warmup: N, count keyword occurrences; skip if below threshold
4. Cooldown check: If entry is in cooldown, skip it
5. Re-injection cooldown: If entry was injected within N generations, skip it
6. Recursive scanning: If enabled, scan matched entries' content for keywords that trigger more entries. Repeats up to Max Recursion Steps. Entries with excludeRecursion: true are skipped.
7. Cascade links: If matched entries have cascade_links, the listed entries are unconditionally pulled in (no keyword check)
8. Active Character Boost: If enabled, auto-match the active character's entry by name/keyword even if not mentioned in chat
9. Constants: Entries tagged #lorebook-always or with constant: true are always included regardless of keywords
10. Bootstrap: If chat is below New Chat Threshold, #lorebook-bootstrap entries are force-included

AI Search

See AI Search for full details. In brief:

1. Build a compact manifest from keyword-matched candidates (or full vault in AI-only mode)
2. Send manifest + recent chat to the AI
3. AI returns a JSON array of selected entries with confidence and reasons
4. Selections replace keyword results (not merged)

Gating

After selection, entries with requires and excludes fields are evaluated:

• requires: ALL listed entry titles must be in the matched set, or this entry is removed
• excludes: If ANY listed entry title is in the matched set, this entry is removed
• Resolution is iterative. Removing one entry can cascade to remove others that require it.

Budget & Formatting

1. Sort remaining entries by priority (lower number first)
2. Apply Max Entries cap (if Unlimited Entries is off)
3. Apply Token Budget cap (if Unlimited Token Budget is off). Entries are added in priority order until budget is reached.
4. Format each entry using the Injection Template ({{title}} and {{content}} macros)
5. Group entries by their effective injection position (global default or per-entry override)

Injection

Each group is injected separately into SillyTavern via setExtensionPrompt():

• Before Main Prompt: Injected before the system prompt / story string
• After Main Prompt: Injected after the system prompt / story string
• In-chat: Injected as a message at the specified depth and role

If Allow World Info Scan is enabled, SillyTavern's built-in World Info system can scan the injected lore for additional WI keyword matches.

Inspecting the Pipeline

Use the /dle-inspect slash command to see a detailed trace of the last generation:

• Which entries matched by keywords (and which keywords triggered them)
• Which entries were selected by AI (with confidence and reasons)
• Whether fallback was used
• The pipeline mode that was active

See Slash Commands for more details.