Pipeline

Pipeline

DeepLore Enhanced uses a sequential pipeline to determine which vault entries get injected into each generation. This page explains every stage.

Pipeline Flow

onGenerate(chat)
  │
  ├─ ensureIndexFresh()              Refresh from Obsidian if cache expired
  │
  ├─ runPipeline(chat)               Core matching pipeline
  │    ├─ matchEntries(chat)           Stage 1: Keyword scan (broad pre-filter)
  │    │    ├─ buildScanText(chat)       Concatenate recent messages
  │    │    ├─ keyword matching          Check each entry's keys against scan text
  │    │    ├─ per-entry scanDepth       Override scan depth for specific entries
  │    │    ├─ warmup/probability/       Per-entry behavior checks
  │    │    │  cooldown checks
  │    │    ├─ cascade links             Pull in unconditionally linked entries
  │    │    ├─ recursive scanning        Scan matched entry content for more triggers
  │    │    └─ Active Character Boost    Auto-match active character's entry
  │    │
  │    ├─ buildCandidateManifest()     Build compact manifest from keyword matches
  │    │
  │    └─ aiSearch(chat, manifest)     Stage 2: AI selects best from candidates
  │         ├─ check cache               Reuse if chat+manifest hash matches
  │         ├─ build AI context          Recent chat + manifest + header
  │         ├─ call AI (profile/proxy)   Send to configured AI connection
  │         └─ parse response            Extract JSON array of selections
  │
  ├─ Re-injection cooldown            Skip entries injected within N generations
  │
  ├─ applyGating(entries)             Apply requires/excludes rules
  │    └─ iterative resolution          Cascade removals through dependencies
  │
  ├─ Strip duplicate injections       Skip entries injected in recent generations
  │
  ├─ formatAndGroup(entries)          Budget limits + injection grouping
  │    ├─ sort by priority              Lower number = higher priority
  │    ├─ apply entry limit             Max entries cap (if not unlimited)
  │    ├─ apply token budget            Max tokens cap (if not unlimited)
  │    ├─ apply injection template      Format with {{title}} and {{content}}
  │    └─ group by injection position   Separate groups for before/after/in_chat
  │
  ├─ setExtensionPrompt()            Inject each group into SillyTavern context
  │
  ├─ AI Notebook injection            Inject per-chat notebook (if enabled)
  │
  └─ Post-processing                  Update cooldowns, analytics, injection history

Three Pipeline Modes

Two-Stage (keywords → AI)

The default and recommended mode. Keywords run first to narrow down candidates, then the AI picks the best from those candidates.

Keywords match 20 entries  →  AI selects 8 most relevant  →  Inject 8 + constants

Error fallback: If AI fails, the keyword results are used directly. Empty fallback: If AI returns [], only constants are injected.

AI Only (full vault)

Skips keyword matching entirely. The entire vault manifest is sent to the AI.

Full vault (100 entries)  →  AI selects 10 most relevant  →  Inject 10 + constants

More thorough but uses more tokens per call. Best for vaults where keywords are sparse or unreliable.

Error fallback: If AI fails, the full vault is used sorted by priority.

Keywords Only (AI disabled)

When AI Search is disabled. Pure keyword matching.

Keywords match 12 entries  →  Inject 12 + constants

No API calls, no latency. Good for simple setups or when you want full control via keywords.

Stage Details

Index Refresh

Before matching, the pipeline checks if the cached vault index is stale (based on Cache TTL). If expired, it re-fetches all #lorebook entries directly from Obsidian's Local REST API, parses frontmatter, extracts wikilinks, and builds the VaultEntry[] array.

Keyword Matching

1. Build scan text: Concatenate the last N messages (Scan Depth setting, default 4)
2. Check each entry's keys: For each entry with keys, test if any keyword appears in the scan text
   • Respects Case Sensitive and Match Whole Words settings
   • Entries with per-entry scanDepth use their own message window
   • If refine_keys is set, at least one refine key must also match (AND filter)
3. Warmup check: If entry has warmup: N, count keyword occurrences; skip if below threshold
4. Probability check: If entry has probability: N (0.0-1.0), roll a random number; skip if roll exceeds probability
5. Cooldown check: If entry has per-entry cooldown and is in cooldown, skip it
6. Cascade links: If matched entries have cascade_links, the listed entries are unconditionally pulled in (no keyword check)
7. Recursive scanning: If enabled, scan matched entries' content for keywords that trigger more entries. Repeats up to Max Recursion Steps. Entries with excludeRecursion: true are skipped.
8. Active Character Boost: If enabled, auto-match the active character's entry by name/keyword even if not mentioned in chat
9. Constants: Entries tagged #lorebook-always or with constant: true are always included regardless of keywords
10. Bootstrap: If chat is below New Chat Threshold, #lorebook-bootstrap entries are force-included

AI Search

See AI Search for full details. In brief:

1. Build a compact manifest from keyword-matched candidates (or full vault in AI-only mode)
2. Send manifest + recent chat to the AI
3. AI returns a JSON array of selected entries with confidence and reasons
4. Selections replace keyword results (not merged)

Gating

After selection, entries with requires and excludes fields are evaluated:

• requires: ALL listed entry titles must be in the matched set, or this entry is removed
• excludes: If ANY listed entry title is in the matched set, this entry is removed
• Resolution is iterative. Removing one entry can cascade to remove others that require it.

Re-injection Cooldown

If the global Re-injection Cooldown setting is non-zero, entries that were injected within the last N generations are skipped. This is different from per-entry cooldown (which is checked during keyword matching) — re-injection cooldown is a global post-selection filter. Constants are exempt.

Injection Deduplication

If "Strip Duplicate Injections" is enabled, entries that were injected in recent generations (within the lookback depth) are stripped before formatting. Constants are exempt. Injection history is tracked per-chat in chat_metadata.deeplore_injection_log.

Budget & Formatting

1. Sort remaining entries by priority (lower number first)
2. Apply Max Entries cap (if Unlimited Entries is off)
3. Apply Token Budget cap (if Unlimited Token Budget is off). Entries are added in priority order until budget is reached. Note: The first entry always bypasses the budget check so that results are never empty — if the highest-priority entry exceeds the budget, it is still injected (a debug warning is logged when Debug Mode is on).
4. Format each entry using the Injection Template ({{title}} and {{content}} macros)
5. Group entries by their effective injection position (global default or per-entry override)

Injection

Each group is injected separately into SillyTavern via setExtensionPrompt():

• Before Main Prompt: Injected before the system prompt / story string
• After Main Prompt: Injected after the system prompt / story string
• In-chat: Injected as a message at the specified depth and role

If Allow World Info Scan is enabled, SillyTavern's built-in World Info system can scan the injected lore for additional WI keyword matches.

AI Notebook

After lorebook entries are injected, the AI Notebook is injected separately (if enabled). The notebook has its own injection position, depth, and role settings. It is independent of the lorebook pipeline — it always injects when enabled, regardless of entry matching.

Inspecting the Pipeline

Use the /dle-inspect slash command to see a detailed trace of the last generation:

• Which entries matched by keywords (and which keywords triggered them)
• Which entries were selected by AI (with confidence and reasons)
• Whether fallback was used
• The pipeline mode that was active

See Slash Commands for more details.