Skip to content

feat(SUP-1506): integrate JarVOS GBrain memory stack#27

Merged
levineam merged 23 commits into
mainfrom
sup-1505-combined-recall-eval
May 10, 2026
Merged

feat(SUP-1506): integrate JarVOS GBrain memory stack#27
levineam merged 23 commits into
mainfrom
sup-1505-combined-recall-eval

Conversation

@levineam

@levineam levineam commented May 10, 2026

Copy link
Copy Markdown
Owner

Summary

  • Reopens the closed stacked JarVOS GBrain work as a single mainline integration PR.
  • Carries the remaining SUP-1498/SUP-1499/SUP-1500/SUP-1501/SUP-1503/SUP-1504/SUP-1505 changes into main.
  • Includes graph-friendly GBrain pages, QMD comparison evals, GBrain graph recall, runtime recall bundling, combined recall scoring, and README/docs updates.

Stack disposition

Verification

  • Prior top-branch SUP-1506 preflight passed and is preserved at /Users/andrew/clawd/memory/projects/jarvos-gbrain/final-audit-preflight-2026-05-10.md.
  • Before closing SUP-1506, run the final mainline audit after this PR merges: node /Users/andrew/clawd/scripts/jarvos-memory-final-audit.js.

Paperclip

  • Parent: SUP-1425
  • Final audit gate: SUP-1506
  • Covered child issues: SUP-1498, SUP-1499, SUP-1500, SUP-1501, SUP-1503, SUP-1504, SUP-1505

Summary by CodeRabbit

  • New Features

    • Added graph-based recall functionality for structured knowledge retrieval
    • Introduced retrieval evaluation supporting comparison across multiple search engines
    • Added runtime recall bundle workflow with markdown output generation
    • New CLI commands for evaluation and graph recall with configurable options
  • Documentation

    • Expanded module documentation with recommended operating patterns and workflows
    • Added configuration guidance for search engine integration and graph settings
    • Documented retrieval validation steps before runtime prompt injection

Review Change Stack

@coderabbitai

coderabbitai Bot commented May 10, 2026

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

This PR extends @jarvos/gbrain with structured graph recall, multi-engine retrieval evaluation (gbrain/qmd/graph), QMD integration, and a unified recall bundle API. New public functions (graphRecall, recallBundle, renderRecallMarkdown) enable layered knowledge retrieval across Obsidian-sourced curated imports, full-vault QMD lookup, and typed graph-based relationships. Configuration now supports jarvos-paths fallback for monorepo integration, QMD binary/mode/filters, and retrieval timeouts. Tests cover path resolution, graph metadata rendering, multi-engine evaluation with timeout handling, and recall bundle orchestration. Documentation updates clarify the validated operating pattern across modules, architecture, and integration layers.

Changes

GBrain Graph Recall & Multi-Engine Evaluation

Layer / File(s) Summary
New Exports & Configuration Defaults
modules/jarvos-gbrain/src/index.js
Adds DEFAULT_QMD_BIN, DEFAULT_RETRIEVAL_TIMEOUT_MS, and new graphRecall, recallBundle, renderRecallMarkdown exports. Updates DEFAULT_VAULT_DIR default. Extends resolveConfig() to support jarvosPaths monorepo fallback and resolve QMD settings (bin/mode/collection/index) plus retrieval timeout.
Graph-Friendly Brain Page Rendering
modules/jarvos-gbrain/src/index.js
Adds graph metadata field extraction, graph-compatible YAML frontmatter generation, and "Graph Links" markdown section for typed relationships rendered from curated manifest items.
Enhanced Command Execution
modules/jarvos-gbrain/src/index.js
Updates runCommand() to support configurable process timeouts with SIGKILL, returning expanded result fields: timedOut, signal, structured stdout/stderr/error.
Evaluation & Recall Helpers
modules/jarvos-gbrain/src/index.js
Adds numeric parsing (positiveInteger), per-engine expected/query selection, QMD argument building, output clause evaluation, GBrain/QMD search with timeout, graph-query parsing, and graph/recall-specific entry configuration.
Graph Recall & Recall Bundle Orchestration
modules/jarvos-gbrain/src/index.js
Implements graphRecall() wrapping gbrain graph-query with seed/depth control and recallBundle() combining direct GBrain search, optional QMD lookup, and graph expansion with auto-seed discovery and markdown rendering.
Multi-Engine Retrieval Evaluation
modules/jarvos-gbrain/src/index.js
Refactors runRetrievalEval() to support conditional per-engine comparisons (gbrain/qmd/graph/recall) with timeout handling, per-engine query/expected/missing tracking, and expanded result summary.
Enhanced Health Checks
modules/jarvos-gbrain/src/index.js
Expands doctor() to validate vault/notes/brain directories and optionally check QMD binary, updating overall health assessment.
CLI Command Routing
modules/jarvos-gbrain/scripts/jarvos-gbrain.js
Adds argValues() helper for multi-flag collection, extends cliConfig() with QMD and recall/graph parameters, updates usage, wires eval/graph/recall commands with proper argument parsing and exit codes.
Package Scripts
modules/jarvos-gbrain/package.json
Adds eval and recall npm scripts calling jarvos-gbrain.js CLI.
Sample Evaluation Questions
modules/jarvos-gbrain/config/eval-questions.json
Populates template with concrete question covering gbrain/qmd/graph engine expectations and graph seed configuration.
Comprehensive Test Coverage
modules/jarvos-gbrain/test/gbrain.test.js
Adds test helpers for environment/cache isolation, resolveConfig tests for shared jarvOS paths and monorepo fallback, renderBrainPage graph metadata tests, extensive runRetrievalEval multi-engine/timeout tests, recallBundle and graphRecall functional tests, and updated doctor directory validation tests.
Module-Level API Documentation
modules/jarvos-gbrain/README.md
Documents new public exports, expanded Quick Start CLI examples for graph/recall, extended configuration with QMD/timeout options, graph-friendly manifest fields, and new Graph Recall/Recall Bundle sections with usage guidance.
Higher-Level Module Documentation
modules/README.md
Describes @jarvos/gbrain's expanded role in retrieval evals, graph sidecar recall, and runtime recall bundles. Adds quick-start eval/recall commands, documents operating loop, and updates config env-vars.
Architecture & Integration Guidance
docs/architecture/jarvos-architecture.md, README.md, runtimes/openclaw/README.md, starter-kit/README.md
Clarifies validated operating pattern across jarvOS layers (Obsidian → QMD → curated GBrain → graph recall → runtime), public/private boundaries, OpenClaw integration, and verification workflow steps.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

  • levineam/jarvOS#17: Both PRs extensively modify @jarvos/gbrain module sources, scripts, package.json, README, and tests with overlapping feature additions for graph/recall/eval capabilities.
  • levineam/jarvOS#8: This PR's jarvos-paths monorepo fallback and config path resolution tests directly depend on the shared path-resolution integration introduced in that PR.
  • levineam/jarvOS#19: Both PRs modify the same vault path resolution (DEFAULT_VAULT_DIR, resolveConfig with jarvosPaths, and vault/notes checks) and doctor health validation logic.

Suggested labels

jarvis-managed

Poem

🐰 A recall that bundles brains,
Graph seeds sprout through the veins,
QMD, GBrain, threads aligned—
Three engines in one mind.
The vault remembers, wikilinks bind,
Knowledge layered, truth we find! 🧠✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly and specifically addresses the primary feature: integrating the JarVOS GBrain memory stack, which is the core objective of consolidating graph-friendly pages, retrieval evaluation, and runtime recall bundling.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch sup-1505-combined-recall-eval
⚔️ Resolve merge conflicts
  • Resolve merge conflict in branch sup-1505-combined-recall-eval

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (2)
modules/jarvos-gbrain/src/index.js (2)

1084-1102: 💤 Low value

Document the intentional ok = recallResult.ok overwrite for compareRecall.

In compareRecall mode, the per-question ok is reset to recallResult.ok (then AND'd only with engines.qmd.ok / engines.gbrain_graph.ok when present), so a failing direct-search engine no longer fails the question. This is exercised and asserted by the "score combined runtime recall separately from direct search" test, but the ok = (vs ok = ok && …) is easy to mistake for a bug on first read. A short comment captures the design intent.

📝 Suggested comment
     if (compareRecall) {
       const recallQuery = typeof entry === 'object' ? queryForEngine(entry, 'recall', query) : query;
       const expectedCandidates = recallExpectedCandidates(entry);
       const recallResult = runRecallEval(
         config,
         entry,
         recallQuery,
         expectedCandidates,
         dryRun,
         limit,
         graphDepth,
         graphSeedLimit,
       );
       engines.gbrain_recall = recallResult;
       engineQueries.gbrain_recall = recallQuery;
+      // When recall scoring is enabled it becomes authoritative for the question's
+      // pass/fail; direct GBrain search status is reported per-engine but no longer
+      // fails the question on its own. QMD/graph engines (if enabled) still gate ok.
       ok = recallResult.ok;
       if (engines.qmd) ok = ok && engines.qmd.ok;
       if (engines.gbrain_graph) ok = ok && engines.gbrain_graph.ok;
     }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@modules/jarvos-gbrain/src/index.js` around lines 1084 - 1102, The assignment
ok = recallResult.ok in the compareRecall branch is intentional (it resets the
per-question success to the recall evaluation result rather than preserving
previous direct-search ok), so add a concise comment above that line explaining
the design intent: that in compareRecall mode the question's pass/fail is
determined by recallResult.ok and then only AND'd with engines.qmd.ok and
engines.gbrain_graph.ok when present; reference the variables runRecallEval,
recallResult, ok, engines.qmd and engines.gbrain_graph so reviewers know this
overwrite is deliberate and not a typo.

82-104: ⚡ Quick win

Wrap shared-paths getter calls in a guard against runtime exceptions.

loadJarvosPaths only catches resolution/require failures, but sharedPathOrFallback invokes jarvosPaths[getterName]() unguarded. If a future @jarvos/secondbrain build throws from getVaultDir()/getNotesDir() (e.g., on a misconfigured jarvOS workspace), that exception escapes from resolveConfig and breaks every public API that resolves config first (createImportPlan, syncBrain, recallBundle, doctor, etc.). A small try/catch keeps the documented "fall back to portable defaults" guarantee intact.

♻️ Proposed defensive call
 function sharedPathOrFallback(jarvosPaths, getterName, fallback) {
   if (jarvosPaths && typeof jarvosPaths[getterName] === 'function') {
-    return jarvosPaths[getterName]();
+    try {
+      const value = jarvosPaths[getterName]();
+      if (typeof value === 'string' && value.trim()) return value;
+    } catch {
+      // Fall back to portable defaults if the shared resolver throws.
+    }
   }
   return fallback;
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@modules/jarvos-gbrain/src/index.js` around lines 82 - 104,
sharedPathOrFallback currently calls jarvosPaths[getterName]() without guarding
runtime errors, so wrap that invocation in a try/catch: inside
sharedPathOrFallback, check the function exists as you do, then call it inside a
try block and return its result; on any exception catch it and return the
provided fallback (optionally log the error). This ensures callers like
resolveConfig (and downstream APIs createImportPlan, syncBrain, recallBundle,
doctor) keep the documented "fall back to portable defaults" behavior even if
getVaultDir/getNotesDir throw.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@modules/jarvos-gbrain/scripts/jarvos-gbrain.js`:
- Line 61: The usage string printed via process.stdout.write in jarvos-gbrain.js
is missing the supported recall flags; update the printed help text (the
multi-line string passed to process.stdout.write) so the recall command line
includes the missing options --max-chars and --dry-run (keeping existing flags
like --no-qmd, --no-graph, --graph-seed, --graph-depth, --graph-seed-limit,
--limit, --timeout-ms, --format). Ensure the recall line in that string is
updated so CLI help accurately reflects all supported recall options.
- Around line 24-29: argValues currently treats any next argv token as a value
even if it's another flag (e.g., "-x"), causing silent misparsing; update the
argValues(name) function to only push process.argv[index+1] when it exists and
does not look like a flag (e.g., does not match /^-/), so skip/ignore tokens
that start with '-' as repeated-arg values; reference the argValues function and
the process.argv[index+1] check to implement this guard.

---

Nitpick comments:
In `@modules/jarvos-gbrain/src/index.js`:
- Around line 1084-1102: The assignment ok = recallResult.ok in the
compareRecall branch is intentional (it resets the per-question success to the
recall evaluation result rather than preserving previous direct-search ok), so
add a concise comment above that line explaining the design intent: that in
compareRecall mode the question's pass/fail is determined by recallResult.ok and
then only AND'd with engines.qmd.ok and engines.gbrain_graph.ok when present;
reference the variables runRecallEval, recallResult, ok, engines.qmd and
engines.gbrain_graph so reviewers know this overwrite is deliberate and not a
typo.
- Around line 82-104: sharedPathOrFallback currently calls
jarvosPaths[getterName]() without guarding runtime errors, so wrap that
invocation in a try/catch: inside sharedPathOrFallback, check the function
exists as you do, then call it inside a try block and return its result; on any
exception catch it and return the provided fallback (optionally log the error).
This ensures callers like resolveConfig (and downstream APIs createImportPlan,
syncBrain, recallBundle, doctor) keep the documented "fall back to portable
defaults" behavior even if getVaultDir/getNotesDir throw.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bd79b5c8-32f5-49dd-80bc-d4528c2b512a

📥 Commits

Reviewing files that changed from the base of the PR and between 696078a and 13f6a3e.

📒 Files selected for processing (11)
  • README.md
  • docs/architecture/jarvos-architecture.md
  • modules/README.md
  • modules/jarvos-gbrain/README.md
  • modules/jarvos-gbrain/config/eval-questions.json
  • modules/jarvos-gbrain/package.json
  • modules/jarvos-gbrain/scripts/jarvos-gbrain.js
  • modules/jarvos-gbrain/src/index.js
  • modules/jarvos-gbrain/test/gbrain.test.js
  • runtimes/openclaw/README.md
  • starter-kit/README.md

Comment on lines +24 to +29
function argValues(name) {
const values = [];
for (let index = 0; index < process.argv.length; index += 1) {
if (process.argv[index] === name && process.argv[index + 1]) values.push(process.argv[index + 1]);
}
return values;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Reject flag tokens as repeated-arg values.

On Line 27, argValues() will treat another flag as a value if a value is missing, which causes silent misparsing.

Suggested fix
 function argValues(name) {
   const values = [];
   for (let index = 0; index < process.argv.length; index += 1) {
-    if (process.argv[index] === name && process.argv[index + 1]) values.push(process.argv[index + 1]);
+    if (process.argv[index] !== name) continue;
+    const value = process.argv[index + 1];
+    if (!value || value.startsWith('--')) {
+      throw new Error(`${name} requires a value`);
+    }
+    values.push(value);
   }
   return values;
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@modules/jarvos-gbrain/scripts/jarvos-gbrain.js` around lines 24 - 29,
argValues currently treats any next argv token as a value even if it's another
flag (e.g., "-x"), causing silent misparsing; update the argValues(name)
function to only push process.argv[index+1] when it exists and does not look
like a flag (e.g., does not match /^-/), so skip/ignore tokens that start with
'-' as repeated-arg values; reference the argValues function and the
process.argv[index+1] check to implement this guard.


function usage() {
process.stdout.write(`jarvos-gbrain\n\nCommands:\n plan [--manifest path]\n import [--dry-run] [--manifest path] [--brain-dir path] [--vault-dir path]\n sync [--dry-run] [--brain-dir path] [--gbrain-dir path]\n eval [--dry-run] [--eval-file path]\n doctor\n\n`);
process.stdout.write(`jarvos-gbrain\n\nCommands:\n plan [--manifest path]\n import [--dry-run] [--manifest path] [--brain-dir path] [--vault-dir path]\n sync [--dry-run] [--brain-dir path] [--gbrain-dir path]\n eval [--dry-run] [--eval-file path] [--compare-qmd] [--compare-graph] [--compare-recall]\n [--graph-depth n] [--graph-seed-limit n]\n [--limit n] [--timeout-ms n] [--qmd-bin path] [--qmd-mode search|query|vsearch]\n [--qmd-collection name] [--qmd-index name]\n graph [--dry-run] --seed slug [--seed slug] [--depth n] [--timeout-ms n]\n recall --query text [--no-qmd] [--no-graph] [--graph-seed slug] [--graph-depth n]\n [--graph-seed-limit n] [--limit n] [--timeout-ms n] [--format markdown]\n doctor\n\n`);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Keep usage() in sync with supported recall flags.

Line 61 omits supported recall options (--max-chars and --dry-run), so the CLI help is currently incomplete.

Suggested fix
-  recall --query text [--no-qmd] [--no-graph] [--graph-seed slug] [--graph-depth n]
-         [--graph-seed-limit n] [--limit n] [--timeout-ms n] [--format markdown]
+  recall [--dry-run] --query text [--no-qmd] [--no-graph] [--graph-seed slug]
+         [--graph-depth n] [--graph-seed-limit n] [--limit n] [--max-chars n]
+         [--timeout-ms n] [--format markdown]
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@modules/jarvos-gbrain/scripts/jarvos-gbrain.js` at line 61, The usage string
printed via process.stdout.write in jarvos-gbrain.js is missing the supported
recall flags; update the printed help text (the multi-line string passed to
process.stdout.write) so the recall command line includes the missing options
--max-chars and --dry-run (keeping existing flags like --no-qmd, --no-graph,
--graph-seed, --graph-depth, --graph-seed-limit, --limit, --timeout-ms,
--format). Ensure the recall line in that string is updated so CLI help
accurately reflects all supported recall options.

@levineam
levineam merged commit ea06679 into main May 10, 2026
6 checks passed
@levineam
levineam deleted the sup-1505-combined-recall-eval branch May 10, 2026 14:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant