fix(embeddings): silent NULL embeddings after marketplace upgrades

[efenocchi]

Summary

Embeddings have been silently failing in production. The capture hook
spawns the embed daemon from the marketplace plugin bundle path
(~/.claude/plugins/cache/hivemind/hivemind/<version>/bundle/embeddings/embed-daemon.js),
but @huggingface/transformers is installed only at
~/.hivemind/embed-deps/node_modules/ — reachable from the bundle via
a one-time symlink that hivemind embeddings install creates. Every
Claude Code marketplace auto-upgrade drops a new versioned cache dir
without the symlink, so the bundle daemon's
import("@huggingface/transformers") returns MODULE_NOT_FOUND,
embed() returns null, and sessions.message_embedding writes through
as NULL forever after. The user-facing log shows ~30 minutes of
embed err: Cannot find package lines.

This PR is a fix-it-forever overhaul of the embed daemon lifecycle:

1. Explicit transformers resolver in the daemon. nomic.ts now
   loads transformers via
   createRequire(pathToFileURL("~/.hivemind/embed-deps/")).resolve()
   followed by an absolute-URL dynamic import, with a bare-specifier
   fallback for dev trees. The daemon resolves transformers regardless
   of which bundle path it was spawned from.

2. Persistent opt-in via ~/.deeplake/config.json. The legacy
   HIVEMIND_EMBEDDINGS env var is read ONCE on first run to seed
   embeddings.enabled, then never consulted again. Source of truth
   is now a real file shared across all 4 agents.

3. New CLI surface. hivemind embeddings install is the heavy
   path (deps + symlinks + enabled:true). enable / disable are
   light config flips. uninstall [--prune] is the heavy counterpart
   (removes symlinks + optionally wipes shared deps). Every agent's
   SessionStart injection advertises the full surface.

4. Hello handshake + stuck-daemon recycle. The socket is per-UID,
   not per-version, so one stuck daemon poisons every subsequent
   session for up to 10 minutes of idle-out. The client now sends a
   hello on first connect; on path mismatch (typical post-upgrade)
   it SIGTERMs the daemon and clears its sock/pid. Same recycle fires
   if the daemon returns a transformers-missing error during
   embed(). Next session spawns fresh from the current bundle.

5. Visible one-time notification. On transformers-missing, the
   client enqueues a warn-severity notification routed through the
   existing SessionStart drain so the user sees a clear "run
   hivemind embeddings install" message. Suppressed when the user
   has explicitly disabled embeddings — no nag for opt-outs.

6. Self-heal symlink across cached versions. Each capture hook
   runs ensurePluginNodeModulesLink({ bundleDir }) at top-level.
   The first capture under a new plugin version (post-marketplace-
   upgrade) creates <pluginDir>/node_modules → ~/.hivemind/embed-deps/node_modules
   atomically; subsequent calls are O(1) no-ops. Conservative — never
   clobbers a real node_modules, never overrides a symlink to a
   different valid target, but cleans up dangling links so it can
   recreate them on the next call. Refuses to act outside the
   production bundle/ layout (guard against tests).

7. Bundle-scan guards. New tests/claude-code/embeddings-bundle-scan.test.ts
   reads each agent's shipped embed-daemon.js, capture.js, and
   the CLI bundle, and asserts every literal string the runtime
   depends on survives bundling. 30-second reviewer guardrail.

Behavioral migration

Was	Now
HIVEMIND_EMBEDDINGS=false env var	~/.deeplake/config.json → embeddings.enabled: false
EmbeddingsStatus = "env-disabled"	EmbeddingsStatus = "user-disabled"
enableEmbeddings() (heavy)	installEmbeddings()
disableEmbeddings({ prune }) (heavy)	uninstallEmbeddings({ prune })
(n/a)	enableEmbeddings() (light) — flip config flag, warn if deps missing
(n/a)	disableEmbeddings() (light) — flip flag + SIGTERM daemon
First-run migration	env-var read exactly once: false/unset → enabled: false; else enabled: true

Most users have the env var unset, so on first run after this lands
they'll get enabled: false and need to run hivemind embeddings enable
or install to opt back in. That's intentional — explicit consent
instead of implicit on-by-default.

Test plan

• npm test → 2457 passing (up from ~2371 on main).
• tsc --noEmit clean.
• Bundle scan asserts every agent's shipped artifact contains the
  fix.
• Self-heal helper handles every fs combination (missing target,
  existing real dir, existing symlink-elsewhere, dangling symlink,
  non-bundle layout).
• Manual sandbox capture test (per CLAUDE.md "never touch prod
  tables"):
  • Switch to test org:
    node codex/bundle/commands/auth-login.js org switch test_plugin
  • Point sessionsTableName at sessions_test in src/config.ts,
    rebuild, reinstall plugin.
  • Kill any pre-existing daemon:
    kill $(cat /tmp/hivemind-embed-$(id -u).pid 2>/dev/null) ; rm -f /tmp/hivemind-embed-$(id -u).{sock,pid}
  • Drive a real Claude Code session for a few prompts.
  • SELECT COUNT(*) AS total, COUNT(message_embedding) AS with_embed FROM sessions_test WHERE creation_date > NOW() - INTERVAL '5 minutes';
  • Expect with_embed == total.
  • Revert src/config.ts before merging.
• Manual disable/enable path:
  • hivemind embeddings disable → next capture writes NULL, no
    daemon spawn, no notification.
  • hivemind embeddings enable → daemon spawns, embedding column
    populated.
• Manual self-heal:
  • rm <marketplace-cache>/<version>/node_modules
  • Next capture event → symlink recreated.

Out of scope (filed separately)

• 403 retries from the Deeplake SQL endpoint visible in
  ~/.deeplake/hook-debug.log. Different auth-layer issue,
  orthogonal to embeddings.

Summary by CodeRabbit

• New Features

  • Added hivemind embeddings install to manage transformer dependencies separately from enablement
  • Added hivemind embeddings uninstall [--prune] for full cleanup with optional dependency pruning
  • Embeddings configuration now persists in ~/.deeplake/config.json for cross-session preference preservation
  • Daemon verification on startup to detect stale bundles and auto-recovery
  • User notifications when required dependencies are missing

• Bug Fixes

  • Improved handling of stuck or incompatible embedding daemons with automatic recycling
  • Better error detection and messaging for missing transformer dependencies

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 12dc047f-564b-45cf-a7ef-e63f94da6a5e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds a hello handshake to embeddings protocol, client verification and recycle, user-config–backed enablement with env migration, transformers shared-deps resolution, CLI install/enable/disable/uninstall/status, plugin node_modules self-heal, and broad bundle/help/test updates.

Changes

Embeddings lifecycle and CLI split

Layer / File(s)	Summary
Protocol hello and daemon identity src/embeddings/protocol.ts, src/embeddings/daemon.ts, */bundle/embeddings/embed-daemon.js	Adds hello request/response with daemonPath/pid/protocolVersion; daemons implement hello and track daemonPath.
Client verify/recycle and transformers-missing handling src/embeddings/client.ts, `/bundle/(capture	pre-tool-use
User config, migration, and status gating src/user-config.ts, src/embeddings/disable.ts, tests	Persists embeddings.enabled in ~/.deeplake/config.json with one-time env migration; status uses user-config; tests updated.
Transformers shared-deps importer src/embeddings/nomic.ts, */bundle/embeddings/embed-daemon.js	Importer prefers canonical shared-deps and falls back to bare import; Nomic uses importer; actionable errors.
CLI commands and routing src/cli/embeddings.ts, src/cli/index.ts, bundle/cli.js	Adds install/enable/disable/uninstall/status; install is heavy; enable/disable light; uninstall optional prune; status reflects config.
Self-heal and hook wiring src/embeddings/self-heal.ts, src/hooks/*/capture.ts, bundles	Adds atomic symlink self-heal for plugin node_modules; hooks perform best-effort self-heal when embeddings enabled.

Sequence Diagram(s)

sequenceDiagram
  participant CLI
  participant UserConfig
  participant Daemon
  participant Client
  CLI->>UserConfig: set embeddings.enabled = true/false
  Client->>Daemon: hello {id}
  Daemon-->>Client: {daemonPath, pid, protocolVersion}
  Client->>Daemon: embed {text}
  Daemon-->>Client: {embedding|error}
  Client-->>UserConfig: enqueue notification (deps missing) [when enabled]

Loading

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120+ minutes

Possibly related PRs

• activeloopai/hivemind#96: Introduces the notifications queue consumed here for embeddings missing-deps warnings.

Suggested reviewers

• kaghni

Poem

A rabbit taps the daemon’s door—hello! what path are you?
If wires cross, it sweeps the burrow, grows the links anew.
It files a note in Deeplake’s nook, “enable if you wish,”
And when transformers hide away, it serves a gentle dish:
“Run embeddings install,” it squeaks—then hops to prune or swish. 🐇

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/embed-daemon-transformers-resolution

[github-actions]

Coverage Report

Scope: files changed in this PR. Enforced threshold: 90% per metric (per file via vitest.config.ts).

Status	Category	Percentage	Covered / Total
🟢	Lines	92.24% (🎯 90%)	1402 / 1520
🟢	Statements	90.82% (🎯 90%)	1633 / 1798
🟢	Functions	91.59% (🎯 90%)	207 / 226
🔴	Branches	83.47% (🎯 90%)	843 / 1010

File Coverage — 19 files changed

File	Stmts	Branches	Functions	Lines
src/cli/embeddings.ts	🔴 49.0%	🔴 50.8%	🔴 75.0%	🔴 48.1%
src/cli/index.ts	🟢 97.4%	🟢 95.9%	🟢 100.0%	🟢 97.3%
src/embeddings/client.ts	🟢 95.2%	🔴 83.3%	🟢 93.5%	🟢 96.6%
src/embeddings/daemon.ts	🟢 92.9%	🔴 75.0%	🔴 78.9%	🟢 97.1%
src/embeddings/disable.ts	🟢 100.0%	🟢 100.0%	🟢 100.0%	🟢 100.0%
src/embeddings/nomic.ts	🟢 97.3%	🟢 94.6%	🟢 100.0%	🟢 100.0%
src/embeddings/protocol.ts	🟢 100.0%	🟢 100.0%	🟢 100.0%	🟢 100.0%
src/embeddings/self-heal.ts	🟢 91.9%	🔴 66.7%	🟢 100.0%	🟢 91.2%
src/hooks/capture.ts	🟢 98.7%	🔴 86.4%	🟢 100.0%	🟢 100.0%
src/hooks/codex/capture.ts	🟢 100.0%	🔴 86.8%	🟢 100.0%	🟢 100.0%
src/hooks/cursor/capture.ts	🟢 94.0%	🟢 90.0%	🟢 100.0%	🟢 95.8%
src/hooks/cursor/session-start.ts	🟢 100.0%	🟢 95.0%	🟢 100.0%	🟢 100.0%
src/hooks/hermes/capture.ts	🔴 86.5%	🔴 83.6%	🟢 100.0%	🔴 86.5%
src/hooks/hermes/session-start.ts	🟢 100.0%	🟢 93.8%	🟢 100.0%	🟢 100.0%
src/hooks/session-start-setup.ts	🟢 100.0%	🟢 95.0%	🟢 100.0%	🟢 100.0%
src/hooks/session-start.ts	🟢 100.0%	🟢 90.5%	🟢 100.0%	🟢 100.0%
src/notifications/queue.ts	🟢 97.2%	🔴 79.2%	🟢 100.0%	🟢 98.4%
src/shell/deeplake-fs.ts	🔴 89.7%	🔴 81.4%	🔴 86.4%	🟢 93.9%
src/user-config.ts	🟢 95.2%	🔴 74.3%	🟢 90.9%	🟢 94.8%

_{Generated for commit ea0af71.}

[coderabbitai]

Actionable comments posted: 17

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

tests/cursor/cursor-capture-hook.test.ts (1)

272-289: ⚠️ Potential issue | 🟡 Minor | ⚡ Quick win

Missing cleanup: temp directory leaks on every test run.

The test creates a temp directory and config file but never removes them. This will accumulate garbage in the system temp dir over repeated test runs.

🧹 Proposed fix to clean up temp resources

  it("user-disabled embeddings short-circuit to NULL without invoking EmbedClient", async () => {
    stdinMock.mockResolvedValue({
      conversation_id: "sid-emb-3",
      hook_event_name: "beforeSubmitPrompt",
      prompt: "disabled",
    });
    // Point user-config at a throwaway path that says enabled:false.
    const { writeFileSync, mkdtempSync } = await import("node:fs");
+   const { rmSync } = await import("node:fs");
    const { tmpdir } = await import("node:os");
    const { join } = await import("node:path");
    const dir = mkdtempSync(join(tmpdir(), "cursor-cap-disabled-"));
    const cfgPath = join(dir, "config.json");
    writeFileSync(cfgPath, JSON.stringify({ embeddings: { enabled: false } }), "utf-8");
-   await runHook({ HIVEMIND_CONFIG_PATH: cfgPath });
+   try {
+     await runHook({ HIVEMIND_CONFIG_PATH: cfgPath });
+   } finally {
+     rmSync(dir, { recursive: true, force: true });
+   }
    const sql = queryMock.mock.calls[0][0] as string;
    expect(sql).toContain("'::jsonb, NULL,");
    expect(sql).toMatch(/, message_embedding,/);
  });

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/cursor/cursor-capture-hook.test.ts` around lines 272 - 289, The test
creates a temp directory (mkdtempSync -> dir) and config file (cfgPath) but
never removes them; update the test around runHook({ HIVEMIND_CONFIG_PATH:
cfgPath }) in cursor-capture-hook.test.ts to ensure the temp directory is
cleaned (use fs.rmSync or fs.promises.rm with recursive/force) in a finally
block or after/teardown so the temp dir and config file created by writeFileSync
are removed regardless of test outcome; reference mkdtempSync, writeFileSync,
cfgPath, dir and runHook when implementing the cleanup.

🧹 Nitpick comments (1)

hermes/bundle/shell/deeplake-shell.js (1)

67811-67823: 💤 Low value

deepMerge performs only one level of nested merging, not recursive.

The function merges only one level deep: nested objects are spread ({...baseVal, ...patchVal}) rather than recursively merged. For the current embeddings.enabled use case this is fine, but deeper structures like {a: {b: {c: 1}}} patched with {a: {b: {d: 2}}} would lose c.

Consider renaming to shallowMergeNested or adding a comment clarifying the single-level constraint if deeper configs are anticipated.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hermes/bundle/shell/deeplake-shell.js` around lines 67811 - 67823, The
function deepMerge only merges nested objects one level deep (it spreads baseVal
and patchVal rather than recursively merging), so rename deepMerge to
shallowMergeNested (or equivalent) and add a concise JSDoc/comment on the
function explaining it intentionally performs only a single-level nested merge;
update any callers referencing deepMerge to use the new name (search for
deepMerge in the bundle) so callers and future maintainers understand the
single-level constraint.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@bundle/cli.js`:
- Around line 3825-3837: The killEmbedDaemon function currently reads a PID from
pidPathFor and sends SIGTERM blindly; instead verify the PID refers to the embed
daemon before signaling: after reading pid (readFileSync8) and before calling
process.kill, check the live process identity (e.g., inspect /proc/<pid>/cmdline
or use a small handshake via socketPathFor to confirm it responds as the embed
daemon) and only signal when that validation succeeds; if validation fails treat
the pidfile/socket as stale and remove pidPath/socketPath rather than killing an
unrelated process.
- Around line 3774-3777: The linkAgent function currently always calls
symlinkForce(join11(install.pluginDir, "node_modules")) which unlinks any
preexisting path and causes EISDIR if an agent already has a real node_modules
directory; before calling symlinkForce, check the fs.lstat/lstatSync on the
target link (the join11(..., "node_modules") path): if it exists and is a real
directory (stats.isDirectory() && !stats.isSymbolicLink()) rename or move it
(e.g., append ".own" or similar) and log that we preserved the agent's own
node_modules, otherwise if it’s a symlink or file remove it and then call
symlinkForce(SHARED_NODE_MODULES, link); update linkAgent to perform this guard
using the same link variable so symlinkForce and SHARED_NODE_MODULES usage
remain unchanged.

In `@claude-code/bundle/capture.js`:
- Around line 1766-1785: The current branch that handles a dangling symlink
(when linkStat.isSymbolicLink() is true) removes the stale symlink with
rmSync(link) and returns without recreating it, leaving self-heal incomplete;
change the catch block inside the dangling-link path so that after successfully
rmSync(link) you immediately recreate the symlink pointing at target using
symlinkSync(target, link[, type]) (choose 'junction' for Windows if needed), and
then return a success kind (e.g., "recreated-link" with link and target) instead
of "stale-link-removed"; update any callers/consumers expecting
"stale-link-removed" to handle the new return kind if necessary.

In `@claude-code/bundle/pre-tool-use.js`:
- Around line 1316-1343: In verifyDaemonOnce, don't set this.helloVerified at
the top; instead only set it after confirming a compatible hello response — move
the assignment to after you've confirmed hello.daemonPath exists and equals
this.daemonEntry (i.e., after the equality check where currently you return on
match), so that failed probes, missing daemonPath, or path mismatches (and
subsequent recycleDaemon calls) do not permanently mark the client as verified;
keep existing behavior for early returns when !this.daemonEntry or on exceptions
by not setting helloVerified in those code paths.

In `@claude-code/bundle/shell/deeplake-shell.js`:
- Around line 68288-68292: resolveEmbedDaemonPath currently builds the daemon
path as join(dirname(fileURLToPath(import.meta.url)), "embeddings",
"embed-daemon.js") which points to bundle/shell/embeddings/embed-daemon.js but
the actual daemon lives in the sibling bundle/embeddings directory; update
resolveEmbedDaemonPath (same function called when spawning EmbedClient) to
include the parent-traversal segment (add ".." into the join parts, mirroring
the pattern used by resolveGrepEmbedDaemonPath) so the join becomes dirname ->
".." -> "embeddings" -> "embed-daemon.js".

In `@codex/bundle/shell/deeplake-shell.js`:
- Around line 68290-68292: resolveEmbedDaemonPath currently constructs the path
"embeddings/embed-daemon.js" relative to the shell bundle directory, which
resolves to codex/bundle/shell/embeddings/embed-daemon.js but the actual bundled
file is at codex/bundle/embeddings/embed-daemon.js; update the path construction
in resolveEmbedDaemonPath so it points to the parent
"embeddings/embed-daemon.js" (i.e., join the dirname of import.meta.url with
".." and "embeddings" then "embed-daemon.js") so daemonEntry references the
correct bundled file.

In `@codex/bundle/stop.js`:
- Around line 1184-1187: enqueueNotification currently always appends the
notification which allows duplicates across processes; change it to honor
n.dedupKey by reading the existing queue via readQueue(), checking if any queued
item has the same dedupKey (and skip push if found), and only push/writeQueue
when dedupKey is absent or unique; reference the enqueueNotification function
and use readQueue/writeQueue and the notification object's dedupKey field to
implement the guard so duplicate "embed-deps-missing" entries are not persisted.

In `@cursor/bundle/shell/deeplake-shell.js`:
- Around line 67910-67918: The embed() flow can send the request on a stale
socket because verifyDaemonOnce() may recycle the daemon; fix by making
verifyDaemonOnce() indicate when it recycled (e.g., return a boolean or the
fresh socket) and then, inside embed(), re-acquire the current socket before
calling sendAndWait() so the request is sent to the new daemon; update embed()
(and the similar block around the 67942-67968 range) to check the return from
verifyDaemonOnce(), refresh/get the socket (the same socket variable passed to
sendAndWait) when a recycle occurred, and only then construct/send the req via
this.sendAndWait().
- Around line 68182-68184: The current isTransformersMissingError function is
too broad because it matches any "MODULE_NOT_FOUND"; change it to only return
true when the missing-module error specifically references transformers-related
packages. Update isTransformersMissingError to check err.code ===
'MODULE_NOT_FOUND' (or the error string) AND test err.message/err for package
names like `@huggingface/transformers`, transformers, hivemind, or accelerate
(e.g. /(`@huggingface`\/transformers|transformers|hivemind|accelerate)/i) so
unrelated missing-module errors are not misclassified.
- Around line 68290-68292: The resolveEmbedDaemonPath function currently builds
a path to "embeddings/embed-daemon.js" relative to the shell bundle directory,
but the emitted daemon actually lives in the parent bundle's embeddings folder;
update resolveEmbedDaemonPath (and the join11/dirname5/fileURLToPath usage) to
ascend one directory before "embeddings" (e.g.
join11(dirname5(fileURLToPath(import.meta.url)), "..", "embeddings",
"embed-daemon.js")) so the resolved path points to
bundle/embeddings/embed-daemon.js instead of shell/embeddings/embed-daemon.js.

In `@hermes/bundle/capture.js`:
- Around line 614-618: enqueueNotification currently always appends
notifications to the persisted queue so cross-process runs re-enqueue “one-time”
warnings; change enqueueNotification to read the queued items (via readQueue),
check for an existing notification with the same unique id/type (e.g.
"embed-deps-missing") or a oneTime flag before pushing, and only push if not
already present, then writeQueue; use the same identifying key the runtime uses
for suppression (the embed-deps-missing id or a new oneTime property) so
_signalledMissingDeps in-memory suppression remains complementary to the
persisted dedupe.
- Around line 1099-1118: The stale-symlink branch currently removes the dangling
link and returns { kind: "stale-link-removed" }, which prevents the caller from
getting a usable node_modules link; instead, after successfully rmSync(link) in
the catch block you should not return immediately but fall through to the same
symlink-creation path used when the link does not exist. Concretely: in the
block under if (linkStat.isSymbolicLink()) where you catch a failed statSync,
remove the early return ({ kind: "stale-link-removed" }); after rmSync(link)
continue execution so the subsequent code that creates the symlink (the same
logic that runs when no link exists) runs and returns the successful "linked"
(or existing success) result; keep use of readlinkSync, statSync, rmSync and the
existing result kinds consistent.

In `@hermes/bundle/wiki-worker.js`:
- Around line 650-651: The current isTransformersMissingError detector is too
broad because it matches any "MODULE_NOT_FOUND"; update
isTransformersMissingError to only treat errors as transformer-missing when the
message explicitly references transformer-related identifiers (for example
'@huggingface/transformers', the word 'transformers' near the MODULE_NOT_FOUND
text, or the specific install hint 'hivemind embeddings install'), rather than
matching bare MODULE_NOT_FOUND; locate the isTransformersMissingError function
and tighten the regex/logic to require transformer-specific context (or prefer
checking for a transformer-specific sentinel error code returned by the daemon
if available) so unrelated packaging/runtime failures are not misclassified.

In `@src/embeddings/client.ts`:
- Around line 128-214: Add focused unit tests for EmbedClient to hit the
conditional branches in verifyDaemonOnce, handleTransformersMissing, and
recycleDaemon: mock sendAndWait to return a HelloResponse with no daemonPath to
assert no recycle; mock sendAndWait to return a mismatched daemonPath and assert
recycleDaemon called once and _recycledStuckDaemon becomes true (and calling
verifyDaemonOnce again does not re-trigger recycle); for
handleTransformersMissing stub embeddingsStatus to return "user-disabled" and
verify enqueueNotification is not called, and stub it to return enabled and
verify enqueueNotification is called once and _signalledMissingDeps is set; for
recycleDaemon mock readFileSync to supply a pid, spy process.kill and unlinkSync
to verify pid kill + socket/pid unlink path and also test the null reportedPid
branch where pidfile is missing. Reset module-level flags (_recycledStuckDaemon,
_signalledMissingDeps) between tests and use spies/mocks for sendAndWait,
recycleDaemon, enqueueNotification, embeddingsStatus, readFileSync, unlinkSync,
and process.kill to assert expected side effects.

In `@tests/claude-code/session-start-setup-hook.test.ts`:
- Around line 230-235: This test sets process.env.EMBEDDINGS_DISABLED_FOR_TEST
but never clears it, causing later tests to inherit the disabled state; after
calling runHook(...) in the "skips warmup..." test, restore or delete
process.env.EMBEDDINGS_DISABLED_FOR_TEST (e.g., delete
process.env.EMBEDDINGS_DISABLED_FOR_TEST or set it back to its original value)
so subsequent calls to runHook() behave deterministically; update the test
around runHook, embedWarmupMock, and debugLogMock assertions to ensure the env
key is cleaned up before the test ends.

In `@tests/hermes/hermes-capture-hook.test.ts`:
- Around line 275-281: Test mutates HIVEMIND_CONFIG_PATH and leaves a temp dir;
wrap the runHook call in a try/finally that saves the original
process.env.HIVEMIND_CONFIG_PATH, sets it to cfgPath, and in finally restores
the original env value (or deletes the key if it was undefined) and removes the
temp directory (use fs.rmSync(dir, { recursive: true, force: true }) or
fs.rmdirSync for compatibility). Locate the code around runHook, cfgPath,
mkdtempSync, writeFileSync and update the test to import/removal functions and
perform cleanup to avoid leaking state between tests.

---

Outside diff comments:
In `@tests/cursor/cursor-capture-hook.test.ts`:
- Around line 272-289: The test creates a temp directory (mkdtempSync -> dir)
and config file (cfgPath) but never removes them; update the test around
runHook({ HIVEMIND_CONFIG_PATH: cfgPath }) in cursor-capture-hook.test.ts to
ensure the temp directory is cleaned (use fs.rmSync or fs.promises.rm with
recursive/force) in a finally block or after/teardown so the temp dir and config
file created by writeFileSync are removed regardless of test outcome; reference
mkdtempSync, writeFileSync, cfgPath, dir and runHook when implementing the
cleanup.

---

Nitpick comments:
In `@hermes/bundle/shell/deeplake-shell.js`:
- Around line 67811-67823: The function deepMerge only merges nested objects one
level deep (it spreads baseVal and patchVal rather than recursively merging), so
rename deepMerge to shallowMergeNested (or equivalent) and add a concise
JSDoc/comment on the function explaining it intentionally performs only a
single-level nested merge; update any callers referencing deepMerge to use the
new name (search for deepMerge in the bundle) so callers and future maintainers
understand the single-level constraint.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c577da79-afa6-4d82-9398-f8195e33ee25

📥 Commits

Reviewing files that changed from the base of the PR and between bdffdf0 and ed4b564.

📒 Files selected for processing (61)

• bundle/cli.js
• claude-code/bundle/capture.js
• claude-code/bundle/embeddings/embed-daemon.js
• claude-code/bundle/pre-tool-use.js
• claude-code/bundle/session-start-setup.js
• claude-code/bundle/session-start.js
• claude-code/bundle/shell/deeplake-shell.js
• claude-code/bundle/wiki-worker.js
• codex/bundle/capture.js
• codex/bundle/embeddings/embed-daemon.js
• codex/bundle/pre-tool-use.js
• codex/bundle/session-start.js
• codex/bundle/shell/deeplake-shell.js
• codex/bundle/stop.js
• codex/bundle/wiki-worker.js
• cursor/bundle/capture.js
• cursor/bundle/embeddings/embed-daemon.js
• cursor/bundle/pre-tool-use.js
• cursor/bundle/session-start.js
• cursor/bundle/shell/deeplake-shell.js
• cursor/bundle/wiki-worker.js
• embeddings/embed-daemon.js
• hermes/bundle/capture.js
• hermes/bundle/embeddings/embed-daemon.js
• hermes/bundle/pre-tool-use.js
• hermes/bundle/session-start.js
• hermes/bundle/shell/deeplake-shell.js
• hermes/bundle/wiki-worker.js
• pi/bundle/wiki-worker.js
• src/cli/embeddings.ts
• src/cli/index.ts
• src/embeddings/client.ts
• src/embeddings/daemon.ts
• src/embeddings/disable.ts
• src/embeddings/nomic.ts
• src/embeddings/protocol.ts
• src/embeddings/self-heal.ts
• src/hooks/capture.ts
• src/hooks/codex/capture.ts
• src/hooks/codex/session-start.ts
• src/hooks/cursor/capture.ts
• src/hooks/cursor/session-start.ts
• src/hooks/hermes/capture.ts
• src/hooks/hermes/session-start.ts
• src/hooks/session-start-setup.ts
• src/hooks/session-start.ts
• src/user-config.ts
• tests/claude-code/embeddings-bundle-scan.test.ts
• tests/claude-code/embeddings-client.test.ts
• tests/claude-code/embeddings-disable.test.ts
• tests/claude-code/embeddings-nomic.test.ts
• tests/claude-code/embeddings-self-heal.test.ts
• tests/claude-code/session-start-setup-hook.test.ts
• tests/claude-code/user-config.test.ts
• tests/claude-code/wiki-worker-plugin-version.test.ts
• tests/cli/cli-embeddings.test.ts
• tests/cli/cli-index.test.ts
• tests/cursor/cursor-capture-hook.test.ts
• tests/hermes/hermes-capture-hook.test.ts
• tests/test-setup.ts
• vitest.config.ts

[kaghni]

@coderabbitai what hello stands for, here, why is hardcoded?

[coderabbitai]

@kaghni — "hello" here is the wire-protocol operation name for a new IPC handshake message, not an arbitrary magic string. It plays the same role as "ping" and "embed" in the daemon's dispatch switch.

Here's what it does end-to-end:

sequenceDiagram
    participant C as EmbedClient
    participant D as EmbedDaemon

    C->>D: { op: "hello", id: "..." }
    D-->>C: { id, daemonPath, pid, protocolVersion }
    Note over C: Does daemonPath match my daemonEntry?
    alt mismatch (stale daemon from old bundle)
        C->>D: SIGTERM
        C->>C: clear sock/pid → spawn fresh daemon
    else match
        C->>C: proceed with embed requests
    end

Loading

Why it exists: After a marketplace plugin auto-upgrade, a new bundle directory is created but the old daemon process can still be running (from the previous bundle path). Without a handshake, the client would silently talk to a stale daemon that may not have the correct @huggingface/transformers path, producing NULL embeddings. The hello op lets the client detect this path mismatch on the very first connection and recycle the poisoned daemon.

Why the string is "hardcoded": This is a compiled/bundled file. In the TypeScript source (src/embeddings/protocol.ts), HelloRequest is declared as a discriminated union literal:

interface HelloRequest { op: "hello"; id: string }
type DaemonRequest = EmbedRequest | PingRequest | HelloRequest;

So "hello" is the typed discriminator value — the same string the client sends as op: "hello" and the daemon checks with req.op === "hello". It's as "hardcoded" as "ping" or "embed" are, which is intentional and correct for a simple line-delimited JSON IPC protocol.

[kaghni]

@coderabbitai, why is there even a problem of overriding node_modules?

[coderabbitai]

@kaghni Great question! The problem comes down to how symlinkForce works internally: it calls unlinkSync to remove the existing path before creating the new symlink. unlinkSync handles files and symlinks just fine, but throws EISDIR when the target is a real directory.

So the danger is:

1. An agent (especially one installed via a marketplace) may have a real node_modules/ directory at <pluginDir>/node_modules rather than a symlink.
2. If symlinkForce is called on such a path, unlinkSync throws EISDIR and the entire hivemind embeddings install command crashes mid-loop.
3. The result is a partially linked state — some agents got their symlinks, others didn't — which is worse than a clean skip.

The fix introduced here checks linkStateFor(install) first:

• If kind === "owns-own-node-modules" → skip with a warning (it's a real dir; unsafe to unlink)
• For everything else (no-node-modules, linked-elsewhere, linked-to-shared) → symlinkForce is safe because those are either absent or already a symlink, which unlinkSync can remove without error.

In short: it's not that overriding is philosophically wrong — it's that the underlying unlinkSync operation is incapable of removing a directory, so attempting it would crash the install loop halfway through.

[kaghni]

@coderabbitai what daemon.ts is for and where is it used? briefily

[coderabbitai]

@kaghni, here's a brief overview:

---

What daemon.ts is for

src/embeddings/daemon.ts implements EmbedDaemon — a long-lived background process that:

• Loads the Nomic embedding model into RAM once and keeps it there.
• Listens on a per-user Unix socket (/tmp/...) for IPC requests from the rest of the plugin.
• Handles three operations over that socket:
  • ping — readiness/health check.
  • embed — vectorize a text string.
  • hello (added in this PR) — handshake that returns the daemon's file path, PID, and protocol version so the client can detect stale/mismatched daemons after upgrades.
• Shuts itself down after an idle timeout (~200 MB RAM savings when unused).

Where it is used

Consumer	Role
src/embeddings/client.ts (EmbedClient)	Spawns the daemon on demand, connects to its socket, and sends hello/embed/ping requests
src/hooks/session-start-setup.ts	Warms up the daemon at session start
src/hooks/{claude-code,codex,cursor,hermes}/capture.ts	Embeds captured messages via EmbedClient
*/bundle/wiki-worker.js	Embeds session summaries before uploading
*/bundle/shell/deeplake-shell.js	Powers semantic search in the shell
src/cli/embeddings.ts	killEmbedDaemon() SIGTERMs the daemon on disable/uninstall

The bundled copies (claude-code/bundle/embeddings/embed-daemon.js, codex/..., cursor/..., hermes/..., embeddings/embed-daemon.js) are the shipped artifacts — each platform's plugin bundle includes its own copy of this file so it can be spawned as a subprocess.

[kaghni]

@coderabbitai same here, brief usage and description of nomic.ts

[coderabbitai]

@kaghni, here's a brief overview of src/embeddings/nomic.ts:

---

src/embeddings/nomic.ts — Overview

Purpose: A lightweight daemon-only wrapper around @huggingface/transformers that drives Nomic-embed text embeddings. It is intentionally isolated so the heavyweight transformer dependency is never pulled into hook bundles.

---

Transformers Resolution (new in this PR)

Function	Role
_importFromCanonicalSharedDeps(sharedDir?)	Resolves @huggingface/transformers from the canonical shared-deps directory (~/.hivemind/embed-deps) via createRequire. Handles both ESM (.mjs) and CJS (.cjs) export shapes.
_importFromBareSpecifier()	Fallback — bare import("@huggingface/transformers") for dev trees or collocated installs.
_normalizeTransformersModule(mod)	Normalizes the two possible module shapes: CJS wraps exports under .default; ESM exposes them at the top level.
defaultImportTransformers(canonical?, bare?)	Orchestrates the two-step resolution: tries canonical first, falls back to bare, throws an actionable error (hivemind embeddings install) if both fail.

The indirection through let _importTransformers allows tests to inject a mock without touching real disk paths.

---

NomicEmbedder Class

Manages the nomic-embed-text model pipeline lifecycle:

• load() — lazily initializes the feature-extraction pipeline (idempotent, deduplicates concurrent calls via a shared Promise).
• embed(text, kind) — embeds a single string with a query/document prefix (Nomic's asymmetric retrieval convention), then truncates + re-normalizes to dims dimensions (Matryoshka).
• embedBatch(texts, kind) — batches multiple strings through the pipeline in one call, then splits and truncates each result.
• truncate(vec) — slices to dims and re-normalizes the resulting sub-vector to unit length (Matryoshka re-norm; skips if norm is zero).

---

Test Seam

_setTransformersImporterForTesting / _resetTransformersImporterForTesting replace the module-level _importTransformers pointer so unit tests can bypass real disk resolution entirely without relying on vi.mock ordering.

[kaghni]

@coderabbitai how protocol works for skillify?

[kaghni]

@efenocchi, Pi and OpenClaw have no changes, but I'm not sure they should.

[efenocchi]

@kaghni Confirmed correct on both — verified the surface:

Pi: pi/extension-source/hivemind.ts ships as raw TS with a 30-LOC tryEmbedOverSocket that fails silent (no EmbeddingsClient, no enqueueNotification, no hello-handshake, no recycle) — none of the surfaces this PR touches exist client-side. The one pi bundle rebuilt (pi/bundle/wiki-worker.js) is the shared wiki-worker, unrelated. Pi does pick up the canonical daemon fix (embeddings/embed-daemon.js: +55/-4) on the next hivemind embeddings install.

OpenClaw: MCP-mode plugin with hivemind_* tool contracts only. It currently doesn't produce embeddings at all — every write explicitly NULLs the vector columns:

• upsertRowSql: summary_embedding = NULL on UPDATE, NULL literal on INSERT
• session capture: message_embedding omitted from the column list → defaults NULL

It only consumes embeddings (semantic search takes queryEmbedding as an input parameter and filters WHERE ARRAY_LENGTH(summary_embedding, 1) > 0, i.e. rows written by other agents). So there's no surface for this PR to touch.

Follow-up PR — tracked separately, not in this one. We're going to:

1. Fix a real bug in pi: the comment at pi/extension-source/hivemind.ts:166-175 promises auto-spawn-on-miss, but the implementation only calls connect() and settles `null` on error. So pi silently degrades to NULL embeddings any time it's the first agent to talk to the daemon on a given box. That's a regression vector identical to what this PR is closing — just on a different code path.
2. Make OpenClaw an embedding producer: ~60 LOC of `tryEmbedOverSocket` + spawn-on-miss, wired into the two write sites. Talks to the same `/tmp/hivemind-embed-.sock` daemon the other agents already spawn.

Why a separate PR rather than tacking it onto this one:

• Scope: this PR is "fix silent NULL embeddings after marketplace upgrades" — restoring producers that should have worked. Adding a new producer (openclaw) is a feature, not a fix; bundling them obscures the bisect/revert story if either side regresses.
• Opt-in invariant: this PR established (`src/user-config.ts`) that embeddings are explicit opt-in — env unset → `enabled: false`, no 600MB download by surprise. OpenClaw becoming a producer needs to respect that: it can only embed if `~/.hivemind/embed-deps/embed-daemon.js` is already present (deposited by `hivemind embeddings install` via some other agent). That decision and its NULL-fallback behavior deserve their own review surface, not a buried sub-commit.
• Edge case coverage: spawn-on-miss has 11 distinct states I want covered by tests (binary-missing, stale-socket, pidfile-points-to-dead-PID, two-process spawn race, pidfile-alive-but-socket-gone, etc.). Those tests double the test footprint of this PR — cleaner to land them alongside the producer code they exercise.

I'll open the follow-up against `main` once #168 is merged.

[kaghni]

@