Provider-neutral ToolSearch: source-backed API seams for safe staged tool activation

[Ashley-QwQ]

Harness's scoped ctx.tools.restrict() already gives progressive disclosure
the right visibility primitive: presentation, lookup, and execution all derive
from the same scoped view. Building a safe, out-of-tree ToolSearch plugin
against the current public API, however, runs into six concrete lifecycle and
observability gaps audited below. This is not a proposal to “add embeddings to
the core” — retrieval policy can remain completely out of tree. The narrower
question is whether Harness can expose a revisioned catalog and provide a
transactional point at which any retriever can safely change the next request's
visible tool set.

I audited the source at a fixed commit, then built and tested an independent
community compatibility adapter to check whether those six gaps are real and
whether four small seams would close them.

The practical motivation came from an external system already operating at
roughly 1,700 tools with both name lookup and purpose-oriented keyword/vector
lookup. That is context for this proposal, not a claim that Harness itself
already has 1,700 tools or a semantic tool retriever. The harder problem is how
discovery, reversible activation, model-visible schemas, execution lookup, hot
updates, and replay remain consistent as overlapping plugins accumulate.

This analysis is pinned to commit
47f943859bef60e4160492346772ded9b24f765a.
The prototype, benchmark, and API sketch are available at
https://github.com/Ashley-QwQ/dsh-hybrid-toolsearch. It is an independent
project, not an official or endorsed DeepSeek implementation.

At this audited revision, the contribution guide says external pull requests
are not currently accepted; it points proposals to Discussions and encourages
independent plugin repositories with the dsh-plugin topic:
CONTRIBUTING.md#L9-L16.

The observed problem

At the audited commit, MCP discovery drains tools/list and registers each
advertised definition in ctx.tools; a later tools/list_changed notification
repeats the synchronization. There is no query-relevance or top-k step in that
path:

• packages/mcp/mcp-client/src/tools.ts#L128-L173
• packages/mcp/mcp-client/src/connection.ts#L254-L269

The existing Skill mechanism is useful progressive disclosure for bodies, but
its model-facing catalog still contains every invocable name and description;
loading is by exact name. Provider rank resolves duplicate providers rather
than ranking a natural-language query:

• packages/skill/tool-skill/src/index.ts#L50-L57
• packages/skill/tool-skill/src/index.ts#L127-L155
• packages/skill/tool-skill/src/index.ts#L220-L228

Harness already has the right visibility primitive. Inherited definitions are
filtered by scoped restrictions, exact-agent tools are added after inherited
filtering, and the derived view supplies schemas, lookup, and execution. The
extension cookbook explicitly identifies ToolSearch/progressive disclosure as
a scoped ctx.tools.restrict() replacement use case:

• packages/core/tools/src/index.ts#L1152-L1192
• packages/core/tools/src/index.ts#L1204-L1236
• docs/cookbook/extension-cookbook.md#L95-L113

The missing pieces are lifecycle and observability seams, rather than a
particular retrieval algorithm.

Why an out-of-tree plugin cannot safely finish the job today

1. No unfiltered scoped catalog after activation. Public
   schemas(agent) returns the post-restriction view, while the complete
   inherited name sets are private. An indexer cannot reliably discover a new
   hidden tool or rebuild from authoritative unfiltered state.

2. Catalog and presentation changes are indistinguishable. Registration,
   unregistration, and restriction changes emit the same synchronous,
   payload-free tools/change. A plugin cannot observe name, scope, action,
   revision, or origin, so its own restriction can feed back into reindexing:
   index.ts#L199-L207,
   index.ts#L811-L814,
   index.ts#L1110-L1115.

3. Restriction replacement is not atomic. restrict() returns a disposer,
   and multiple restrictions intersect. Installing the new restriction before
   disposing the old one is fail-closed but exposes a temporary intersection
   and emits two events; disposing first creates an unacceptable temporarily
   unrestricted interval:
   index.ts#L713-L744,
   index.ts#L1064-L1097.

4. There is no awaited safe commit boundary. Tool schemas are derived
   during system-prompt assembly, before agent/pre-step. A restriction change
   there can make execution disagree with the already-rendered prompt.
   step/end observation is synchronous and is not an awaited transaction:
   dispatch.ts#L167-L175,
   agent.ts#L225-L242,
   system-prompt/index.ts#L467-L535,
   session/index.ts#L381-L398.

5. Changing visibility inside the search call is unsafe. Calls from one
   model response are planned before dispatch, but later siblings can re-read
   live visibility. Code Mode captures generated bindings and also rechecks
   live lookup on nested dispatch. A search body that narrows immediately can
   invalidate a sibling call or a still-running program:
   tool-calls.ts#L70-L92,
   tool-calls.ts#L198-L229,
   code-mode.ts#L594-L607.

6. Request audit data does not restore a plugin restriction. Headers record
   the old rendered view, but resume assembles a new current composition;
   ephemeral restriction ownership is not replayed:
   session/types.ts#L196-L209,
   agent.ts#L458-L470.

Requested provider-neutral seams

Would the maintainers consider four small, retrieval-agnostic APIs?

1. Read-only unfiltered scoped catalog

   ctx.tools.catalog(agentScope, {
     visibility: "inherited-unfiltered",
   }): ToolCatalogSnapshot

   The snapshot would contain stable runtime/provider identity, cloned schemas,
   schema hashes, and a monotonic catalog revision. Reading it would not change
   execution visibility.

2. Structured catalog changes, separate from presentation changes

   ctx.tools.onCatalogChange(({ revision, scope, action, toolIds, origin }) => {
     if (origin !== pluginOrigin) index.schedule(revision);
   });

   Restriction changes would keep the backward-compatible tools/change event
   but would not masquerade as catalog mutations.

3. Atomic restriction replacement

   restriction = await ctx.tools.replaceRestriction(restriction, {
     allow: selectedNames,
     origin: pluginOrigin,
   });

   Readers would observe either the old or new immutable view, never a temporary
   intersection or open set. Validation/notification failure would retain the
   old handle, and one structured visibility notification would be emitted.

4. Awaited before-assembly boundary

   ctx.agent.onBeforeAssemble(async ({ agent, previousStep, signal }) => {
     await toolSearch.commitStagedSelection(agent, signal);
   });

   It must run after all sibling calls or the complete Code program settle and
   before native schemas, SDK text, request headers, or provider requests are
   derived. Rejection should preserve the old prompt/view pair and prevent
   dispatch.

An optional explicit bootstrap/control-tool marker could replace reliance on
exact-scope registration while still applying normal guards and approval.

These seams do not put BM25, embeddings, a vector database, top-k policy, or a
vendor in Harness core. They only let an independently chosen retrieval policy
remain consistent with prompt assembly, lookup, execution, hot updates, and
session replay.

To check whether these seams are actually sufficient and correctly shaped, I
built and tested an independent compatibility adapter against the current
public API where that is possible, leaving the rest as explicit, disclosed
workarounds.

Prototype design

The prototype deliberately separates retrieval policy from Harness lifecycle:

revisioned unfiltered catalog
  -> exact name/alias + BM25 + replaceable dense provider
  -> structured constraints and semantic reranking
  -> reciprocal-rank fusion with deterministic ties
  -> discovery top-k with lane scores and reasons
  -> durable staged activation top-k
  -> all sibling calls / current Code program finish under old view
  -> awaited pre-assembly commit
  -> next request sees the control tool plus selected inherited tools

Each Discovery Card has a stable ID, canonical name/aliases, provider,
capabilities, effects, input/output and negative constraints, risk/cost/latency
hints, version, schema hash, and catalog revision. Each selection event records
the query, candidates and all lane/constraint/fusion scores, selected names,
revision, and commit status.

The exact-scope search tool remains visible independently of the inherited
allow-list. Search only stages. At the next safe boundary, replacement is
install-new-before-dispose-old; failure retains the old restriction. Resume or
fork restores the last compatible committed event. A revision mismatch records
an explicit invalidation and remains control-only until re-search. Empty,
low-confidence, failed-dense, and no-result paths never fall back to the full
catalog.

The frozen reference adapter also installs the inherited empty/replayed
restriction before registering the session's control tool, avoiding a bootstrap
full-open interval. On hot update it durably appends invalidation plus a
control-only fallback before publishing the new in-memory revision; a durable
write failure leaves the old revision and restriction live and surfaces a
retryable failure. The event store is still non-transactional across those two
records, as disclosed below.

For native, the next request receives selected function schemas. For code,
the generated SDK and all nested dispatches retain one view for the entire
program. For both, native and SDK presentations share one revision and their
duplicated presentation cost is measured separately. Code Mode is not assumed
to be inherently cheaper.

The compatibility adapter and reference runtime test those invariants, but the
audited public API cannot attach the required awaited boundary. This is a
reference host contract, not a claim of safe drop-in integration today. Full
design and tests:

• architecture
• compatibility matrix
• source audit
• minimal seam proposal

Reference retrieval implementation — validation results

The seams above are retrieval-policy-agnostic; they do not require BM25,
embeddings, or any specific algorithm. To exercise them concretely, the same
adapter drives one reference retrieval implementation (exact/alias + BM25 + a
replaceable dense provider, fused with RRF). That implementation's own quality
claim is disclosed in full below because it is part of the same evidence trail
as the source audit — not because it is what I am asking maintainers to adopt.

Environment: Node v24.19.0, Windows x64, Intel Core i5-10600T (12 logical
cores), seed 1592598566. Both quality comparisons use the fixed synthetic
1,000-tool catalog. The gate was unchanged: hybrid Recall@5 must be at least
90%, Recall@10 at least 97%, and Recall@5 must exceed the better keyword/dense
single lane by at least 5.00 percentage points.

Historical holdout-v1 — FAIL

The original 48-query sealed result is retained unchanged:

Runner	Recall@5	Recall@10	MRR@10
keyword-only	93.75%	93.75%	0.8125
dense-only	93.75%	95.83%	0.7292
hybrid	97.92%	100.00%	0.8628

Hybrid gained two hits over the better single lane: 4.17 points, not the
required 5.00. The sole hybrid miss placed the gold tool at rank 6. This result
was not used for later tuning and was never rerun with the new ranking code.

Preregistered holdout-v2 one-shot — FAIL

Option A used 144 independently authored, provider-blind, text-only semantic
queries. They were sealed before development changes. Final implementation
commit 62d8f2b302fd27bea3517ebc1b46899dc35569f4 was bound by manifest SHA-256
3da516b09e55be0aec8e68e1ef971689f8c5427dda64419e670df6e6f5243fd4.
Run 78276923-eed1-4471-b8a8-6936765e82c7 durably wrote its started ledger
before loading sealed text and completed exactly once:

Runner	Recall@5	Recall@10	MRR@10
keyword-only	132/144 (91.67%)	136/144 (94.44%)	0.7152
dense-only	122/144 (84.72%)	129/144 (89.58%)	0.6443
hybrid	133/144 (92.36%)	133/144 (92.36%)	0.7589

Hybrid recovered only one additional top-5 query, an improvement of
0.69 points. It therefore failed the 5-point relative gate. Its 92.36%
Recall@10 also failed the 97% absolute gate; the 92.36% absolute Recall@5 gate
passed. Separate exact suites achieved 24/24 canonical names and 24/24 stable
IDs at Rank@1.

Frozen v2 scale gate	Result	Threshold	Status
1k warm hybrid p95	19.26 ms	<=100 ms	PASS
10k warm hybrid p95	156.35 ms	<=500 ms	PASS
1k native top-5 presentation reduction	99.39%	>=95%	PASS
1k code / both reduction	99.30% / 99.34%	reported separately	—

The like-for-like development raw-text result had shown a 5.36-point gain, so
the v2 result records a material development/holdout transfer failure. I did
not change weights, redefine a baseline, lower a threshold, or rerun the
one-shot after disclosure. Both failed validations remain first-class evidence.

The corpus is synthetic, v2 is process-isolated model-authored rather than
production-log or external-human data, and the local embedding is a
deterministic test provider. The roughly six-second 10k cold build/query,
MRR@10 truncation, presentation-character proxy, and frozen implementation
edge cases are disclosed in the linked limitations rather than hidden behind
the passing latency/schema gates.

Evidence:

• v2 one-shot report
• v2 machine result
• v2 append-only ledger
• v2 failure analysis
• historical v1 report
• v1 failure analysis
• known limitations

Reproduction from a checkout uses Node ^22.19.0 || >=24.0.0 and pnpm
11.19.0. It verifies the completed ledger/artifact hashes and recomputes the
disclosed deterministic projection; it is not a second independent holdout:

pnpm install --frozen-lockfile
pnpm verify
pnpm generate:corpus
pnpm historical:v1:verify  # hashes preserved v1 FAIL; does not re-evaluate it
pnpm benchmark             # safe v2 reproduction; writes nothing
pnpm demo

Security and limitations

restrict() is a scoped composition/visibility mechanism, not an authorization
boundary. ToolSearch can reduce what the model sees and what ordinary scoped
lookup resolves, but it must not replace Harness guards, approvals, credentials,
or sandboxing. Risk metadata in the prototype affects ranking only.

The prototype is synthetic, single-process, and uses a deterministic local
embedding provider. It failed the relative quality gate twice and has not
demonstrated production vector-store behavior or a safe dynamic attachment to
the audited public API. The tested adapter is a reference host contract, not a
drop-in Harness plugin.

Known frozen edge cases were deliberately documented rather than repaired
after the one-shot result:

• field-aware interaction is computed for a top-100 coarse union, but
  differently scaled coarse scores outside that window are mixed back into the
  final ordering; the top-100 window is therefore not a hard candidate bound;
• the lane named exact also contains contained-name/token-overlap heuristics,
  although the separate strict identity suites passed 48/48;
• the ranking fingerprint binds declared configuration/provider while the
  final manifest, not that fingerprint alone, binds every implementation byte;
• durable visibility events retain candidate scores/revision but omit the full
  engine ranking fingerprint, and staging does not prove every selected item
  came from the supplied candidate list;
• control-name/run_code collisions and every disposer-failure retry path are
  not fully preflighted;
• hot-update invalidation uses multiple durable appends, not a transaction; a
  partial failure can make cold replay more restrictive than the live process;
• malformed-card quarantine, duplicate-identity rejection, and dense-provider
  failure are deterministic fail-closed paths but do not all return a normal
  search fallback with equally detailed diagnostics.

The complete list, including benchmark measurement limits, is in
known-limitations.md.
These points are not excuses to reinterpret the failed result. They are the
frozen boundary for deciding whether another iteration is worthwhile.

Those limitations are why I am asking primarily about the four
lifecycle/catalog seams and their invariants, not proposing the ranking
implementation for core adoption.

I would especially appreciate feedback on whether the proposed boundary should
be agent/before-assemble, a transactional replacement around system-prompt
assembly, or another lifecycle point that also covers the initial request,
zero-tool turns, retry, resume, fork, and complete Code programs.

[Yan-Zero]

I kept coming back to this audit while trying a smaller compatibility path on the current public surface. That work is now published as dsh-progressive-tools (0.2.3).

The plugin leaves ToolRuntime visibility and execution authority live, then owns only the model-facing projection. search_tools({}) (or *) returns the complete lightweight name + description catalog, loose text queries rank that same catalog, and describe_tools returns exact schemas for the selected names. Native APIs with pi-ai deferred loading get ordinary disclosed tool calls; other Native APIs keep an invoke_tool fallback; Code continues through run_code.

That narrower design avoids changing restrictions inside a search call and keeps a stable top-level tool prefix across ordinary catalog registration changes. It also reconstructs Native disclosures from durable describe_tools call/result history instead of keeping a mutable reveal set.

There is still a sharp edge that lines up with the seams requested here: DSH exposes llm/stream, but no adapter context-transform hook for native deferred schemas. The current bridge therefore wraps the resolved pi-ai model instance with feature detection and falls back to invoke_tool when the expected shape is unavailable. A provider-neutral request-context hook would remove the most fragile part of the implementation.

Install, for anyone who wants to compare the compatibility approach with the staged-restriction prototype:

dsh plugin --profile web add dsh-progressive-tools

The README documents the stable surfaces, cache behavior, and the pi-ai boundary in detail. I would be interested in comparing traces from the two approaches, especially around hot catalog changes and resumed sessions.

[labmimors]

One concrete data point from the MCP side: a useful user-facing reduction path already exists today without requiring core retrieval policy in Harness.

I maintain MCP Lens, a community plugin now published as dsh-mcp-lens@0.1.0-rc.9. Its scope is deliberately narrower than a general ToolSearch layer: for large MCP catalogs, it keeps the model-facing surface fixed at two interfaces (mcp_search, mcp_call), ranks against an indexed MCP catalog out of band, and reveals the selected tool's exact schema only when needed. Small catalogs are usually simpler to expose directly.

On the fixed synthetic 1,000-tool fixture we used for release verification, the model-facing schema JSON changed from 1,000 schemas / 647,962 UTF-8 bytes to 2 schemas / 1,114 bytes. That number is only serialized schema bytes, not a universal token, latency, or task-quality claim. The released package also keeps MCP servers disabled by default and requires an explicit allowlist.

Why this seems relevant to the seams in this thread: it suggests the problem can be split cleanly into two layers.

1. A product/plugin layer can already prove demand for bounded model-visible tool surfaces.
2. A future provider-neutral Harness seam could make staged activation transactional and replay-safe across native/code paths, instead of each plugin carrying its own lifecycle workarounds.

So from a maintainer perspective, the highest-leverage core API still looks like catalog/lifecycle consistency, not "which retriever should core bless".

Reference:
https://github.com/labmimors/dsh-mcp-lens
Disclosure: I maintain MCP Lens.

[weijiafu14]

你这份审计做得很细，而且把问题问在了正确的位置——"能不能有一个事务性的时点，让任意 retriever 安全地改变下一次请求可见的工具集"。

我认为这个时点今天已经存在，只是它不叫 tools 相关的名字。 这条我们实测过，可能会直接关掉你六个缺口里的几个。

system-prompt/assemble 是一个 awaited async waterfall，assembly.tools 可以在里面被改写

两条实测结论（在真 DSH 组合上跑探针跑出来的，不是读类型推的）：

1. system-prompt/assemble 是一个 async waterfall，装配时跑，返回值是权威的；
2. assembly.tools 的已有条目可以改写，不只是追加——我们在那个监听里拿到已注册工具的条目，改了 description、从 parameters.properties 里删掉一个字段，assemble 返回的结果里两处改动都在。

而 assembly.tools 本身是在这个 waterfall 之前就被快照的——也就是说，它是"这一轮模型将要看到的工具集"的一份可写副本，而 waterfall 的返回值决定最终结果。

这正好是你要的那个事务点：每轮装配时跑一次、可以 await（所以异步的检索可以在里面完成）、改动只影响这一次请求、下一轮重新来过。

为什么这条容易被漏掉（我们也漏过）

我们当初的问题是"怎么让模型看不见某些工具"，第一反应去看名字最像的 ctx.tools.restrict()，撞了墙就得出"DSH 的 scope 模型不给"的结论。

后来改了方法才找到：不要从"我手上这个 API 能干什么"出发，要从"模型看到的那个工具列表是从哪来的"倒着查。 数据总得从某处来——一路追下去就到了 assembly.tools，而它就在一个可写的 waterfall 里。

我们为此立了条规矩：说"平台做不到"之前，必须能说出"这个结果的数据流我追到了哪一步、断在哪个具体符号上"。你这份审计的质量已经远超一般的功能请求，我建议对那六个缺口各做一次这个倒推——如果其中几个的答案都落在 assemble 这一层，你的提案会从"请开四个新 seam"缩成"请把 X 补上"，好推得多。

具体对上你审计里的几点

• "可逆的激活"：assemble 里的改写天然可逆——它只作用于这一次装配，你不需要一个 undo 路径，下一轮不改就恢复了。这比 restrict() 那种带作用域生命周期的东西简单。
• "模型可见的 schema"：上面第 2 条实测的正是这个——你可以给模型一份精简 schema，而执行权威仍在 ToolRuntime。@Yan-Zero 的 dsh-progressive-tools 说它"只拥有 model-facing projection、把可见性和执行权威留给 ToolRuntime"，那正是这一层能给的东西。
• "执行查找"：这一层不改执行，所以模型如果调了一个被你隐藏的工具，ToolRuntime 那边依然认得——这是好事（不会因为投影不同步而炸），但也意味着隐藏不等于禁用，你的安全模型要建立在这个前提上。
• "热更新与重放"：每轮重新装配天然跟随最新目录；重放时也是按当轮重新算，不需要持久化投影状态。这可能是你六条里最省事的一条。

必须说清的边界

• 我验的是改写 assembly.tools 的条目（description / parameters）。"能不能整条删掉一个工具、让它彻底不在列表里"我没有单独验过——从形状上看它是同一个数组，但我不能替你断言，值得你自己跑一下；
• 我们的验证场景不是 ToolSearch，是别的用途（把一个会诱发死循环的字段从原生工具 schema 里摘掉）；
• 另外已实测证伪的一条，省你时间：tools/pre-execute 改不了 DSH 原生工具的参数——目标是原生工具时改 event.input 会被直接拒，理由是核心在策略之前就把 arguments 记进日志了，这是有意为之。所以如果你的设计里有"在执行前改写调用"的环节，那条走不通。改 schema（模型看到什么）可行，改 arguments（模型已经发出来的调用）不可行。

关于两位评论者的方案

@Yan-Zero 的 dsh-progressive-tools 和 @labmimors 的 dsh-mcp-lens 都在做"固定几个入口 + 按需展开 schema"，@labmimors 那个 1000 工具 → 2 schema / 647,962 字节 → 1,114 字节的数字很有说服力（而且他很谨慎地限定了那只是序列化字节数）。这两个已经发布的包证明了需求真实、且在现有公开面上做得出来——建议在提案里引上，它们是"这个 seam 有真实消费者"的最好证据，而这恰恰是这类提案最难自证的部分。

边界与利益相关

我们不修 DSH 自家组件——ctx.tools、assembly.tools、MCP discovery 都在 DSH 里。上面是一条实测机制加一条方法建议。

利益相关：我维护 pi2dsh（Pi 生态兼容层）。这条不推销：system-prompt/assemble 是任何插件都能监听的公开 waterfall，不需要经过我们；上面两个 ToolSearch 方向的包是别人的，不是我的。