Skip to content

Releases: xienda/dsh-jev-verify

v0.8.6

Choose a tag to compare

@xienda xienda released this 07 Oct 15:03

dsh-jev-verify 0.8.6

Fixes the startup crash reported in #1.

Root cause

@deepseek-ai/dsh-tools was declared as a normal dependency (0.1.5-rc.2) - the only plugin in the ecosystem that did. On a profile using nodeLinker: hoisted + hoistPattern: ["*"], pnpm materialised a real copy under <profile>/node_modules/@deepseek-ai/dsh-tools. The loader entry id: tools resolved to that copy instead of the host's built-in one; the copy could not import its own peers (Cannot find package '@deepseek-ai/cordis'), so the tools service never registered, every plugin with inject: ["tools"] stalled, and startup ended with:

DesktopHostFatalError: dsh: startup failed: 1 required plugin did not activate

Fix

  • @deepseek-ai/dsh-tools moved from dependencies to peerDependencies (^0.1.5-rc.2) and is marked optional: true in peerDependenciesMeta, so no installer will ever materialise it - it cannot shadow the host.
  • dependencies now carries only the leaf library @deepseek-ai/schemastery.
  • Runtime resolution is unchanged: lib/index.js:28-39 imports both host packages defensively (null on failure) and the host's own store still provides them.
  • New test/manifest.test.mjs fails if any host runtime package reappears in dependencies - verified by reintroducing the bug and watching the test go red. Full suite: 38/38.

Upgrading from a broken install

Removing the plugin entry is not enough, because the hoisted copy outlives it. After upgrading to 0.8.6, delete <profile>/node_modules/@deepseek-ai/dsh-tools (or re-run pnpm install) and the host boots again. The same applies to the manually installed copy of @deepseek-ai/dsh-tools under any profile that used a hoisted linker.

Everything else is identical to 0.8.5: seven tools including jev_batch, a ledger that survives restarts, a remaining balance derived from your declared balance minus locally measured spend, per-turn auto-triage, and the live 27-question benchmark (96.3%).

v0.8.5

Choose a tag to compare

@xienda xienda released this 07 Oct 04:00

dsh-jev-verify 0.8.5

The ledger survives a restart, the remainder is honest, and a new tool does the paperwork.

0.8.5 answers a real session complaint ("the panel still shows nothing, and there is
no remaining balance") and turns the plugin from something you must remember to call
into something that does a whole batch of judgments in one shot.

1. The ledger survives a restart

The local ledger now persists to $DSH_HOME/jev-usage.json by default (quota.persist
defaults to true), so the cumulative total no longer resets with the host.

Two real bugs behind the old silent zero were found and fixed:

  • the ledger path only honoured DSH_HOME; the host process does not export it, so
    defaultUsageFile() returned null and persistence was silently disabled. It now
    falls back to %USERPROFILE%\.dsh\jev-usage.json.
  • save() ran before the first load() (it is triggered by the first record() of a new
    process), so it wrote the still-empty in-memory ring over the real history file, and the
    follow-up write was then swallowed by the 3 s debounce. save() now loads first, and
    configure() only writes when it actually has data.

A restart round-trip probe (record 3 entries, build a new module, read back) returns
today=3 / all=3 and the same per-tool and per-kind breakdown.

2. Remaining is derived, never invented

TypeSafe exposes no balance endpoint: /v1/usage, /v1/quota, /v1/account,
/v1/account/balance, /v1/balance, /v1/credits, /v1/billing, /v1/me, /v1/limits
and a dozen more all return 404, and only /v1/models answers — with no balance headers.
So the panel derives the remainder instead:

remaining = quota.declaredBalanceUsd - locally measured spend since quota.balanceSince

Set those two fields in Settings → Plugins → Jev and the pill leads with
Jev · 剩余 $4.9997 · 今日 3 次. Without them it says plainly that it cannot be fetched
instead of printing a number.

3. jev_batch — one set of judgments over many texts

New module lib/batch.js, and a seventh tool:

  • 1–20 items × 1–5 questions, one real API call per item, 4-way concurrency by
    default (6 max)
  • returns a paste-ready markdown table, sortable by a judged dimension
    (auto sorts descending only when the single question is a score)
  • built-in sentiment / priority / intent / pii / spam presets, or your own
    questions in the jev_decision shape
  • a failed row is reported as failed with the raw error — never guessed — and if every
    row fails the tool throws the first error
  • every row is metered separately (kind: "batch"), so N items really are N calls and N costs

Live end-to-end (bench/batch-e2e.mjs, real API, 3 + 2 items, 5 calls, ≈$0.0000744):
sentiment over three Chinese feedback lines sorted 2/4 (100%), 1.23/4, 0.93/4 in 708 ms
wall; then has_pii + urgency over two texts gave 100%/0.53 and 1%/0.01 in 261 ms.

4. Smaller honest fixes

  • the composer pill re-reads every 20 s instead of 60 s
  • the collapsed pill leads with the remainder once a balance is declared, and the
    cumulative row is renamed when the ledger is on disk
  • tests are isolated from the real ledger (test/_isolate.mjs points DSH_HOME at a temp
    dir); 12 fixture lines a previous test run had appended to the real
    ~/.dsh/jev-roll.jsonl were removed

Tests

npm test → 37/37 (~2.5 s): new test/batch.test.mjs (7 cases), new balance and
restart assertions in test/usage.test.mjs, pill/balance cases in test/client.test.mjs,
and the tool count moved to seven across boot/compat/present/toolview.

Scope

  • Client half (pill, cards): refresh the page.
  • Server half (persistence, remainder, jev_batch): restart the host.

v0.8.4 - honest money-first panel, metered guard, auto-triage

Choose a tag to compare

@xienda xienda released this 07 Oct 02:39

dsh-jev-verify 0.8.4

Honest panel, metered guard, and a plugin that participates on its own.

0.8.4 answers three user-visible complaints from a real session: the panel implied
a quota that does not exist, the guard's own Jev calls never reached the ledger,
and the agent simply never called Jev.

1. No fabricated quota

TypeSafe is a metered API with no balance endpoint. If you never configured a
local budget, the old panel still printed "额度正常", drew a percentage bar against
∞ and showed "额度重置 13 小时 41 分后" — a countdown to a budget that did not exist.

  • New quota.budgetConfigured drives every surface: jev_usage text, the /jev
    status page, the pill JSON route, the composer pill and the full dashboard.
  • Without a budget: no status verdict, no bar, no countdown; the text says
    pay-as-you-go, nothing resets, only locally measured usage is reported.
  • With a local budget: the countdown returns, explicitly labelled
    "本地预算重置 / 本地自设上限", and percentage bars become meaningful.

2. Money first

The collapsed pill now reads Jev · 今日 $0.000534 · 29 次; the popover leads with
today's cost and cumulative cost, and call counts are secondary rows.

3. The guard's Jev verdicts are metered

judgeRisky/judgeStall now return { noul, latencyMs, usage } (token usage comes
from body.usage, since requestSystemOne returns { body, latencyMs }) and report
through a new onJevCall event. index.js records them as kind: "judge",
tool: "auto-guard".

Before: jev_guard_status reported safety.jevCalls: 2 while jev_usage said
"今日调用 0 次". That gap was a real accounting bug, not a wording issue.

4. Auto-triage: the plugin judges the turn itself

New module lib/triage.js. On the first step of a turn the plugin asks Jev four
routing questions (intent, needs_judgment, judgment_kind, recommended_tool) and
injects the advice as a plugin-sourced user message, so the agent reads a
recommendation instead of having to remember to call Jev.

  • at most once per (agent, turn) via WeakMap; maxCallsPerSession 200
  • state clipped to 6000 chars, minChars 12, timeoutMs 4000
  • createUserMessage is imported lazily from @deepseek-ai/dsh-llm; if it does not
    resolve, the call is still metered but nothing is injected
  • fail-open on unknown host shape, missing key, timeout, abort or any API error
  • each call emits exactly one onTriage ledger event (kind: "triage")

5. Harder guidance

The system-prompt section adds mandatory-trigger rules, a ban on bypassing Jev
"because I already know the answer", and the real cost magnitude
(~$0.00002–0.0001 per call; 25 questions cost the same as 1).

Tests

npm test → 30/30 (~2.5 s). New test/triage.test.mjs (5 groups), plus new
assertions in test/usage.test.mjs (no-budget branch), test/client.test.mjs
(money-first label, no-budget pill) and test/guard.test.mjs (onJevCall payload).

Scope

  • Client half (composer pill): refresh the page.
  • Server half (guard metering, auto-triage, guidance, status page): restart the
    host.

v0.8.3 - /jev stops 404ing, plus a composer usage pill

Choose a tag to compare

@xienda xienda released this 07 Oct 01:53

dsh-jev-verify v0.8.3

Settings you save now actually reach the plugin.

What was wrong

Saving a value in Settings → Plugins → Jev reported success and changed nothing. The card sent the
field name straight to scope.set(field), and that call stores path: [field] — a single
segment. A field such as dashboard.basePath was therefore written into the settings document as one
literal key named dashboard.basePath, which no schema resolves, so the value was stored and then
ignored by every reader.

Found on a live host, not by inspection: after the 0.8.2 restart the settings document was read over the
settings/describe RPC and its user layer held two literal keys —
"autoGuard.maxJevCallsPerSession": 60 and "dashboard.basePath": "jev" — while the resolved document
still showed the schema defaults (50 and /jev). The card had lied politely for several releases.

The same read confirmed that the 0.8.1 fix is live: the decoded value has no empty-object paths and
carries apiKeyEnv: "TYPESAFE_API_KEY".

What changed

  • Nested writes. The card builds { op: "set", path: [<segment>, …] } operations and commits them
    in a single atomic scope.mutate(ops) call. Hosts that expose no mutate still get the old chained
    scope.set fallback, so nothing regresses on older harness builds.
  • Repair on open. A one-time mount effect rewrites documents written by <= 0.8.2 that contain literal
    dotted keys: for each one it sets the nested path and unsets the literal key, all inside one atomic
    mutation. A WeakSet makes it run once per settings scope, and the whole effect is wrapped, so a
    failed repair can never break the card.
  • Client-only. Refresh the page; no harness restart is required.

Verification

  • test/client.test.mjs PASS 7: path splitting, the generated operations (including numeric coercion),
    exactly one atomic repair mutation with the expected operation order, no repeat on a second render, and
    zero writes for a clean document.
  • npm test → 29/29 passing (2725.8 ms); node --check client/client.js clean.
  • Live evidence (2026-10-07): the running host's /jev answered 200 with the status page and
    /jev/api/usage answered 200 with the pill payload; the served client bundle already carried 0.8.2,
    confirming that a page refresh is enough for the client half.

Nothing else changed: six tools, the two guard tiers, the prompt section, the usage/quota accounting, the
/jev status page and pill, and every measured number from 0.7.5 / 0.8.0 / 0.8.1 / 0.8.2 stand as
published.

Upgrading

npm i dsh-jev-verify@0.8.3 and reload the page. The first time the card opens it repairs any literal
dotted keys left by an earlier version.

v0.8.2 - /jev stops 404ing, plus a composer usage pill

Choose a tag to compare

@xienda xienda released this 06 Oct 16:06

dsh-jev-verify v0.8.2

The board path stops 404ing, and the usage panel becomes a small pill by the composer.

What was wrong

Opening http://127.0.0.1:3080/jev returned the host's plain 404 page. Two independent mistakes:

  1. The routes were never mounted. The board only registered itself when dashboard.enabled === true,
    and that option defaults to false — so the path a user had configured (dashboard.basePath: jev)
    existed nowhere.
  2. The path was not normalized. basePath was used verbatim, so the value jev registered the literal
    path jev; a browser asks for /jev, which never matched. The trailing-slash form /jev/ was not
    registered either.

And the panel the board showed was the wrong shape for the request: a full standalone page is not a
lightweight usage readout next to the input box.

What changed

  • basePath is normalized (normalizeBasePath(): trim, add a leading slash, trim trailing slashes, fall
    back to /jev for an empty value or a bare slash) and both base and base + "/" are registered.
  • /jev never 404s. With dashboard.enabled: false the path now answers 200 with a server-rendered,
    JS-free status page (today's calls, today's cost, this plugin instance, today's tokens, median latency,
    quota state) that points at the composer pill and explains how to turn the full board on.
  • /jev/api/usage always mounts — a small read-only JSON projection of the same locally measured
    snapshot jev_usage returns, with an honest {ok:false, error} body (still HTTP 200) when the snapshot
    fails. It does not wait for dashboard.enabled.
  • A composer usage pill in the conversation.input.right slot (beside the submit button), modeled on
    dsh-opencode-go: Jev · 今日 29 次 · $0.000534 with a warn/danger tone dot, expanding into a 270 px
    popover — today's calls and cost against their limits, this instance's calls, today's tokens, median and
    p95 latency, guard denials, the top three tools, time to reset, the last refresh and a manual refresh.
    It polls every 60 s and on tab focus, falls back to the full board API when the pill route is absent, and
    degrades to Jev · 额度不可用 carrying the exact error. It never throws, and it reports local measurements
    only — never a vendor balance.

Verification

  • test/dashboard.test.mjs (rewritten, 10 assertions): the exact route table with the board on
    (/jev, /jev/, /jev/api/usage, /jev/api, /jev/api/try), with it off (three routes, /jev a 200
    status page), and on a custom basePath: "/custom/" (six routes, still the full board); the normalizer on
    six inputs including / and /// (which fall back to /jev so the root is never hijacked); the pill
    payload's fields with no roll key; and a throwing snapshot still yielding 200 plus ok:false.
  • test/boot.test.mjs: five routes registered, and with dashboard: {enabled:false} the /jev and
    /jev/api/usage routes exist while /jev/api/try does not.
  • test/client.test.mjs PASS 6: the pill payload's three input shapes and three junk inputs, four label
    states, five tone states, both popover states by text, the slot registration (id, order, name) and
    the rendered fallback label.
  • npm test → 29/29 passing (2470 ms).

Nothing else changed: six tools, the two guard tiers, the prompt section, the settings card fix, the
usage/quota accounting and every measured number from 0.7.5 / 0.8.0 / 0.8.1 are exactly as published.

Upgrading

npm i dsh-jev-verify@0.8.2. The server half (routes, normalizer, status page) needs a harness restart; the
client half (the pill) only needs a page refresh — the host's client-hmr row stat-polls client bundles and
re-hashes them within 500 ms.

v0.8.1 - a configured key no longer reads as unconfigured (settings card fix)

Choose a tag to compare

@xienda xienda released this 06 Oct 14:24

dsh-jev-verify v0.8.1

The Settings > Plugins card now tells the truth: a configured key is never reported as "unconfigured".

The bug

Symptom: ~/.dsh/.env holds TYPESAFE_API_KEY, and the in-chat jev_overview reports
API Key 已配置, but Settings > Plugins > Jev showed a 未配置 (unconfigured) badge with every
field blank — as if nothing had ever been saved. Saving from the card looked like it did nothing.

Root cause, located layer by layer on a live host (dsh 0.1.5-rc.2):

  1. The settings service publishes the resolved value verbatim
    (@deepseek-ai/dsh-settings/lib/index.js:364 value: registration.resolved); nothing simplifies it.
  2. The schema this plugin registered marked its GUI fields with the volatile marker. The host resolves
    @deepseek-ai/schemastery at 3.18.4, whose resolver (src/index.ts:769) turns every marked field
    into a live reference object: schema.meta.volatile ? createVolatile(...) : default.
  3. So the value on the wire was {"enabled":{},"apiKey":{},"apiKeyEnv":{},"quota":{"enabled":{},…}}.
  4. The GUI carries its own older, vendored schemastery (3.18.2) with no volatile support at all, so
    decode() (dsh-client-ui-settings/lib/client.js:1107-1117) failed its validation — the real error was
    $.enabled expected boolean but got [object Object] — returned undefined, and the draft stayed empty.
  5. The card therefore read value === {} and had no key to show.

The plugin's own config reads were never affected: normalizeConfig unwraps those references, which is
why judgments, the guard and jev_usage kept working the whole time. Only the human-facing card was blind.

The fix

  • One field definition, two schemas. buildConfig() builds both Config (entry form; keeps the
    volatile marker for hosts that understand it) and SettingsConfig (plain — the one registered with the
    settings service). The 23 fields cannot drift apart.
  • The card no longer depends on a single decode path. If the decoded value comes back empty it reads the
    raw base/user layers the service already sends, shows an explicit note that the server value could
    not be decoded (schema-generation skew), and stays fully editable and savable.
  • A credential reference is not the same as "configured". The badge has three honest states: a literal
    key that was saved, 环境变量 <NAME> (a reference, unverified), or none.

Verification

Wire probe against the deployed module (_wire_probe.mjs, 2026-10-06):

schema empty-object fields on the wire apiKeyEnv GUI decode
Config (entry form only) 8 + every quota.* {} FAILS — $.enabled expected boolean but got [object Object]
SettingsConfig (registered) none "TYPESAFE_API_KEY" passes
  • test/config.test.mjs — the registered schema resolves to plain, JSON-safe values; a save round-trip
    stays plain.
  • test/compat.test.mjs PASS 1b — volatilePaths(SettingsConfig) must be empty; wire probe asserts
    enabled === true and apiKeyEnv === "TYPESAFE_API_KEY".
  • test/client.test.mjs PASS 4 — a mini-renderer drives the real card against a snapshot whose decoded
    value is empty; the badge text must be 环境变量 TYPESAFE_API_KEY and the tree must never contain
    未配置.
  • npm test → 29/29 passing (2473 ms).

Nothing else changed: six tools, the guard tiers, the prompt section, the usage/quota panel and every
measured number from 0.7.5/0.8.0 are exactly as published.

Upgrading

npm i dsh-jev-verify@0.8.1. The server half (the registered schema) needs a harness restart; the client
half only needs a page refresh.

v0.8.0 — usage/quota panel (jev_usage)

Choose a tag to compare

@xienda xienda released this 06 Oct 14:00

dsh-jev-verify v0.8.0

Jev usage, measured — and an honest quota panel.

What's new

jev_usage: a usage/quota panel you can call

Every jev_decision, jev_choose, jev_verify and guard event is now recorded locally with its
measured input/output tokens, estimated cost, latency, question-type mix and tool name. jev_usage
turns that into:

  • rolling windows today / 7 days / 30 days / all / session;
  • calls, input/output tokens, estimated cost, latency median and p95, per-question-type and per-tool mix;
  • a projected daily burn rate — "at this pace the daily call limit is reached in N h";
  • a per-day sparkline over the retained history;
  • an optional hard stop (quota.enforce): a call that would cross a configured budget is refused
    before the API request, with a message naming the budget. Display-only by default.

The honest boundary, stated everywhere

TypeSafe publishes no balance or quota endpoint. Reproduced 2026-10-05: GET /v1/usage,
/v1/quota, /v1/account, /v1/me, /v1/balance, /v1/credits, /v1/limits, /v1/billing,
/v1/subscription, /v1/plan, /v1/user, /v1/health and /v1/ all return 404; only
/v1/models answers (jev-latest, jev-preview). A panel that showed a number from nowhere would be
exactly the kind of surface this plugin refuses to ship, so the panel reports only what this machine
measured
, against limits you set: quota.dailyCallLimit, quota.dailyCostLimitUsd,
quota.sessionCallLimit and quota.warnAtPercent. The boundary is printed in the tool description,
the text result, the in-chat card and the README.

Four surfaces

  • In the conversation — a usage tool view: status / threshold / reset-countdown chips, eight stats,
    three budget meters, the daily sparkline, the per-tool breakdown and the boundary note at the bottom.
  • jev_overview — a usage chip row (status, today's calls against the limit, today's cost, whether
    the hard stop is armed).
  • The optional /jev board (dashboard.enabled: true) — a quota section with bars, projections and a
    30-day trend.
  • Settings > Plugins — a new "usage/quota" group for every option, with numeric validation.

Persistence

In-memory by default (a 400-sample ring). Set quota.persist: true to keep up to quota.historyDays
(default 30, range 1–365) days of daily history in $DSH_HOME/jev-usage.json — written atomically
(tmp + rename) and fail-open: a panel problem can never break a judgment call.

Also in this release

  • jev_verify now records its 27 calls, tokens and cost in the ledger instead of only the accuracy text.
  • Cost accounting uses one injected constant ($0.042 per million input tokens, output free) everywhere,
    including the dashboard playground; no hard-coded copy remains.
  • Six tools, all registered and covered by boot/compat tests: jev_decision, jev_choose, jev_usage,
    jev_overview, jev_guard_status, jev_verify.

Verification

  • npm test → 28/28 passing (2,540 ms). New test/usage.test.mjs covers the empty state and its
    honest note, token-to-cost accounting, a 27-call batch (guard events counted separately, not as calls),
    quota status and all three hard-stop paths, persistence with day-boundary pruning, reload, reset, and
    every formatter.
  • Cost constant cross-checked against real measurements: 8,696 input tokens × $0.042 / 1e6 =
    $0.000365232 ≈ $0.000365 — the same figure the 0.7.5 live report published. The panel derives cost
    from measured tokens with that constant; it does not guess.
  • Live benchmark (2026-10-04, concurrency 6): 96.3% (26/27), median 282 ms, p95 1468 ms, 8,696 input +
    822 output tokens ≈ $0.000365, wall 3999 ms versus 10.07 s serial (2.52×); report
    bench/results/2026-10-04T12-40-07-421Z.json.
  • The guard audit from 0.7.5 still holds: two deterministic tiers (7 hard, position-aware + 3 soft), a
    Jev backstop, onSafetyDeny / onLoopAdvisory reported with the tool name, fail-open on Jev failure.

Upgrade notes

  • Server half needs one harness restart (the tool, the settings group and the ledger live in the
    server plugin). The client half only needs a page refresh.
  • Fully backwards compatible: all new options have defaults (quota.enabled: true, quota.enforce: false,
    quota.persist: false), so an existing config keeps working unchanged and the panel starts out
    display-only.
  • Tier-9 pipeline patch: npm dsh-jev-verify@0.8.0.

Links

v0.7.5 — honest ledger, two-tier position-aware guard, parallel jev_verify

Choose a tag to compare

@xienda xienda released this 04 Oct 12:49

v0.7.5 — the guard stops over-blocking, the ledger stops lying, jev_verify gets 2.5× faster

Release date: 2026-10-04 · npm: dsh-jev-verify@0.7.5 · commit: 57ccb0e · full diff: 6a91e7d...57ccb0e

A three-way audit of 0.7.4 (server, client, docs/packaging) produced 24 findings. This release fixes all of them. Three were more than cosmetic: the deterministic guard both over-blocked harmless text and under-detected real command variants, the dashboard ledger recorded nothing at all unless the dashboard page was enabled, and jev_verify ran its 27 live calls strictly serially at ~10.1 s — close to its own host timeout.


1. The auto-guard: two deterministic tiers, and it now reads position

Before, all patterns matched anywhere in the argument text, case-insensitively. Writing about a dangerous command — in a README, in a test, in the plugin's own guard source, in a commit message, even in these release notes — was blocked exactly like running it. At the same time the root-recursive-delete matcher required the target to follow the flags immediately, so extra flags in between, or the --flag=value form, slipped past it entirely and fell through to the shared risky-verb branch.

Now:

  • 7 hard rules fire only at an executable position — the start of the text, or after a separator — and only when the tokens before the match are at most four wrappers / env assignments / flag words. A hard-rule match anywhere else is demoted to the soft tier and its pattern is reported with a (quoted/described) suffix, so the audit trail still says which rule matched.
  • The root-recursive-delete matcher walks tokens instead of using a word regex: flags in any order, extra flags, --flag=value, and separator-delimited commands are all handled; the first non-flag token after the flags is the target. Recursive deletes aimed at a build directory or a temp path are no longer hard blocks — they go to Jev as a soft signal — while a target that is the filesystem root, a home directory, a drive root or the current directory stays a hard block.
  • 3 soft rules (taking the host down, git-history rewrite short of a force push, always-true conditional on a delete/update) are always handed to Jev against denyThreshold.
  • Totals are unchanged: 7 hard + 3 soft = 10 rules.

2. The guard now reports what it does

onSafetyDeny fires for both tiers and carries the tool name; the Jev tier adds the confidence. The loop check fires onLoopAdvisory with the tool name. Before, a single event existed for one tier only, which made two consumers dead code: the dashboard's Jev-verdict branch and its advisory counter always saw zero.

3. jev_overview no longer reports zeros

The ledger's record() returned early unless the dashboard routes had been mounted — and routes mount only when dashboard.enabled === true, which defaults to false. Result: every jev_overview call reported 0 calls, 0 tokens, $0, and $DSH_HOME/jev-roll.jsonl was never written, contradicting both the README and the settings text.

Fix: the in-memory ring always records; only the HTTP route and the JSONL append are gated by dashboard.enabled. Cost accounting uses the injected per-MTok constant instead of a hardcoded literal.

4. jev_verify runs six calls at a time

runCase + batches of Promise.all with VERIFY_CONCURRENCY = 6, results accumulated in case order. Measured wall clock 3.999 s versus 10.07 s serial (2.52×), comfortably inside the host timeout. Accuracy is unchanged at 96.3% (26/27).

5. Better errors, more honest audit

  • missingKeyError(tool) names the tool that was actually called. It used to hardcode jev_decision, so jev_choose and the dashboard playground told users to check the wrong key.
  • TYPESAFE_BASE_URL / TYPESAFE_MODEL now really override the schema default. The config default was read first, so a user pointing the plugin at a self-hosted endpoint was silently sent to the public API.
  • The in-session verdict cache is capped at 200 entries (MAX_VERDICTS); a long session can no longer grow it without bound.
  • jev_verify reports the full mislabeled list plus mislabeledHighConfidence. Previously, one high-confidence mislabel replaced the whole array, so low-confidence misses disappeared from the report.
  • safe(tag, fn) accumulates registration failures, and jev_guard_status exposes them under registrations, so a silently swallowed defineTool failure can no longer ship unnoticed.

6. Client fixes

  • noul confidence now reports the probability of the answered side: a "no" at p 0.02 used to render "置信度 2%" with warning colour, while the overview card showed the same value as 2% — the two surfaces contradicted each other.
  • jev_choose views are auditable: the state text and every candidate's text are kept, so a ranking table can be read without re-running the call.
  • Overview cards show avgLatencyMs and typeCounts, which the server already sent and the card dropped.
  • Dashboard page shows an error surface instead of staying on "加载中…" forever when a fetch fails.
  • Credential badge is tri-state: an env-var name alone reads "read from environment (not verified)" and never "configured".
  • Removed the dead CSS variable (highlight and chip fill work again in dark themes); the running chip only claims "0 questions" where that is meaningful.

7. Packaging and docs

  • test/functional.mjs → test/functional.test.mjs, so the real HTTP-contract test actually runs in npm test (27 tests, was 26).
  • bench/choose-e2e.mjs no longer hardcodes one machine's home directory: it reads TYPESAFE_API_KEY from the environment, then falls back to the DSH home.
  • bench/bench.mjs gained --concurrency (default 6) and reports it.
  • README p95 band corrected to the measured tail (825–1468 ms; single-run peaks reach ~1.5 s and a congested network ~5 s), test count corrected, engines.dsh documented as informational only (npm does not enforce it), and the rule split documented as 7 hard / 3 soft / 10 total.
  • package-lock.json brought to 0.7.5.

Verified

  • npm test: 27/27 green (12 files, 2559 ms), including 9 guard suites that assert position-aware tiering, flag-order independence, both event payloads with the tool name, and the 200-entry cache cap.
  • Live benchmark (2026-10-04, concurrency 6): 96.3% (26/27), median 282 ms, p95 1468 ms, min 231, max 1471, wall 3999 ms, 8696 input + 822 output tokens ≈ $0.000365; report bench/results/2026-10-04T12-40-07-421Z.json. The single mislabel is unchanged (severity-low, 0.01 vs expected 0).
  • Live jev_choose E2E (2026-10-04): recommended "verify, then publish" fit 1.89/3 · risk 22% · composite 49% · confidence 42%; the reckless option fit 0.1/3 · risk 90%; 933 ms, $0.0000674 (1605 in + 99 out), model jev-1.13.0.
  • The guard rewrite is its own evidence: the old deterministic tier blocked this release's source, its commit message and these notes while they were being written; the new tier lets the same text through while still blocking the same operations at executable position.

Install / upgrade

dsh plugin --profile web add dsh-jev-verify

Afterwards, cycle the host process: plugin code is loaded at start-up, so a running host keeps the previous version in memory.

Honest caveats

  • jev_overview counts from the moment the plugin instance started; it is not a persistent history unless dashboard.enabled is on (which is when the JSONL file is written).
  • Latency is network-dependent. The p95 above is one measured run; a congested network can reach ~5 s, which is why DEFAULT_TIMEOUT_MS is 15000.
  • The guard is a fail-open backstop, not a sandbox: if Jev is unreachable or the per-session budget is exhausted, suspicious commands pass. Hard rules stay in force because they are local and free.
  • Jev returns verdicts and calibrated probabilities, never explanations. Any reason shown in a tool view is written by the calling model, not by Jev.

中文摘要

三路审计(服务端 / 客户端 / 打包文档)在 0.7.4 上共提 24 条问题,本版全部修复。三处要害:确定性护栏既误拦又漏检——旧实现按整段文本做大小写不敏感匹配,写在文档、测试、插件自身源码、提交信息甚至本发布说明里的危险命令都会被硬拦,而对根目录递归删除的识别要求 flag 后紧跟目标,额外 flag 或 --flag=value 写法直接漏过;看板账本何时都记 0——record() 在路由未挂载时直接返回,而路由只在 dashboard.enabled 为真时挂载(默认 false),于是 jev_overview 恒报 0 调用 0 token $0,JSONL 永不落盘;jev_verify 串行 27 次调用墙钟 10.07 s,逼近宿主超时。

本版改动:护栏改为硬/软两层 + 位置感知(7 硬 + 3 软 = 10 条不变;硬规则只在可执行位置生效,正文/引号内降级为软并标注 (quoted/described);根目录删除改为逐 token 解析,flag 乱序、额外 flag、= 取值与分隔符之后均能识别;对普通相对目录的递归删除降为交给 Jev),两层都上报 onSafetyDeny(带工具名,Jev 层带 confidence)、循环检测上报 onLoopAdvisory(带工具名);账本内存 ring 永远记录、仅 HTTP 路由与 JSONL 落盘受 dashboard.enabled 控制,成本常量改为注入;jev_verify 以 6 路并发运行(墙钟 3999 ms,串行 10.07 s 的 2.52×),精度仍是 96.3%(26/27);missingKeyError 按工具名报错、TYPESAFE_BASE_URL/TYPESAFE_MODEL 覆盖真正生效、判定缓存上限 200、mislabeled 全量并新增 mislabeledHighConfidence、注册失败累积到 jev_guard_status.registrations;客户端修 noul 置信度、jev_choose 可审计、概览卡补 avgLatencyMs/typeCounts、看板错误面、凭据徽章三态、死 CSS 变量;打包把 functional 测试改名以真正进入 npm test(27 项)、bench 去掉硬编码家目录并支持 --concurrency、README 延迟尾部分位与测试计数诚实化、lockfile 升到 0.7.5。

实测:npm test 27/27;真实基准 2026-10-04(并发 6)96.3%(26/27),中位 282 ms、p95 1468 ms、墙钟 3999 ms、约 $0.000365;真实 jev_choose E2E 推荐项契合 1.89/3 · 风险 22% · 933 ms · $0.0000674。诚实边界:账本自插件实例启动起算;护栏是 fail-open 兜底而非沙箱;Jev 只给判定与概率,不给解释。

v0.7.4 — the guidance section really registers (and jev_choose actually works)

Choose a tag to compare

@xienda xienda released this 01 Oct 01:52

v0.7.4 — the guidance section really registers (and jev_choose actually works)

Release date: 2026-10-01 · npm: dsh-jev-verify@0.7.4 · commit: 64ee834 · full diff: 9dd64a6...64ee834

This release fixes two silent registration failures. Both shipped code that looked right in tests and did nothing in a real host; both now have regression tests that fail when the real host contract is violated.


1. The system-prompt guidance section now exists — this is what makes the harness call Jev on its own

Until 0.7.3 the plugin asked the host for its prompt slot via getSectionOrder("TOOL_JEV"). That slot does not exist in the current DSH generation (SECTION_ORDERS ends at TOOL_REPORT = 2900, then jumps to TOOLS_SDK = 5000), so the lookup returned undefined — and SystemPrompt.section() rejects a non-finite order:

TypeError: prompt section "<name>" order must be a finite number

The throw was swallowed by the same per-surface quarantine that protects the profile, so the section silently never existed and the model only ever saw the tool descriptions. That is why the harness did not reach for Jev on its own.

Fix: resolve the order as getSectionOrder("TOOL_JEV") → guidance.order → DEFAULT_GUIDANCE_ORDER = 3000, register the section as tool:jev, leave the text empty when jev_decision is not registered, and log the outcome:

[dsh-jev-verify] system prompt: guidance registered | section tool:jev | order 3000 | chars 838

The section is written as trigger rules (tool + situation + boundary), not marketing prose: what each of the five tools is for, when to use it, when not to (no text generation, no long-chain reasoning), the confidence < 0.6 → re-check with the main model rule, and the honest "Jev scores, it does not explain" contract. Measured length: 838 characters with the auto-guard on, 717 with it off.

New guidance config group — enabled / order / extra — all volatile so they are editable from the settings card without a restart.

2. jev_choose was not callable in any host in 0.7.3 — now it is

The tool declared minItems/maxItems on its options array, keywords the host value-schema DSL rejects:

JsonSchemaError: unsupported JSON schema: parameters.options.minItems is not supported by the value schema DSL

defineTool threw, the per-tool quarantine swallowed the throw, and the boot test only asserted four tools — so 0.7.3 published a headline feature that no host could call. Dropped the keywords; the real constraints (2–10 options, ≤800 characters per option, ≤2000 characters of context) are now enforced at runtime by validateOptions in lib/counsel.js, and test/boot.test.mjs asserts all five tools register through the host's real defineTool schema compiler.

3. Honesty and robustness

  • choosePresentation(value) never throws and is JSON-reversible — presentationMeta is projected while a call is running or failed, and undefined keys vanish in a JSON round-trip, so only defined keys are copied.
  • formatChoose(value) degrades to jev_choose | 尚无结果(仍在运行或已失败) instead of throwing on a non-object.
  • test/boot.test.mjs's fake context now mirrors the real host contract (section() rejects a non-finite order, getSectionOrder() only knows TOOL_GOAL), so the contract that failed silently is now covered by a failing test.
  • Tool descriptions for all five tools were rewritten to say exactly what each one returns and when it must not be used.

Verified

  • npm test: 26/26 green.
  • Live API E2E (bench/choose-e2e.mjs, key read from ~/.dsh/.env, never printed): recommended 「先验证后发布」 fit 1.93/3 · risk 23% · composite 50% · confidence 41%; trap option 「直接发布」 fit 0.11/3 · risk 90%; 「只发 npm」 1.03/3 · risk 56%. Total 895 ms, $0.0000674 (1605 in + 99 out), model jev-1.13.0. This is exactly the code path 0.7.3 never ran.
  • Benchmark (unchanged since 0.7.2): 96.3% (26/27) on the built-in 27-question set against the live API, measured 2026-09-21 and re-measured 2026-09-28; median 283–484 ms on jev-latest; $0.000365 per full run. The single mislabel is severity-low (score 0 → 0.01).
  • Boot-log proof this release is live: [dsh-jev-verify] settings namespace registered: jev-verify | schema: present | scope: object.

What the plugin gives a DSH agent

  • Five verdict tools: jev_decision (1–25 parallel atomic judgments: choice / score / noul), jev_choose (rank 2–10 candidate approaches), jev_overview and jev_guard_status (read-only audit), jev_verify (27-question self-check against the live API).
  • In-chat tool views — structured panels for every call, with latency, cost, model, threshold and expandable state preview.
  • A full settings card (Settings → Plugins → Plugin configuration → Jev) covering the auto-guard, guidance, dashboard and budget options.
  • An auto-guard on every shell-class call (bash / pwsh / run_code / terminal): deterministic hard rules first, then a Jev risk judgment against denyThreshold; stall/loop detection only advises and never blocks. Jev failure is fail-open.
  • A local /jev dashboard (opt-in, dashboard.enabled).

Install / upgrade

dsh plugin --profile web add dsh-jev-verify

Restart the host afterwards. A correct boot prints both lines quoted above; the guidance line is the one that proves the harness can now reach for Jev by itself.

Honest caveats

  • Skip 0.7.3 for jev_choose: the tool existed in the package but registered in no host.
  • The guidance section is the mechanism behind proactive Jev use; with it absent the model sees only tool descriptions (this was the 0.7.3 and earlier behaviour).
  • Real API only. Without TYPESAFE_API_KEY every tool returns an explicit error; results are never fabricated, and there is no mock fallback.

中文摘要

v0.7.4 修掉两个「静默注册失败」——它们在测试里看起来是对的,在真实宿主里却什么也没发生,现在都有会真失败的回归测试兜底。

  1. 引导段真的注册上了(这才是 harness 会主动调用 Jev 的原因)。0.7.3 用 getSectionOrder("TOOL_JEV") 取槽位,而这个 DSH 代际的槽位表里根本没有 TOOL_JEV → 返回 undefined → SystemPrompt.section() 抛 TypeError: prompt section "<name>" order must be a finite number → 异常被隔离逻辑吞掉 → 引导段从未存在,模型只看得到工具描述。修复:order 解析链 getSectionOrder("TOOL_JEV") → guidance.order → DEFAULT_GUIDANCE_ORDER = 3000,段名 tool:jev,成功日志 [dsh-jev-verify] system prompt: guidance registered | section tool:jev | order 3000 | chars 838。该段写的是触发规则(每个工具何时用、何时不要用、置信度 <0.6 回主模型复核、Jev 只给判定不写理由),不是宣传语。新增 guidance 配置组(enabled / order / extra)。
  2. jev_choose 在 0.7.3 的任何宿主里都调不到。它的 options 参数声明了 minItems/maxItems,被宿主值模式 DSL 拒绝(JsonSchemaError: unsupported JSON schema: parameters.options.minItems is not supported by the value schema DSL),defineTool 抛错被单工具隔离吞掉,而 boot 测试只断言 4 个工具 —— 于是 0.7.3 发布了一个没人能调的主打功能。现已删掉这两个关键字,2–10 个候选 / 每项 ≤800 字符 / context ≤2000 字符由 lib/counsel.js 的 validateOptions 在运行时保证。
  3. 诚实性与健壮性:choosePresentation 永不抛且 JSON 可逆;formatChoose 对非对象降级为 jev_choose | 尚无结果(仍在运行或已失败);boot 测试的 fakeCtx 改为镜像真实宿主契约(非有限 order 会抛、只认 TOOL_GOAL);五个工具的描述全部改写为「返回什么、什么时候不要用」。

验证:npm test 26/26 全绿;真实 API E2E 中「先验证后发布」契合 1.93/3 · 风险 23% · 综合 50%,陷阱项「直接发布」0.11/3 · 风险 90%,总 895 ms、$0.0000674(1605 in + 99 out)、模型 jev-1.13.0;内置 27 题基准 96.3%(26/27),2026-09-21 实测、2026-09-28 复测,jev-latest 中位 283–484 ms,单轮 $0.000365。

安装/升级:dsh plugin --profile web add dsh-jev-verify,然后重启宿主。请跳过 0.7.3 的 jev_choose(该版本该工具在任何宿主都无法注册)。本插件只走真实 API,没有 key 时明确报错,绝不伪造结果。