[Community solution] Reasoning-token breakdown for DSH evaluations: a one-line source change

[zhangdw156]

Unofficial community solution. An optional source change for users who need reasoning-token analytics. This has not been incorporated into an official release.

Chinese version

DSH already includes reasoning tokens in output tokens, consistent with its documented accounting model. For evaluations, reasoning-effort comparisons, and failure analysis, some of us also need to know how much of that output was reasoning.

We encountered this while integrating DSH into an evaluation workflow: reasoning text was recorded, but the usage emitted by llm-pi-ai did not retain pi-ai's breakdown. This optional change preserves usage.reasoning as DSH's existing reasoningTokens field while keeping the official output and total accounting unchanged.

Implementation and tests · Download patch

Related community discussions: #3825, #7110, #10840, and #11352. This post adds an implementation people can use and a reproducible validation path.

The change

Add one line to the object returned by mapUsage() in packages/llm/llm-pi-ai/src/stream.ts:

...usage.reasoning === undefined ? {} : { reasoningTokens: usage.reasoning },

It preserves positive counts and explicit zero, and omits the field when pi-ai supplies undefined. Both successful completion and error termination use this mapping.

A local HTTP fixture exercises the actual Loader, adapter, pi-ai parser, DSH LLM stream, and BlockAssembler. With 3 input tokens, 10 output tokens, and 6 reported thinking tokens:

Field	Default implementation	With this change
inputTokens	3	3
outputTokens	10	10
reasoningTokens	Absent	6
totalTokens	13	13

Reasoning is a subset of output. Do not add it to output or total tokens again. The change exposes a breakdown; it does not reduce consumption or change totals.

Apply to your own DSH source checkout

From the root of an existing DSH checkout with dependencies installed and a working build environment:

curl -fL https://github.com/zhangdw156/deepseek-harness/commit/f23717593d228e7143c7fb7e9ec68fbe6133cfdc.patch -o reasoning-usage.patch
git apply --include=packages/llm/llm-pi-ai/src/stream.ts reasoning-usage.patch
pnpm run build

The --include filter applies only the stream.ts change: one added line of runtime logic and an updated adjacent comment. It does not bring the commit's tests or package documentation into your checkout. Continue updating and building your own upstream checkout.

Restart from this checkout with your usual pnpm dsh arguments, for example pnpm dsh web. An evaluation program that loads DSH directly must also use this source or its built artifacts. This does not update a global npm or Desktop installation.

Updates and compatibility

Git locates the patch by code context rather than fixed line numbers. This keeps the change small, but does not guarantee compatibility with every future release. Changes to source paths, usage fields, or adapter interfaces still require validation. Do not force a patch that fails to apply.

As of November 1, 2026, the patch applies cleanly and modifies only the intended source file on:

• Upstream master: ce7073d51254c26cd63b22a60835b8f948deb447.
• Release tags: dsh-v0.2.2-alpha.2, dsh-v0.2.1-rc.5, dsh-v0.2.1-rc.2, dsh-v0.2.1-alpha.3, and dsh-v0.2.0-rc.2.

The pi-ai versions locked by those tags, 1.3.0, 1.1.0, and 0.87.1, all declare optional Usage.reasoning. These release-tag checks establish patch applicability and field availability, not successful builds or runtime behavior on each release. The full adapter validation below uses the current master baseline.

The public llm/stream hook inspected here runs after adapter conversion, so it cannot recover an omitted value. llm-pi-ai/request-headers does not expose responses. A future hook exposing raw usage before conversion could support an independent plugin and further reduce dependence on internal source. Suggestions for such an optional extension point are welcome.

Reproduce without an API key

The complete example branch, pinned to f23717593d228e7143c7fb7e9ec68fbe6133cfdc, includes regression tests. The minimal application above intentionally does not copy those test files.

Run the reproducer in a separate directory

Requires Node.js 24.9 or newer and pnpm 11.28.5; see the official development prerequisites.

git clone --branch fix/pi-ai-reasoning-token-usage --single-branch https://github.com/zhangdw156/deepseek-harness.git dsh-reasoning-check
cd dsh-reasoning-check
git switch --detach f23717593d228e7143c7fb7e9ec68fbe6133cfdc
pnpm install --frozen-lockfile
pnpm exec vitest run packages/llm/llm-pi-ai/tests/convert.spec.ts packages/llm/llm-pi-ai/tests/loader-composition.spec.ts -t 'preserves reported reasoning usage'

Expected: 8 passed. Six of these cases fail on the default implementation. Coverage includes positive counts, zero, absence, and both terminal event types. The Loader test replaces only the HTTP provider with a local fixture; DSH and pi-ai run their actual code.

The full adapter suite passed 452 tests locally. Artifact builds, final Host/Client type checks, and documentation checks passed; lint exited with code 0. Environment: macOS arm64, Node 26.13.0, pnpm 11.28.5, pi-ai 1.3.0. No live-model inference or Linux/Windows validation was performed.

Scope and feedback

• This covers pi-ai to the DSH LLM stream. ACP, SDK, and final statistics projections need separate verification.
• Values come from pi-ai. Its current Chat Completions parser can emit reasoning: 0 when the raw response has no breakdown, so zero is not universally proof that the model did no reasoning.
• This cannot recover previously discarded historical breakdowns. Reasoning text length is not a substitute for provider-reported token usage.

If you need this breakdown too, please share your DSH commit, API protocol, and the layer where the count appears or disappears. A redacted usage snippet is enough; no credentials or full conversations are needed.