[BUG] An agent acted on an instruction the user never sent, and the assembled prompt is not recorded

[MitterMeierGithub]

Environment

Harness: DeepSeek Harness (dsh) 0.1.5-rc.1; session log format v3
Runtime: Node.js v24.15.0 on Windows 10 Home (22H2, build 19045.7725), x64
Agent preset: standard; delegation depth 0; session not seeded
Provider route: go (via the dsh-llm-pi-ai adapter) → model deepseek-v4.1-flash
contextWindow 1,000,000; maxTokens 384,000; declared input modalities: text; compat.chatTemplateKwargs: {}
reasoning effort: not captured in the session artifacts
Permissions: preset workspace-write; sandbox workspace-write; approval policy ask
Tools registered at session start: 36
Summary An agent turn acted on a "user instruction" that no user ever sent, and modified a document in the workspace as a result. Local artifacts cannot show which layer produced that instruction: the assembled per-step prompt is not recorded, so we cannot tell whether a user-role message was delivered without being logged, or whether the model fabricated it.

Observed

A single agent turn contained 38 steps and 44 tool calls. Its only user input was a two-word "continue" (one user/message event).
At step 28 of that turn, the model's own assistant message text ended with a line reading user — a role marker glued to content, emitted inside its own output.
At step 29, that model's reasoning opened with The user says "", and the turn then performed 9 file edits to a document in the workspace.
The log contains no user/message event for that instruction. Counts reconcile exactly: 5 turn/start events and 5 user/message events with source.kind == "user", each carrying its own rpcId, all matching real human inputs. Within the turn in question exactly one user/message exists, and no agent/inbox/spliced insertion carries the instruction.
Impact

A contract-like document in the workspace was modified without authorization (9 edit calls across ~7 locations), and the agent reported the change as completed.
The approval policy was ask, but it did not apply: the writes were inside the workspace, which the workspace-write sandbox permits without approval. A turn whose only human input was two words therefore reached file modification with no confirmation step.
We reverted the change; this is not an ongoing data-loss report. We report it because the instruction's origin could not be established.
What we could not determine Whether (1) the harness delivered a user-role message that was not recorded, or (2) the model fabricated the instruction. Separating these requires the assembled prompt of step 29, which was not obtainable from local artifacts.

What we checked for an existing record

session.v3.jsonl.zstd stores events only; the stored request/header contains provider/model/maxTokens and tool schemas, not the prompt.
No log directory or logging switch under $DSH_HOME; settings.yaml exposes no audit or prompt-log option.
dsh-llm-pi-ai replay state is provider metadata used to reconstruct assistant messages, not the request prompt.
dsh-session-log-deepseek uploads the session log to the official API (opt-in, default off); it is not a local payload record.
Request Because the two explanations above cannot be separated locally, the following would settle it — optionally, and behind an audit/debug flag: persist the assembled per-step request, or at minimum a digest sufficient to answer "was this text ever delivered to the model as a user message?" — an ordered list of (role, message id, content hash) plus the identity/hash of the effective system prompt. Full content is not required for the digest.

Context (observation only — no request attached) In this session, instruction-like content reached the model through the user channel: user/message events carried source.kind values of agent-instructions (workspace instruction files, form: instructions, including a mid-session update), plugin (a runtime-context snapshot) and skill-catalog, alongside genuine source.kind == "user" messages. We mention this only because it means user-channel text is not by itself evidence of human authorship — part of why we wanted to inspect the assembled prompt.

Not claimed

No specific defect is claimed in DSH. We verified that dsh-llm-deepseek (serializeMessages), dsh-llm-pi-ai (toPiContext) and @earendil-works/pi-ai (openai-completions) all pass structured, role-tagged messages; none flattens messages into role-prefixed text.
No specific fix for the underlying authorization question is proposed here. This issue is scoped to auditability.

[MitterMeierGithub]

wrong submit

[MitterMeierGithub]

reopen

[argszero]

I checked the four things you looked at, and your blocker is real. The assembled per-step prompt is not recorded anywhere: the stored request/header carries provider/model/maxTokens and tool schemas but not the prompt, $DSH_HOME has no log switch, and the replay state is provider metadata. So the two explanations — a user-role message delivered without being logged, versus the model fabricating it — genuinely could not be separated from local artifacts.

Two findings that may be useful in their own right, then the tool.

1. The system prompt is the one prompt surface with no per-section attribution, and the others were designed for it.

RuntimeContextProjection.project (packages/core/agent-loop/src/runtime-context.ts:147-155) records a user-role snapshot as source: { kind: 'plugin', plugin: <system-prompt plugin>, form: 'snapshot', sections } — one entry per contributing subsystem. SystemPromptProjection.project (:83-98, at :87, :97, and :100-105) records the system prompt as createSystemMessage(text, SOURCE) with SOURCE a single constant string. No sections, no digest.

That asymmetry looks unintentional, because renderContextSections (packages/core/system-prompt/src/index.ts:303-311) documents the attribution as the reason the sections are kept at all: "a consumer that presents the snapshot uses them to attribute each part to the subsystem that contributed it, without re-splitting the joined prose." The intent is recorded; the system-prompt side does not carry it out. Since the system prompt is the largest single block of user-channel text in a request, that is the part with the most to attribute and the least attribution.

2. Your "Not claimed" section is correct, but there is a related mechanism that matters for your case.

I confirmed structured role-tagged messages reach the adapters unflattened (serializeMessages, toPiContext). Nothing in the harness puts a user marker inside content.

But deriveEventMessage (packages/core/session/src/surface.ts:113-117) replays assistant/message events as history. So at step 29 the model was shown its own step-28 output — the message ending in user<instruction> — as an assistant turn, and downstream of it a two-word human turn. A model reading that transcript has a very plausible path to "the user says X" without anything having been delivered: it is completing a pattern it set up itself. That does not prove what happened in your turn, but it means the fabrication branch has a concrete mechanism rather than being a bare alternative, which is consistent with what you observed.

3. A plugin that closes the gap without a core change.

@argszero/cordis-plugin-prompt-audit — https://github.com/argszero/cordis-plugin-prompt-audit

It mounts on llm/stream and writes one JSON line per model call to a sidecar file:

<DSH_HOME>/prompt-audit/requests.jsonl

Your requested shape is what it records — an ordered list of (role, message id, content hash) plus the identity and hash of the effective system prompt, no full content:

{"request":{"sessionId":"...","turn":1,"step":29,"provider":"go","model":"deepseek-v4.1-flash","attempts":1},
 "system":{"chars":4102,"digest":"sha256:..."},
 "messages":[
   {"role":"user","id":"m1","chars":32,"digest":"sha256:...","source":{"kind":"plugin","plugin":"@deepseek-ai/dsh-agent-instructions","form":"instructions"}},
   {"role":"user","id":"m2","chars":8,"digest":"sha256:...","source":{"kind":"user"}}]}

Two design points worth stating, because they are what make it answer your question rather than merely be present:

The record is not a reconstruction. llm/stream is the seam the loop itself dispatches through, and on that same boundary agent-loop's own invariant re-derives session.deriveMessages() on every call and fails when options.messages diverges (packages/core/agent-loop/src/invariant.ts:38-42), with the request and its messages frozen at agent.ts:604-616. So the digest covers exactly the payload the harness asserts equals the durable history.

Each message's source is recorded, not just its role. Your Context section is the reason: agent-instructions, plugin, and skill-catalog text all arrive in the user role, so role alone cannot establish human authorship. In the record above both messages are role: "user" and only source separates them. That is what turns "was this delivered" into "was this delivered, and by whom".

The package also ships an offline verifier:

node node_modules/@argszero/cordis-plugin-prompt-audit/tools/verify-request.mjs \
  <text file> <sidecar> [session.jsonl[.zstd]]

Given the instruction text and your session.v3.jsonl.zstd, it gives a three-way verdict — and the middle case is the one the session log cannot produce alone:

delivered to the model	user/message in the log	verdict
yes	yes	an ordinary recorded message
yes	no	delivered over the user channel without being logged
no	—	never delivered; the model produced it itself

Run against step 29, that separates your two explanations directly.

Why a sidecar and not the session log. The storage contract rejects unknown event types on append unless the event is declared ignorable, and that declared-ignorable write path is not reachable from a plugin — so an audit record cannot live in the log without a core change. A sidecar keeps the audit available today. If the maintainers would rather have it in the log, the system-prompt section attribution in point 1 is the change that would make the log itself sufficient.

Configuration is optional (path, dshHome, includeSessionless). The listener delegates first, never alters the stream it observes, and swallows every I/O failure — an audit that could break a model call would be worse than no audit. Verified on dsh 0.1.2-rc.1, 0.1.3-alpha.2, and 0.1.5-rc.2 by installing the packed tarball into an empty project and driving a real llm.stream call through the installed package. npm publication is still propagating; the repository above is live and installable from git.

One caveat I would rather state than let you discover: the digest answers was this exact text delivered. It will not match a paraphrase or a partial delivery, so it is a delivery record, not full-text search.

[MitterMeierGithub]

Thanks — this is a genuinely useful reply, and your second point closed
something we had left open.

We agree on the system-prompt side, and we're narrowing this issue to it.
The system prompt is the largest block of user-channel text in a request
and the only surface without per-section attribution, while the
runtime-context snapshot already carries sections for exactly that
purpose — so if the goal is for the session log to be sufficient on its
own, that looks like the minimal change, and a smaller one than recording
an assembled prompt or a digest.

Your replay point gave us the mechanism our report was missing, so we're
treating our case as the model-side branch rather than leaving it open.
One detail that we think makes it stronger: in our transcript the
two-word human turn was at the start of the turn, and the self-authored
user<instruction> line sat at the end of the model's own step-28 output,
immediately followed by a tool result rather than by a human turn. That is
why the pattern completes — in a flat role-prefixed rendering that line
reads as the most recent user turn, while the real human turn is hundreds
of events earlier.

[argszero]

Follow-up: npm propagation has completed — @argszero/cordis-plugin-prompt-audit@0.1.0 is now served by the registry (npm view resolves 0.1.0), and I verified it by installing the published tarball into an empty project. So the caveat at the end of my previous comment no longer applies: the package is installable as of now.