Skip to content

fix(session): move datetime from env block to user message#35310

Open
malventano wants to merge 1 commit into
anomalyco:devfrom
malventano:fix/move-datetime-from-env
Open

fix(session): move datetime from env block to user message#35310
malventano wants to merge 1 commit into
anomalyco:devfrom
malventano:fix/move-datetime-from-env

Conversation

@malventano

Copy link
Copy Markdown

Issue for this PR

Closes #29672, #32622

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

The env block's Today's date is the only dynamic field in an otherwise static system prompt. It changes at midnight, invalidating the KV cache for every active session, subagent, and background task that continues into the next day. The cost impact ranges from 10x (Anthropic) to 120x (DeepSeek) cache miss premium per session, and self-hosted users take a TTFT stall from full re-prefill on every resumed session.

This removes the date from env and moves it to a <system-reminder> tag on the last user message, using info.time.created (DB-persisted message creation timestamp), upgrading from date-only to full datetime (toString()) so the model gets time and timezone context for elapsed-time reasoning across tool call chains. The env block is now fully static; the stamp lives after the cache boundary, re-derived from immutable metadata each step.

Cost impact

When the cached prefix changes, providers reprocess the full system prompt at the uncached rate:

Provider Cached Uncached Ratio
Anthropic Sonnet 4.6 $0.30/MTok $3/MTok 10x
Anthropic Opus 4.8 $0.50/MTok $5/MTok 10x
DeepSeek v4-pro $0.000000003625/tok $0.000000435/tok 120x

For a 100K-token session crossing midnight, avoidable cost is $0.27 (Sonnet) to $0.45 (Opus) per session. A real-user report measured ~$0.50/session spike at midnight (#31266). Multiplied by every concurrent session, subagent, and background task across all users crossing midnight simultaneously.

Performance impact

For self-hosted users (vLLM, TGI, Ollama, llama.cpp), a midnight boundary forces a full re-prefill on every session continuing into the next day. For a 100K-token prompt, prefill can take tens of seconds to minutes, the user feels this as a TTFT stall on the first prompt after midnight, on every session resumed the next day. With a static env block, the KV cache persists across days and session resumptions, so prefill runs once and is reused indefinitely.

References: #31266, #31363, measured DeepSeek cost data

Supersedes #32677, #33246.

How did you verify your code works?

Built locally. Confirmed the env block no longer contains the date string, and the last user message receives a <system-reminder> with the full datetime stamp including timezone. The stamp is re-derived from info.time.created each step, so it persists across the conversation without DB writes.

Screenshots / recordings

N/A -- terminal behavior

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

@malventano

Copy link
Copy Markdown
Author

Equivalent plugin that does the same thing for those not wanting to rebuild Opencode with the PR:

~/.config/opencode/plugins/time-context.js

export default {
  id: "time-context",
  server() {
    return {
      'experimental.chat.system.transform': async (_input, { system }) => {
        system[0] = system[0].replace(/\n\s*Today's date: .+/, '')
      },
      'experimental.chat.messages.transform': async (_input, output) => {
        const last = output.messages.findLast(m => m.info.role === 'user')
        if (!last) return
        const part = last.parts.find(p => p.type === 'text' && !p.synthetic)
        if (!part || part.text.includes('<system-reminder>')) return
        part.text += `\n\n<system-reminder>${new Date(last.info.time.created).toString()}</system-reminder>`
      },
    }
  }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE]: Modify system prompt to improve cache hit rate

1 participant