Skip to content

prompt-optimizer: warn that instructions embedded in descriptive context markers are weak #122

@dcramer

Description

@dcramer

Affected files:

  • skills/prompt-optimizer/references/core-patterns.md ("When markers help" and "High-value prompt moves")
  • skills/prompt-optimizer/references/transformed-examples.md (could add a before/after illustrating this)

What's missing

The skill currently treats all XML-style sections as roughly equivalent carriers of instruction, as long as each rule has one authoritative owner. That's incomplete. Based on direct observation during a prompt refactor, the location of a rule matters: instructions embedded inside a descriptive/state marker are reliably weaker than the same instruction placed in the canonical rules section.

Concrete example from the field

We added a resume-detection block to a harness prompt:

<turn-state>resumed</turn-state>
This turn continues from a prior checkpoint. Post a brief continuation
notice (e.g., 'Connected — continuing.') and then the resumed answer
as a separate message.

Eval result: the model ignored the "post a brief continuation notice" directive and combined both into one message (LLM-judged score dropped from 1.0 to 0.5 on the relevant scenario).

We moved the exact same sentence into the prompt's <behavior> bullets (no other change) and re-ran. The model then posted the two separate messages and the scenario passed at ≥0.75. Two runs, identical scores, stable result.

Why this happens (hypothesis)

Descriptive markers (<context>, <turn-state>, <environment>, <artifact-state>, etc.) signal to the model "this is information about the situation." Models treat their content as facts to read, not rules to follow. Imperative directives buried in those blocks read as incidental commentary. The canonical behavior/rules section gets read as policy; context sections get read as data.

Suggested fix

Add a short "Where rules live" note in core-patterns.md:

  • Use descriptive markers (<context>, <state>, etc.) strictly for facts about the current situation.
  • Every directive must live in the canonical rules section (e.g., <behavior>, <constraints>, <tool_rules>) — not embedded inside a state/context block, even when the directive is situation-specific.
  • If a rule only applies to a specific state, phrase it there but in the rules section: "When <turn-state> is resumed, do X."

Consider adding a transformed example showing the before (instruction-in-context) vs. after (instruction-in-rules + state referenced by name).

Why it matters

"Same rule, different section" is exactly the kind of move an author makes while cleaning up a prompt — especially under the skill's current advice to collapse duplicates and group related content. The skill doesn't currently flag that moving a directive out of the rules section (even to a perfectly reasonable-looking state block) can silently degrade its effect.

Metadata

Metadata

Assignees

No one assigned
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions