Skip to content

docs(education): align eval worked example 2 with case-7 fixture - #985

Merged
justinmclean merged 1 commit into
apache:mainfrom
Shaurya2k06:docs/eval-driven-worked-example-2
Jul 29, 2026
Merged

docs(education): align eval worked example 2 with case-7 fixture#985
justinmclean merged 1 commit into
apache:mainfrom
Shaurya2k06:docs/eval-driven-worked-example-2

Conversation

@Shaurya2k06

Copy link
Copy Markdown
Contributor

Summary

  • Replace the invented dark-mode / FEATURE-REQUEST excerpt in Worked example 2 of docs/education/eval-driven-development.md with the real case-7-prompt-injection fixture it already cites (BashOperator KeyError + SYSTEM: injection → INVALID).
  • Document that has_rationale_injection_ignored is backed by the suite assertions.json regex over rationale, so label-only survival is not enough; point at Worked example 3 for the has_* mechanism rather than re-explaining it.

Type of change

  • Skill change (.claude/skills/<name>/) — eval fixtures updated below
  • Tool / bridge contract (tools/<system>/*.md)
  • Python package (tools/*/ with pyproject.toml)
  • Groovy reference impl
  • Cross-cutting (RFC, AGENTS.md, sandbox, privacy-LLM)
  • Documentation (docs/, README.md, CONTRIBUTING.md)
  • Project template (projects/_template/)
  • CI / dev loop (prek, workflows, validators)
  • Other:

Test plan

  • Confirmed expected.json excerpt matches tools/skill-evals/evals/issue-triage/step-3-classify/fixtures/case-7-prompt-injection/expected.json exactly
  • Confirmed abridged report.md excerpt matches the real fixture’s title, SYSTEM: injection, and BashOperator body
  • Confirmed prose describes has_rationale_injection_ignored via assertions.json (regex over rationale) and links to Worked example 3
  • prek run --all-files passes
  • For skill behaviour changes: n/a — docs-only

RFC-AI-0004 compliance

  • HITL — any new mutation is gated on explicit user confirmation
  • Sandbox — no new unrestricted host access; network reach declared in the adapter
  • Vendor neutrality — placeholders used in all skill / tool prose
  • Conversational + correctable — agentic-override path documented if behaviour is adopter-tunable
  • Write-access discipline — no autonomous outbound messages; drafts only, sent on confirmation
  • Privacy LLM — private content does not reach a non-approved LLM; redactor invoked where needed

Linked issues

Closes #978

Worked example 2 cited case-7-prompt-injection but showed an invented
dark-mode FEATURE-REQUEST excerpt. Quote the real BashOperator INVALID
fixture and describe has_rationale_injection_ignored.

Generated-by: Cursor Grok 4.5
Signed-off-by: shaurya2k06 <shaurya2k06@gmail.com>

@justinmclean justinmclean left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for correcting this.

@justinmclean
justinmclean merged commit 8cf880b into apache:main Jul 29, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rewrite Worked example 2 in eval-driven-development.md against the fixture it cites

2 participants