Skip to content

claude: add roachtest failure investigation command#152444

Open
dt wants to merge 1 commit intocockroachdb:masterfrom
dt:claude-command
Open

claude: add roachtest failure investigation command#152444
dt wants to merge 1 commit intocockroachdb:masterfrom
dt:claude-command

Conversation

@dt
Copy link
Copy Markdown
Contributor

@dt dt commented Aug 25, 2025

This is an attempt to have AI assist with roachtest failure investigation. AI seems too eager to jump to conclusions to let it try on its own, but it can automate data collection tasks and present information to a human detective to explore further.

This command was written by asking Claude to reflect on where it made mistakes and modify its instructions through repeated iterations on a few test-case tickets. This process revealed an extreme propensity to jump to often incorrect conclusions, thus the heavy emphasis on sticking to rigid rules around role and how to perform the task, including explicitly controlling the todo list.

Release note: none
Epic: none

This is an attempt to have AI assist with roachtest failure investigation.
AI seems too eager to jump to conclusions to let it try on its own, but it can
automate data collection tasks and present information to a human detective to
explore further.

This command was written by asking Claude to reflect on where it made mistakes
and modify its instructions through repeated iterations on a few test-case
tickets. This process revealed an extreme propensity to jump to often incorrect
conclusions, thus the heavy emphasis on sticking to rigid rules around role and
how to perform the task, including explicitly controlling the todo list.

Release note: none
Epic: none
@dt dt requested review from golgeek and jeffswenson August 25, 2025 14:20
@cockroach-teamcity
Copy link
Copy Markdown
Member

This change is Reviewable

@dt
Copy link
Copy Markdown
Contributor Author

dt commented Aug 25, 2025

The referenced guides with the actual technical details of various tests and how to read the logs are not claude-specific, but rather are just general technical detail on and strategies for inspecting a roachtest, so I put them in their own folder in their own PR to keep this one focused on just slowly repeating myself at claude.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants