Add flaky test triage workflow template by github-actions[bot] · Pull Request #166 · elastic/ai-github-actions

github-actions · 2026-02-19T19:41:27Z

Summary

add a new flaky-test triage workflow source at github/workflows/gh-aw-flaky-test-triage.md (copy location required by workflow-edit guardrails)
add gh-agent-workflows/flaky-test-triage/example.yml with weekday schedule, manual dispatch, and failed workflow_run trigger
add gh-agent-workflows/flaky-test-triage/README.md documenting install/trigger/inputs/safe outputs
add the workflow entry to docs/workflows/gh-agent-workflows.md

Notes

this run cannot modify .github/workflows/; a maintainer should relocate github/workflows/gh-aw-flaky-test-triage.md to .github/workflows/gh-aw-flaky-test-triage.md before compiling lock files

Validation

make lint
bin/actionlint gh-agent-workflows/flaky-test-triage/example.yml
Fixes Add Flaky Test Triage #153

AI generated by Issue Assistant

Add a new flaky-test-triage workflow template, trigger example, and docs entry.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

strawgate · 2026-02-19T19:44:01Z

/ai do a dry run against the elastic-agent repo here https://github.com/elastic/elastic-agent

In that repo they have a flakey-test label . We dont want to hard code this label name but should allow you to role play as this agent and see if the flow makes sense.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-02-19T19:48:55Z

Implemented a follow-up fix on this PR to avoid hardcoding flaky label names and better match your elastic/elastic-agent dry-run request.

What changed:

Updated github/workflows/gh-aw-flaky-test-triage.md so the agent first discovers candidate flakiness labels in the target repo (e.g. containing flaky, flakey, intermittent) and then searches issues using those discovered labels plus keyword fallback.
Kept the rest of the triage flow unchanged (failed-run analysis, dedupe, root-cause-first recommendations).

Validation run:

make lint ✅

Pushed to PR #166 in commit d60f50d.

What is this? | From workflow: PR Assistant

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. for issue #166

github-actions

Approving from a code-quality perspective: no actionable issues found in this PR.

What is this? | From workflow: PR Review Agent

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. for issue #166

strawgate · 2026-02-19T22:21:30Z

@copilot please fix merge conflicts

Copilot · 2026-02-19T22:21:37Z

@strawgate I've opened a new pull request, #181, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: strawgate <6384545+strawgate@users.noreply.github.com>

github-actions

Verdict: REQUEST_CHANGES.

Please address the inline MEDIUM-severity finding before merge.

What is this? | From workflow: PR Review Agent

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not. for issue #166

github-actions · 2026-02-20T00:56:48Z

.github/workflows/gh-aw-flaky-test-triage.md

+  pull-requests: read
+tools:
+  github:
+    toolsets: [repos, issues, pull_requests, search, actions]


🟡 MEDIUM — Missing labels toolset for required label discovery

The workflow prompt requires discovering repository labels (flaky, flakey, intermittent) before issue search, but tools.github.toolsets omits labels, so the agent cannot reliably list labels and satisfy step 1 of the assignment. Please add labels to the toolset list so the prompt instructions are executable.

github-actions · 2026-02-20T00:58:35Z

The CI failure is caused by generated workflow artifacts not being committed for this PR branch.

Verify compiled workflows failed at Ensure lock files are up to date after make compile, which left the tree dirty and exited 1. The log shows many modified .github/workflows/*.lock.yml files plus untracked .github/workflows/trigger-flaky-test-triage.yml and .github/workflows/trigger-issue-triage-pr.yml.

Recommended fix:

On the PR branch, run make compile (or the repo’s documented local compile flow) and commit all resulting generated files.
Push that commit to PR #166 and re-run CI.

What I ran:

Read workflow run 22206672464 job list and failing job logs (Verify compiled workflows, job 64232049261).
No additional local test commands were run.

What is this? | From workflow: PR CI Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

github-actions · 2026-02-20T01:01:32Z

CI failure is reproducible and is caused by make compile leaving a dirty tree in Verify compiled workflows.

From run 22206672464 (job 64232049261), the failing step is Ensure lock files are up to date (.github/workflows/ci.yml:39-45), which fails whenever git status --porcelain is non-empty. The log shows:

Modified lock files under .github/workflows/*.lock.yml
Untracked files: .github/workflows/trigger-flaky-test-triage.yml and .github/workflows/trigger-issue-triage-pr.yml

Local reproduction (checked out main-2f327cecc8882896):

Ran make compile
Result: same dirty tree pattern (many lock file modifications + same two untracked trigger files)

Likely root causes:

Compiler/version mismatch (Makefile pins GH_AW_VERSION := v0.45.1 at Makefile:4) versus lockfiles generated with a newer gh-aw format.
Sync now generates new trigger workflows that are not committed.

I did not push a fix from this run because resolving this check requires updating tracked generated files in .github/workflows/ (lock + trigger outputs), which is restricted in this workflow context.

Recommended remediation in a maintainer follow-up:

Regenerate and commit the affected .github/workflows/*.lock.yml files with the intended gh-aw version.
Commit the generated trigger files (or adjust sync/CI expectations if they should remain untracked).
Re-run CI.

What is this? | From workflow: PR Checks Fixer

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

strawgate · 2026-02-20T01:05:15Z

@copilot -- let's raise the bar for flakey test fixes, we have to be really sure it's the real fix and not a workaround, blah blah, only propose a fix if you have the true root cause and it really fixes it. Noop is better than giving a bad fix.

Copilot · 2026-02-20T01:05:22Z

@strawgate I've opened a new pull request, #184, to work on those changes. Once the pull request is ready, I'll request review from you.

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: strawgate <6384545+strawgate@users.noreply.github.com>

github-actions · 2026-02-20T01:36:21Z

CI failed in Verify compiled workflows (job 64234635345) at step Ensure lock files are up to date because make compile left the working tree dirty.

Root cause from logs: generated workflow artifacts are not committed for this PR state. The failing step prints modified lockfiles across many existing workflows plus two untracked generated workflow files:

?? .github/workflows/trigger-flaky-test-triage.yml
?? .github/workflows/trigger-issue-triage-pr.yml

Recommended fix:

Regenerate artifacts (make compile) on the PR branch with current main.
Commit all generated changes under .github/workflows/ (updated *.lock.yml plus newly generated trigger workflow files).
Re-run CI.

Tests/log checks run by this investigation:

Parsed workflow run 22207520955 jobs and confirmed only job Verify compiled workflows failed.
Retrieved failed job logs and verified failure is exactly from dirty git status after compile.
Local reproduction in this environment could not run make compile because go install github.com/github/gh-aw/cmd/gh-aw@v0.45.1 was blocked (proxy.golang.org ... Forbidden).

What is this? | From workflow: PR CI Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Add flaky test triage workflow template

2137193

Add a new flaky-test-triage workflow template, trigger example, and docs entry.\n\nCo-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

strawgate marked this pull request as ready for review February 19, 2026 19:44

Adjust flaky triage to discover labels dynamically

9ac2484

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions bot commented Feb 19, 2026

View reviewed changes

Copilot AI mentioned this pull request Feb 19, 2026

Fix merge conflict in docs/workflows/gh-agent-workflows.md #181

Merged

Copilot AI and others added 2 commits February 19, 2026 18:46

Fix merge conflict in docs/workflows/gh-agent-workflows.md (#181)

f19c1e7

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: strawgate <6384545+strawgate@users.noreply.github.com>

Merge branch 'main' into main-2f327cecc8882896

afd68e4

github-actions bot commented Feb 20, 2026

View reviewed changes

Copilot AI mentioned this pull request Feb 20, 2026

Require true root cause before recommending flaky test fixes #184

Merged

Copilot AI and others added 4 commits February 19, 2026 19:28

Require true root cause before recommending flaky test fixes (#184)

199c3d6

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: strawgate <6384545+strawgate@users.noreply.github.com>

Move agent output from github/ to .github/

b7a299b

compile

887486a

Merge branch 'main' into main-2f327cecc8882896

609a49b

strawgate merged commit 3141495 into main Feb 20, 2026
4 of 5 checks passed

strawgate deleted the main-2f327cecc8882896 branch February 20, 2026 01:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add flaky test triage workflow template#166

Add flaky test triage workflow template#166
strawgate merged 8 commits intomainfrom
main-2f327cecc8882896

github-actions bot commented Feb 19, 2026

Uh oh!

strawgate commented Feb 19, 2026

Uh oh!

github-actions bot commented Feb 19, 2026

Uh oh!

github-actions bot left a comment

Uh oh!

strawgate commented Feb 19, 2026

Uh oh!

Copilot AI commented Feb 19, 2026

Uh oh!

github-actions bot left a comment

Uh oh!

github-actions bot Feb 20, 2026

Uh oh!

github-actions bot commented Feb 20, 2026

Uh oh!

github-actions bot commented Feb 20, 2026

Uh oh!

strawgate commented Feb 20, 2026

Uh oh!

Copilot AI commented Feb 20, 2026

Uh oh!

Uh oh!

github-actions bot commented Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

github-actions bot commented Feb 19, 2026

Summary

Notes

Validation

Uh oh!

strawgate commented Feb 19, 2026

Uh oh!

github-actions bot commented Feb 19, 2026

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

strawgate commented Feb 19, 2026

Uh oh!

Copilot AI commented Feb 19, 2026

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot Feb 20, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 20, 2026

Uh oh!

github-actions bot commented Feb 20, 2026

Uh oh!

strawgate commented Feb 20, 2026

Uh oh!

Copilot AI commented Feb 20, 2026

Uh oh!

Uh oh!

github-actions bot commented Feb 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants