GitHub Actions CI error annotations

[lukel97]

A nice feature of the buildkite premerge CI was that it showed a summary of the failed tests from lit at the top.

Is it possible to do something similar for GitHub Actions? At least on the runs I've been looking at there's no extra information:

There's this ::error format you can use when logging to have github pick it up as an annotation I wonder if lit could be taught to emit errors this way when passed a flag.

[llvmbot]

@llvm/issue-subscribers-infrastructure

Author: Luke Lau (lukel97)

A nice feature of the buildkite premerge CI was that [it showed a summary of the failed tests from lit at the top](https://discourse.llvm.org/t/fyi-new-test-result-reporting-in-pre-commit-builds/83114).

Is it possible to do something similar for GitHub Actions? At least on the runs I've been looking at there's no extra information:

<img width="433" alt="Image" src="https://github.com/user-attachments/assets/c8540d25-c149-48ff-9e6d-b422ae51440c" />

There's this ::error format you can use when logging to have github pick it up as an annotation I wonder if lit could be taught to emit errors this way when passed a flag.

[lukel97]

Oops I just found #133197. Is this already enabled? Maybe it's just that the run I'm looking at didn't have any failed lit tests: https://github.com/llvm/llvm-project/actions/runs/15309681789/job/43071103385?pr=141845

[boomanaiden154]

It does exist (https://github.com/llvm/llvm-project/actions/runs/15309681789?pr=141845), it's just not as easy to find.

We need to do some more investigation to figure out how to surface this better.

[lukel97]

It does exist (https://github.com/llvm/llvm-project/actions/runs/15309681789?pr=141845), it's just not as easy to find.

Ah thanks, that's exactly what I was looking for.

I guess that's the one benefit of using the github annotations in that they show up at the top of each job, not just the run, and can provide a link to the failed files. But I think this something to be done more on the lit.py side.