Build Cop: when more than N tests fail in a package, group them into a single issue

[tbpg]

If something is wrong with an entire package, all of the tests in that package can fail. This can lead to filing dozens of issues at a time. We should have a stopgap solution to avoid filing dozens of issues at once.

Proposal:

• If more than 10 issues fail in a single build, file an issue with the title, "packagename: many tests failed"
• Include the name of every failed test in the body of the issue.
• The issue is closed when all of the tests pass for that package.
• No new issues should be opened for that package.
• If there are already individual issues for some of the failed tests, we should leave them open and reference them in the overall issue.
  • When that test starts passing, we can close the individual issue.
  • If the test fails after the issue is closed, we should leave it closed and not mark it flaky. We can comment on the issue

[bcoe]

A couple thoughts:

Include the name of every failed test in the body of the issue.

I like this idea, you might look at GitHub markdown's support of blocks of text that can be minimized and maximized, if you're finding the size of the message is ridiculous.

If more than 10 issues fail in a single build, file an issue with the title, "packagename: many tests failed"

This sounds like a reasonable default configuration to me, you might consider it being something we could set in a config file though.

---

In general I'm pro this approach 👍