Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

docs: modernize BrewTestBot info for maintainers #16634

Merged
merged 1 commit into from
Feb 15, 2024
Merged

docs: modernize BrewTestBot info for maintainers #16634

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
25 changes: 14 additions & 11 deletions docs/Brew-Test-Bot-For-Core-Contributors.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +1,26 @@
# Brew Test Bot for Maintainers
# BrewTestBot for Maintainers

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice to make filename and title consistent (in either direction, don't mind)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I plan to rename the file after we agree on changes, since it screws the diff up since it's a "new file".

If a build has run and passed on `brew test-bot` then it can be used to quickly bottle formulae.
`brew test-bot` is the command our CI runs to test and build bottles for formulae.

## Bottling
## Publishing Bottles

If a pull request is correct and doesn't need any modifications to commit messages or otherwise:
If CI is passing on a pull request and it doesn't need any modifications (e.g. commit message, revision bump, etc.):

1. Review and approve the pull request. Be sure to thank the contributor!
2. Wait for BrewTestBot to automatically merge the pull request. This happens about once an hour. BrewTestBot will comment if there is a failure.
2. Wait for BrewTestBot to automatically merge the pull request. This job usually starts within a minute if both of the following are true:
- The pull request is approved by a maintainer who has write access to homebrew-core.
- CI is passing.

If a pull request won't be automatically merged by Brew Test Bot (has the labels `do not merge`, `new formula`, or `automerge-skip`), but the [commit messages and commit style](Formula-Cookbook.md#commit) are correct:
If any jobs did not complete successfully, the pull request will not automatically merge. Additionally, BrewTestBot will comment on the pull request if there is a publishing failure.

1. Ensure the job has already completed successfully.
If a pull request won't be automatically merged by BrewTestBot (has the labels `autosquash`, `automerge-skip`, or`new formula`, or has some kind of acceptable CI failure):

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The goal is to never have failing CI, and I assume we don't have documented what's an „acceptable CI failure”, so I don't think it is useful to mention here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe new formula require manual intervention at the moment

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that having failing CI is never ideal, but in this case I'm thinking about things like: checksum mismatches that have been verified, new formula that we are waiving the notability requirement for, etc. The alternative would be setting up flags for BrewTestBot to ignore all the different types of "acceptable" failures we currently see.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The alternative would be setting up flags for BrewTestBot to ignore all the different types of "acceptable" failures we currently see.

I mean that's how it already works for casks using ci-* labels.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, and we have some of that in core. I just don't think it's currently as comprehensive. @MikeMcQuaid has been prodding us to move toward a "no red CI" approach, so this may be the time to figure out any remaining instances of this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may be a good candidate for a "repo admins can merge" branch protection and we limit this to TSC.

Also: we should consider pulling these "flaky audits" into a separate job that can be non-required.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also also: if we have known flaky audits that are unfixed: this should be tracked in an issue.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's not really known flaky per se but more along the lines of say the formula homepage (but not source) temporarily goes down during a long build, which is something that's silly to perform a rebuild over. The nature of the internet means these things happen.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeh, I think we should consider pulling all these sorts of audits into another job (also because they are not OS dependent).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's not really known flaky per se but more along the lines of say the formula homepage (but not source) temporarily goes down during a long build, which is something that's silly to perform a rebuild over. The nature of the internet means these things happen.

Strongly agree with this for homepage, livecheck, etc.

1. Ensure that bottles have built successfully.
2. Run `brew pr-publish 12345` where `12345` is the pull request number (or URL).
- Approving a PR for an existing formula will automatically publish the bottles and close the PR, taking care of this step.
3. Watch the [actions queue](https://github.com/Homebrew/homebrew-core/actions) to ensure your job finishes. BrewTestBot will usually notify you of failures with a ping as well.
3. Watch the [actions queue](https://github.com/Homebrew/homebrew-core/actions) to ensure your job finishes. BrewTestBot will notify you of failures with a ping as well.

If a pull request needs its commit messages changed in a way that autosquash doesn't support (has the label `automerge-skip`):

1. Ensure the job has already completed successfully.
1. Ensure that bottles have built successfully.
2. Run `brew pr-pull 12345` where `12345` is the pull request number (or URL).
3. Amend any relevant commits if needed, then run `git push` to push the commits.
3. Amend any relevant commits if needed, then run `git push` to push the commits to the pull request.