Fix bors problems or decide to remove it

[ekager]

Because I don't like it 👸🏼

┆Issue is synchronized with this Jira Task

[ekager]

My arguments:

• Have had more issues with it than time it has saved us
• Flakiness / weird error messages that aren't easily resolved (at least with my level of bors understanding)
• When it autocloses the PR no one has the immediate responsibility to "QA-Needed" the issue that was linked or go and add the milestone - so it's still a manual process you just have to remember to go back and do it

[ekager]

If it would be possible to make bors an optional check so that non-admins could still merge issues if bors fails or without bors at all (once all tests pass on the PR like before) I would consider keeping it around if people have strong opinions.

[ekager]

In the small amount of time we've had bors, I've run into multiple times where bors refuses to merge a PR (even a small change) and after a million "bors retry"s I'm pretty over it. I've wasted enough time coming back and checking on it it would have been 1000x faster to just merge it myself

[NotWoods]

It's also impossible (as far as I can tell) to see the problems now if a build fails. This might be due to configuration, since A-C shows taskcluster checks still.

[sblatz]

When it autocloses the PR no one has the immediate responsibility to "QA-Needed" the issue that was linked or go and add the milestone - so it's still a manual process you just have to remember to go back and do it

I think we originally thought we could automate more of this so it’d not be manual at all. Since we can’t, I think this is one of the stronger arguments for removing it. Things are more likely to fall through the cracks.

I’m not sure how AC hasn’t run into all of these merging problems. I wonder if @pocmo can shine some light on that? We consistently get PR’s that get in an impossible-to-merge state no matter how many times we retry.

If we can’t resolve these issues I’m definitely in @ekager’s camp of removing it.

[pocmo]

I’m not sure how AC hasn’t run into all of these merging problems. I wonder if @pocmo can shine some light on that? We consistently get PR’s that get in an impossible-to-merge state no matter how many times we retry.

bors has been very stable for us. If there were problems then they were caused by taskcluster tasks failing, freezing, not reporting status etc. - even without bors that seems to be something we should get resolved.

I'm happy to help fix and debug that. What errors did bors report (build failed, timeout, not responding with anything at all..)?

I usually start by looking at the taskcluster checks and tasks for the merge commit bors creates. One of them is usually the culprit.

In general bors is super helpful to keep a healthy green master since it merges PRs that have been tested against the latest master change. I wouldn't give that up if we can fix it. :)

[pocmo]

What I just saw on my PR is:

• UI tests are taking ages and at the end fail (-> Long-term: Lets see if we can make them faster, short-term: Fix the ui tests and/or Consider removing ui tests from the checks bors waits for)
• Bors did not fail immediately when a task failed and was instead still waiting on the other tasks (and eventually timed out). I'm not sure whether we can configure that but in AC we only let bors wait on one "complete" task and that worked.

[JohanLorenzo]

Hey there!

Thanks for raising concerns about bors! I don't personally have any preference about using bors itself. Although I should say a group of Mozillians from different teams came to the consensus that having an autoland bot is the best solution we can have at the moment. The context and the decision are gathered in this document. Bors is one autoland bot, I'm open to switching to another bot.

Flakiness / weird error messages that aren't easily resolved (at least with my level of bors understanding)

I confess I'm disappointed in bors try. It intermittently times out, without any explanatory error message. This issue has been reported at bors-ng/bors-ng#739. So far, no active developer interacted with us.

An example of weird error message I got is this one: #6117 (comment).

I think no matter what autoland bot we use, we will run into setup issues. Bors has been proven good enough to suit https://github.com/servo/servo, which has a similar complexity as Gecko. What changes between servo and Fenix+AC is the way we use taskcluster-github. Fenix+AC have used Github Checks (the new API) for the past month, while Servo remains on Github Statuses (the old API). I believe the situation can be improved if bors provides a better support for Github Statuses.

It's also impossible (as far as I can tell) to see the problems now if a build fails. This might be due to configuration, since A-C shows taskcluster checks still.

I might not have the full context, but Fenix has showed Taskcluster checks (aka Github Checks) for the past month. Do you have an example of a failed build where you cannot see what the problem is?

UI tests are taking ages and at the end fail (-> Long-term: Lets see if we can make them faster, short-term: Fix the ui tests and/or Consider removing ui tests from the checks bors waits for)

👍
Another short term fix I can help with:

fenix/taskcluster/ci/test/kind.yml

Line 51 in 210e358

# TODO Generate APKs in a build task instead

This way the UI tests just run the tests and doesn't build a dedicated APK in the same APK task.

Bors did not fail immediately when a task failed and was instead still waiting on the other tasks (and eventually timed out). I'm not sure whether we can configure that but in AC we only let bors wait on one "complete" task and that worked.

We can be at parity with AC, if I manage to land #6117 (let's discuss the technical details of why I cannot in the PR itself).

Please let me know if this helps addressing your major concerns 🙂

[liuche]

For now, we've decided:

• On Friday, we'll turn on bors 11/1 but not make it a merge requirement
• If bors is running (bc someone started a bors task), DO NOT MANUALLY MERGE
• otherwise, can merge manually if things look fine

[sblatz]

I'll take this away and investigate if it's working now :)

[sblatz]

After some more testing with this, I don't think bors solves a use case for us as we still need to babysit it. Because of that I'm going to move ahead with removing it.

[JohanLorenzo]

👍 I haven't managed to get it working on Fenix since then. I think I tried like half a dozen times. It has always timed out. I'm fine removing bors from Fenix.