Catalog and analyze real world occurrences of merge trains

[rudymatela]

... following up from #77

#137 was deployed to @OpsBotPrime. Here we should catalog all real world occurrences of merge trains that happen over the next few days. We should do so by linking to this issue. Then we should go through each to double-check that everything happened as expected.

Any issues that are found should be created separately on Hoff's issue tracker and listed here from most critical to least critical:

• Restart all builds in the train after a failed promotion #170
• fix double rebase with single merge command #161

In addition to fixing the issues, we can also compute how much waiting time has been reduced.

[rudymatela]

⬆️ first (simulated) merge ✔️ (reviewed)

[rudymatela]

⬆️ first (simulated) merge train ✔️ ❌ ✔️ ✔️ (reviewed)

[rudymatela]

⬆️ first merge train ✔️ ✔️ ✔️ (reviewed, 1 issue: #161)

[rudymatela]

⬆️ second merge train ✔️ ✔️ (reviewed)

[rudymatela]

⬆️ third merge train ✔️ ✔️ (reviewed)

[rudymatela]

⬆️ fourth merge train ✔️ ✔️ ✔️ (reviewed)

[rudymatela]

⬆️ fifth merge train. The first on a less busy repository. ✔️ ✔️ (reviewed)

[rudymatela]

⬆️ sixth merge train ✔️ ✔️ ✔️ (reviewed)

[rudymatela]

⬆️ seventh merge train ✔️ ✔️ (reviewed)

[rudymatela]

⬆️ second simulated merge train 🔴 ✔️ ❗ ✔️ ❌: (reviewed)

• 🔴 aborted (PR closed)
• ✔️ build success
• ❗ rebase failure
• ✔️ build success
• ❌ build failure (quick, before others)

Note the id of the PR with rebase failure does not appear when I grep with:

journalctl -Stoday -uhoff | grep "Posted.*S.*ebased as"

So I should update my regex to include rebase failures somehow and review the previous merge trains.

[rudymatela]

Here's how to look for the latest merge trains (and possibly-related) rebase failures:

journalctl -Stoday -uhoff | grep -E "Posted.*(Speculatively rebased as|Failed to rebase)"

... and with colours on PR ids:

journalctl -Stoday -uhoff | grep -E "Posted.*(Speculatively rebased as|Failed to rebase)" | grep -v grep | grep --color -E '(#[0-9]+| [0-9]+: )'

[rudymatela]

⬆️ ninth merge train ✔️ ❗ ✔️ ❌ ✔️ in Hoff repo itself! (reviewed)

[rudymatela]

⬆️ eighth merge train ✔️ ✔️ (reviewed)

[rudymatela]

⬆️ tenth merge train ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ ✔️ (reviewed)

[rudymatela]

⬆️ eleventh merge train ✔️ ✔️ (reviewed)

[rudymatela]

⬆️ twelfth merge train ✔️ ✔️ (reviewed, 1 issue: #170)

[rudymatela]

⬆️ thirteenth merge train ✔️ ❌ (reviewed)

The behaviour was correct: in the second PR the build was started speculatively, but by the time it finished its base had already been merged into master so it was reported as a real failure.

[rudymatela]

⬆️ fourteenth merge train ✔️ ❌ (reviewed)

[rudymatela]

⬆️ fifteenth merge train ✔️ ✔️ ✔️ ✔️

[rudymatela]

⬆️ sixteenth merge train ✔️ ❌

The second PR failed after the first was merged and the build was successfully reported as (not-speculatively) failed.

[rudymatela]

⬆️ seventeenth merge train ✔️ ✔️

[rudymatela]

⬆️ eighteenth merge train ✔️ ✔️ ✔️ ✔️ ✔️ ✔️

[rudymatela]

⬆️ nineteenth merge train ✔️ ✔️

[rudymatela]

Nineteen occurrences of merge trains and two issues found so far.

I checked the git history using gitg and everything has been merged correctly and in the correct order.

I am now going to stop actively looking at Hoff's actions and focus on fixing the two issues.