Skip to content

Latest commit

 

History

History
55 lines (49 loc) · 2.78 KB

block_merges.md

File metadata and controls

55 lines (49 loc) · 2.78 KB
Raw

Blocking PR merges in the event of regression.

As mentioned in the charter, SIG scalability has a right to block all PRs from merging into the relevant repos. This document describes the underlying "Rules of engagement" of this process and the rationale why this is needed.

Rules of engagement.

The rules of engagement for blocking merges are as following:

  • Observe as scalability regression on one of release-blocking test suites (defined as green to red transition - if tests were already failing, we don't have a right to declare a regression).
  • Block merges of all PRs to the relevant repos in the affected branch, declaring which repos those are and why.
  • Identify the PR which caused the regression:
    • this can be done by reading code changes, bisecting, debugging based on metrics and/or logs, etc.
    • we say a PR is identified as the cause when we are reasonably confident that it indeed caused a regression, even if the mechanism is not 100% understood to minimize the time when merges are blocked
  • Mitigate the regression. This may mean e.g.:
    • reverting the PR
    • switching a feature off (preferably by default, as last resort only in tests)
    • fixing the problem (if it's easy and quick to fix)
  • Unblock merges of all PRs to the relevant repos in the affected branch.

The exact technical mechanisms for it are out of scope for this document.

Rationale

The process described above is quite drastic, but we believe it is justified if we want kubernetes to maintain scalability SLOs. The reasoning is:

  • reliably testing for regressions takes a lot of time:
    • key scalability e2e tests take too long to execute to be a prerequisite for merging all PRs, this is an inherent characteristic of testing at scale,
    • end-to-end tests are flaky (even when not at scale) requiring retries,
  • we need to prevent regression pile-ups:
    • once a regression is merged, and no other action is taken, it is only a matter of time until another regression is merged on top of it,
    • debugging the cause of two simultaneous (piled-up) regressions is exponentially harder, see issue 53255 which links to past experience
  • we need to keep flakiness of merge-blocking jobs very low:
  • regarding benchmarks, there were several scalability issues in the past caught by (costly) large-scale e2e tests, which could have been caught and fixed earlier and with far less human effort if we had benchmark-like tests. Examples include:
    • scheduler anti-affinity affecting kube-dns,
    • kubelet network plugin increasing pod-startup latency,
    • large responses from apiserver violating gRPC MTU.

As explained in detail in an issue, not being able to maintain passing scalability tests adversely affect:

  • release quality
  • release schedule
  • engineer productivity