branch protection rules impede PRs that only modify conformance from being merged

[japaric]

as it has been observed in PRs like #2258 the current branch protection rules require that tasks like 'cleanliness' and 'compatibility' run and that prevents PRs like that from being merged.

when we introduced the conformance test suite, we tweaked CI so that only the 'conformance' job runs when code outside \conformance is not modified. in this scenario, 'cleanliness' and others do not run so the branch protection rules prevent the PR from being merged.

I investigated an alternative to this setup that preserves the behavior of running only the conformance test suite when hickory source code is not modified in this repository: https://github.com/japaric/github-branch-protection-rules-experiment/

Starting from something that resembles the hickory-dns setup, I replaced ignore-paths with an equivalent job that performs a git diff check. this job produces an output that other jobs check to decide if tests are to be executed or not.

under this new setup, all jobs run but the actual tests steps are skip based on the outcome of the git diff job. this preserves the CI time saving properties of the path-ignores solution without clashing with the branch protection rules.

these two PRs show that is the case:

• change outside /conformance japaric/github-branch-protection-rules-experiment#4 performs a change outside /conformance, which causes both the entirity of the 'unit tests' and 'conformance tests' jobs run
• change under /conformance japaric/github-branch-protection-rules-experiment#3 performs a change inside /conformance, which still causes both jobs to run but the 'unit tests' job does not perform any testing so it has effectively been omitted

I have set up branch protection rules to require both jobs, 'unit tests' and 'conformance tests', to run and in both of the above cases, they don't block the PRs from being merged. both required jobs have green checks next to them

I didn't find a way to have a step that early returns from a job so if this alternative is implemented we'll have to add an if condition to most steps of all jobs minus the 'conformance' job

japaric/github-branch-protection-rules-experiment#6 should give you an idea of the magnitude of the required changes. namely,

• the 'conformance.yml' workflow gets merged into the "main" workflow
• a git diff job needs to be added to the workflow
• several steps (in the PR only step is modified) gets an if conditional
• (none of the existing branch protection rules needs to be modified)

note that I didn't get everything working correctly in that PR so you should check https://github.com/japaric/github-branch-protection-rules-experiment/blob/98386e017a5e19873f3244734351a01fd82c113e/.github/workflows/main.yml for the actual final version

cc @djc

[djc]

Hmm... would this be better if we implemented the conformance tests in a different job? Or maybe can invert the "burden of change proof" so that only the conformance tests job tests for changes instead of all the jobs?

[japaric]

another simple fix is to run all the tests all the time. this is less CI time efficient but given that CI is not the bottleneck in getting PRs merged (+), we can probably live with that.

(+) unlike say rust-lang/rust where PRs are approved faster than it takes to run the full CI pipeline (1-2 hours?). there it makes sense to "roll up" PRs or the queue of approved PRs would never shrink