Batching

[gbrindisi]

Pull Request

This pull request implement a configuration to enable batching the maximum amount of eligible repositories in scope for every Evergreen run.

Proposed Changes

Add a config variable that is compared to the eligible_repositories counter and breaks the repositories loop when the batch value is met.

Readiness Checklist

Author/Contributor

• If documentation is needed for this change, has that been included in this pull request
• run make lint and fix any issues that you have introduced
• run make test and ensure you have test coverage for the lines you are introducing

Reviewer

• Label as either bug, documentation, enhancement, infrastructure, or breaking

[zkoppert]

Hey @gbrindisi ! Thanks for opening up this pull request. Can you help me understand the use case for limiting the number of eligible repos to a max count? Is this for making sure the workflow doesn't take too long?

[gbrindisi]

@zkoppert hi! when rolling out changes to a large sample of targets (such as rolling out dependabot), we might not always have the confidence to fully assess the impact from the start.

In cases like this it's preferred to gradually roll out in batches so that we get the opportunity to address any unexpected outcome and correct course with limited costs. It is also a way to let developers digest the change without being too overwhelmed by it.

This is what I am experimenting in an enterprise account at $DAYJOB with evergreen: I rolled out in smaller orgs and incorporated the learning (and made some pull requests, for example #69), and as I am approaching bigger organizations I need a more ergonomic way to control the rollout batches.

The idea of this patch is to have evergreen rollout pull requests / issues at every schedule run to a fixed amount of repositories, and have it organically increase towards full coverage after every following run.

[zkoppert]

Sounds great! Thanks for clarifying.

Let me know if you'd like support on resolving the ci failures.

[gbrindisi]

@zkoppert hey Zack yes please I could use some help.
Locally I run make test and make lint and it passes alright, I am not very familiar with mypy

[zkoppert]

I'm getting failing results locally for make test. 3 test failures.

FAILED test_env.py::TestEnv::test_get_env_vars_with_batch_size - AssertionError: Tuples differ: ('my_[251 chars]ndabot.yaml', None, False, [...
FAILED test_env.py::TestEnv::test_get_env_vars_with_no_batch_size - AssertionError: Tuples differ: ('my_[254 chars]bot.yaml', None, False, ['pr...
FAILED test_env.py::TestEnv::test_get_env_vars_with_repos_exempt_ecosystems - AssertionError: Tuples differ: ('my_[271 chars]False, ['private', 'public']...

I will take a look at the linter fixes first and come back to these.

[zkoppert]

Super minor changes to pass the checks! 😅 I'll cut a release shortly after merging.

Thank @gbrindisi for this awesome addition of batching!

[gbrindisi]

thanks @zkoppert !