-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cirrus: use faster VM's for integration tests #22698
Conversation
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: Luap99 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Ephemeral COPR build failed. @containers/packit-build please check. |
Cockpit tests failed for commit 79bd109. @martinpitt, @jelly, @mvollmer please check. |
Cockpit tests failed for commit f5f1f74. @martinpitt, @jelly, @mvollmer please check. |
Timings with 4 core 4 GB
int tests make great use of the extra cpu cores here 15-17 min vs 27-30 min on main right now |
However it doesn't seem to help at all for system tests, which also makes sense as they run in sequence not in parallel so extra cores do not help |
Use 4 core VM vompred to the standard 2 cores, integration tests scale almost linear with extra cores, as such doubling the cores makes the tests almost twice as fast. This brings the test time down to 15-17 min in CI. Signed-off-by: Paul Holzinger <pholzing@redhat.com>
I made the change to only effect the integration tasks as other tests does not seem to benifit from more cores and it would be a waste of money to use more cores there. I leave it as draft until I get approval from @cevich. |
The approach I've been working on (side branch) for speeding up system tests is:
I'm focusing on the local-registry thing today and probably next week, because I see that as higher ROI. |
Does this cost us seriously more money? Or even Less? IE if E2E tests complete in X time for Y Dollars/time, is this less or greater then before and how much difference? If it ain't much we should make the change, or at least get Red Hat an estimate of additional cost. Saving time on tests helps developers a great deal especially when we get near releases. |
Yes, agreed. Note right now I focus only on e2e per the current Jira card (https://issues.redhat.com/browse/RUN-2107) |
I don't know how the pricing structure works, but assuming double the cores double the price we now only need half the time so it should be not significantly more expensive IMO. But that only goes for integration tests which this is about. The pricing comment is about tests that do not scale at all see timings comment above. There is absolutely no point in running system tests on higher core count VM's right now as we would pay more for no time gain. |
Ephemeral COPR build failed. @containers/packit-build please check. |
Latest change (47f01e8) LGTM |
int test timings are phenomenal. /lgtm |
@Luap99 Can you remove Draft so this will merge. |
@Luap99 this LGTM.
The time developers spend waiting for tests to complete is FAR more expensive than the VMs. Granted this is also hard to measure, since developers aren't likely just sitting on their hands waiting. In either case, there are cost-increases built-in to cloud-spend budgeting. It's incredibly rare that testing is reduced and/or cloud-costs go down magically. That's all to say, I'm not concerned at all about the cost increase by this PR. |
d85d563
into
containers:main
Use 4 core VM vompred to the standard 2 cores, integration tests scale
almost linear with extra cores, as such doubling the cores makes the
tests almost twice as fast. This brings the test time down to 15-17 min
in CI.
Does this PR introduce a user-facing change?