test: Run colima tests native instead of github, fixes #5598 #5886
Conversation
|
Download the artifacts for this pull request:
See Testing a PR |
|
Both colima tests (vz and qemu) ran in about an hour and a half (and succeeded). Yay! |
This reverts commit 3b47419.
There was a problem hiding this comment.
Looks great to me!
And I think we can add comments to all buildkite config yml files with clear info about where it can be run at the top of the file, without using the branches: stanza, to understand the expected behavior for each test:
For example:
add to .buildkite/macos-orbstack.yml
# See https://buildkite.com/ddev/macos-orbstack/settings/repository
# Runs on master, PRs, and 3rd party PRs
This is not ideal, of course, but it gives us a better understanding of the test, and when we return to it in six months or a year, we can already see how it should behave.
rfay
force-pushed
the
20240222_colima_buildkite
branch
from
d56ccba
to
6fe7779
Compare
|
Added suggested comments to buildkite builds and made some adjustments so we don't see builds that don't happen. I think we may not be using buildkite entirely right and may be able to improve how we're doing it. |
|
Right now we're running 3 tests on every PR (colima, colima_vz, orbstack) that require tb-macos-arm64-(5-6-7), so they're fully occupied for one PR. On master, we're running 4 (colima, colima_vz, orbstack, rancher) I think we'll probably want to dial back one of the colima ones soon. |
The Issue
We have never been able to get predictable results from colima tests on Github Actions. Maybe the VMs are too small
Also, we were never able to do normal https tests, so there are a bunch of things that require http, we might be able to remove those.
How This PR Solves The Issue
Run the builds on our larger buildkite test machines.
Detail
Testing by running TestExtraPortExpose with:
GOTEST_SHORT=true make testpkg TESTARGS="-run TestExtraPortExpose -count=1"It seems to work after a
colima startand then acolima restart. And then it stays working. So weird.