Migrate to new ci runners #7082
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
matipan
force-pushed
the
migrate-to-new-ci-runners
branch
from
93bd8a4
to
412358b
Compare
matipan
force-pushed
the
migrate-to-new-ci-runners
branch
17 times, most recently
from
9a08f48
to
e87002c
Compare
gerhard
approved these changes
Apr 17, 2024
matipan
force-pushed
the
migrate-to-new-ci-runners
branch
3 times, most recently
from
23690ba
to
8ff9744
Compare
gerhard
reviewed
Apr 18, 2024
gerhard
reviewed
Apr 18, 2024
matipan
force-pushed
the
migrate-to-new-ci-runners
branch
2 times, most recently
from
e6537ad
to
3083e9e
Compare
gerhard
force-pushed
the
migrate-to-new-ci-runners
branch
from
660ad83
to
e31aa58
Compare
This was referenced Apr 25, 2024
|
Minor request - could we link somewhere in our docs here where we can find the definition for the github workers in https://github.com/dagger/dagger.io? (even though it's private) |
|
Todo:
Follow-up:
|
|
I'm picking this one up again. |
gerhard
force-pushed
the
migrate-to-new-ci-runners
branch
from
7892bf0
to
f72cc50
Compare
…engine This one requires Docker with specific fixes that we don't yet have in the new CI setup. Signed-off-by: Gerhard Lazu <gerhard@dagger.io>
The setup we want for production is: - For all <LANG> SDK jobs, run them on the new CI only - For testdev, run them on the docker-fix legacy CI - For test/dagger-runner, run them on both legacy CI and new CI - For all the rest, run them on the new CI and github runners for the really simple jobs Signed-off-by: Matias Pan <matias@dagger.io>
Signed-off-by: Matias Pan <matias@dagger.io>
Signed-off-by: Matias Pan <matias@dagger.io>
Signed-off-by: Matias Pan <matias@dagger.io>
Signed-off-by: Matias Pan <matias@dagger.io>
Signed-off-by: Matias Pan <matias@dagger.io>
Signed-off-by: Matias Pan <matias@dagger.io>
Otherwise the workflows are too slow on the new CI runners and are blocking the migration off the legacy CI runners. Signed-off-by: Gerhard Lazu <gerhard@dagger.io>
Signed-off-by: Gerhard Lazu <gerhard@dagger.io>
gerhard
force-pushed
the
migrate-to-new-ci-runners
branch
from
f72cc50
to
cc7c562
Compare
Signed-off-by: Gerhard Lazu <gerhard@dagger.io>
Large GitHub Runners are failing consistently, not worth debugging at this point since we know this works on a vanilla Ubuntu 24.04 instance with Docker - must be an issue related to GitHub Large Runners. FTR: dagger#7223 (comment) Signed-off-by: Gerhard Lazu <gerhard@dagger.io>
vikram-dagger
pushed a commit
to vikram-dagger/dagger
that referenced
this pull request
May 8, 2024
* Use Dagger v0.11.2 via the new CI setup for all workflows except dev-engine This one requires Docker with specific fixes that we don't yet have in the new CI setup. Signed-off-by: Gerhard Lazu <gerhard@dagger.io> * Setup CI for new, legacy and vertical scaling The setup we want for production is: - For all <LANG> SDK jobs, run them on the new CI only - For testdev, run them on the docker-fix legacy CI - For test/dagger-runner, run them on both legacy CI and new CI - For all the rest, run them on the new CI and github runners for the really simple jobs Signed-off-by: Matias Pan <matias@dagger.io> * Rename concurrency group Signed-off-by: Matias Pan <matias@dagger.io> * Install curl on production vertical scaling runner Signed-off-by: Matias Pan <matias@dagger.io> * Add customizable runner for separate perf tests Signed-off-by: Matias Pan <matias@dagger.io> * Rename to _async_hack_make Signed-off-by: Matias Pan <matias@dagger.io> * Upgrade missing workflow to v0.11.1 Signed-off-by: Matias Pan <matias@dagger.io> * Target nvme Signed-off-by: Matias Pan <matias@dagger.io> * CI: Default to 4CPUs & NVMe disks Otherwise the workflows are too slow on the new CI runners and are blocking the migration off the legacy CI runners. Signed-off-by: Gerhard Lazu <gerhard@dagger.io> * Bump to v0.11.2 & capture extra details in comments Signed-off-by: Gerhard Lazu <gerhard@dagger.io> * Debug dagger-engine.dev in large GitHub Runner Signed-off-by: Gerhard Lazu <gerhard@dagger.io> * Continuer running engine:testdev in dagger-runner-docker-fix runner Large GitHub Runners are failing consistently, not worth debugging at this point since we know this works on a vanilla Ubuntu 24.04 instance with Docker - must be an issue related to GitHub Large Runners. FTR: dagger#7223 (comment) Signed-off-by: Gerhard Lazu <gerhard@dagger.io> --------- Signed-off-by: Gerhard Lazu <gerhard@dagger.io> Signed-off-by: Matias Pan <matias@dagger.io> Co-authored-by: Gerhard Lazu <gerhard@dagger.io> Co-authored-by: Matias Pan <matias@dagger.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This switches all our GitHub workflows to use the second generation of Dagger Runners. They are based on Run Dagger on Amazon EKS with GitHub Actions Runner and Karpenter, currently running:
v1.28.7v0.32.0As of today, we expose the following runner options:
dagger-v0-10-1dagger-v0-10-2dagger-v0-10-3dagger-v0-10-3-4cdagger-v0-10-3-8cdagger-v0-10-3-16cdagger-v0-11-0dagger-v0-11-0-4cdagger-v0-11-0-8cdagger-v0-11-0-16cdagger-v0-11-1dagger-v0-11-1-4cdagger-v0-11-1-4c-nvmedagger-v0-11-1-8cdagger-v0-11-1-8c-nvmedagger-v0-11-1-16cdagger-v0-11-1-16c-nvmedagger-v0-11-2dagger-v0-11-2-4cdagger-v0-11-2-4c-nvmedagger-v0-11-2-8cdagger-v0-11-2-8c-nvmedagger-v0-11-2-16cdagger-v0-11-2-16c-nvmeAs soon as a new version of Dagger gets released - e.g.
v0.11.3- a bunch of variants with that version will appear automatically.This change is great because:
To learn more, see the following (private) infra repo. @matipan @gerhard that have all the context.
Something worth mentioning separately is that this also introduces asynchronous triggers.
We are running a specific job -
engine:testrace- in a production bare-metal K8s cluster. This is done for comparison reasons only since we know that it's going to fail (we only have 12CPU threads available). See #7223 (comment) for more details.The goal of this is to capture measurements - #6492 - and see how they change over time.
FTR, this job will fail until we merge into
mainsince the new workflow will not be recognised:Split out to avoid blocking the release: