Allow spot requests in multiple availability zones #76
Comments
|
Hmm, docker/machine#4701 (and docker/machine#4588) looks quite relevant / concerning... |
|
@declension you can spin now a runner in multiple zones, would this help see #77 and PR #96 |
|
Thanks @npalm, we'll try this soon. Closing for now |
This was referenced Mar 30, 2023
kayman-mk
added a commit
that referenced
this issue
Apr 21, 2023
… us to use multiple instance types and AZs (#777) ## Description The [fork](https://gitlab.com/cki-project/docker-machine/-/tree/v0.16.2-gitlab.19-cki.2) of docker-machine used in this module has released a version that supports [AWS Spot Fleet](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet.html) 3 months ago. In our case, spot fleets are useful when using spot instances because it fixes the issue when there is no capacity available for a certain instance type in a certain AZ. With spot fleets, we can use multiple instance types and AZ so when there is no capacity on one instance type, AWS will take the next instance type and so on. This fixes issues [#76](#76) [#435](#435) [#77](#77) and [#474](#474) ## Migrations required NO our solution is backward compatible so if you don't plan to use spot fleets, no migration is required. YES if you want to use spot fleets you will have to do the following changes: `docker_machine_instance_type` to `docker_machine_instance_types` `subnet_id` to `subnet_ids` You also have to put these 2 new parameters: `use_fleet` and `key_pair_name` ### Important We have seen that the [fork](https://gitlab.com/cki-project/docker-machine/-/tree/v0.16.2-gitlab.19-cki.2) of docker-machine this module is using consume more RAM using spot fleets. For comparison, if you launch 50 machines in the same time, it consumes ~1.2GB of RAM. In our case, we had to change the `instance_type` of the runner from `t3.micro` to `t3.small`. ## Verification ```hcl docker_machine_instance_types = ["r5a.xlarge", "r5.xlarge", "r5ad.xlarge", "r5d.xlarge", "m5a.2xlarge"] use_fleet = true key_pair_name = "<key-pair-name>" ``` We are using this solution in production for a week now with 150+ developers and over 20 000 jobs a day without any issue for now. --------- Signed-off-by: François Bibron <francois.bibron@polyconseil.fr> Co-authored-by: François Bibron <francois.bibron@polyconseil.fr> Co-authored-by: Marvin Wichmann <marvin@fam-wichmann.de> Co-authored-by: Matthias Kay <github-public@matthiaskay.de> Co-authored-by: kayma <kayma@hlag.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Background
At $WORK, we have a pre-defined VPC, and one subnet within that (but want more to avoid any further capacity issues) for Gitlab & runners.
Having multiple AZs is entirely possible, as the spot request wizard has checkboxes for each:
Proposal
aws_zone) can be specified as parameters, orsubnet_ids_runnersperhaps?) in various AZs, with the AZ inferred automatically, so deprecatingaws_zoneperhaps?Notes
I don't know if this necessarily makes sense, but am definitely keen to find ways to improve our
capacity-not-availableproblems (not pricing), and this seems like one major causeThe text was updated successfully, but these errors were encountered: