Support multiple regions/availabilty zones if an instance type cannot be started

[gilShin]

We are using instances types that require lots of reasources. Because of, from time to time it is impossible to start an instance of some type in a certain region and a certain AZ.
It'll be great if this plugin will be able to run over list of regions/AZs (subnets and security groups) and start the instance where its type is available.

[stas00]

I just came to file the same request, getting 95% of the time this:

Error: AWS EC2 instance starting error
Error: InsufficientInstanceCapacity: We currently do not have sufficient p3.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get p3.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b.
Error: We currently do not have sufficient p3.8xlarge capacity in the Availability Zone you requested (us-east-2a). Our system will be working on provisioning additional capacity. You can currently get p3.8xlarge capacity by not specifying an Availability Zone in your request or choosing us-east-2b.

and this tool fails if I remove aws-region from the config file.

Needing this for a CI, so this is a big issue.

Thank you!

[philschmid]

To support multi-AZ We could make config.input.subnetId a list and try to iterate randomly over it here and stop when an instance started.
I think multi-Region is hard to implement since Security groups, AMIs, etc are always bound to a Region.

[gilShin]

Sounds like a very good idea Thanks!

…

On Fri, Aug 27, 2021, 09:55 Philipp Schmid ***@***.***> wrote: We could make config.input.subnetId a list and try to iterate randomly over it here <https://github.com/machulav/ec2-github-runner/blob/8911518f6d7d3cbfc7de6cf9ecac0931f8025798/src/aws.js#L35> and stop when an instance started. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#60 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAULLE5TX6QKIUH46I5DDRLT64ZFLANCNFSM5BZWQ76A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jpalomaki]

FWIW, when using EC2 launch templates, one can apparently omit the explicit selection of AZ/subnet.

It might be worth it to investigate/test if this facility would imply intelligent automatic selection of AZ (taking capacity into account).

[jpalomaki]

Turns out one can only omit subnet ID from run parameters, if you are using the default VPC. Per related AWS docs:

[EC2-VPC] If you don't specify a subnet ID, we choose a default subnet from your default VPC for you. If you don't have a default VPC, you must specify a subnet ID in the request.

[gilShin]

If I do use the default vpc, will it try in one AZ or will keep on trying in all AZs? Anyhow I would like that to be across regions as well. I did solve it using a very ugly workaround but seems like working. In the start job, I have added a step that if it fails, it triggers the workflow again with different subnet ID. The subnet is of the start step uses an env variable that uses an input parameter.

…

On Mon, Aug 30, 2021, 21:39 Jukka Palomäki ***@***.***> wrote: Turns out one can only omit subnet ID from run parameters, if you are using the default VPC. [EC2-VPC] If you don't specify a subnet ID, we choose a default subnet from your default VPC for you. If you don't have a default VPC, you must specify a subnet ID in the request. See AWS docs <https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_RunInstances.html> — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#60 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAULLE2MPVD3PKELGNZXOITT7PF6RANCNFSM5BZWQ76A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jpalomaki]

If I do use the default vpc, will it try in one AZ or will keep on trying
in all AZs?

I don't know, but I suspect it would just pick one. What I'd like to know is whether the automatic subnet selection algorithm is AZ capacity-aware or not.

[gilShin]

I doubt that and I'll explain why. Capacity of what? Vcpu? Mem? GPU? You can have enough vcpus in a certain AZ but no GPUs. I think it just picks up a subnet randomly and hope for the best. For us it is even worst because we are using mostly GPUs and not all AZs in a certain region has that type that we are looking for.

…

On Mon, Aug 30, 2021, 22:18 Jukka Palomäki ***@***.***> wrote: If I do use the default vpc, will it try in one AZ or will keep on trying in all AZs? I don't know, but I suspect it would just pick one. What I'd like to know is if the automatic selection algorithm is AZ capacity-aware. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#60 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAULLE3G443Y42KOWGTW5D3T7PKR3ANCNFSM5BZWQ76A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jpalomaki]

All needed capacity. AWS must be checking them anyhow, at instance launch time, to be able to distribute/optimize load.

Looking at https://aws.amazon.com/premiumsupport/knowledge-center/ec2-insufficient-capacity-errors/ and then https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/troubleshooting-launch.html#troubleshooting-launch-capacity, the latter link does mention "not specifying AZ" as one solution. 🤷‍♂️

That said, one option could be to abstract have the runner start job in a composite action that tries to try ec2-github-runner start with different region/AZ settings (in consecutive, conditional steps) until it succeeds, returning the start outputs from the first successful step?

EDIT: amended workaround proposal because it looks like composite actions do not support if conditions

[gilShin]

Can be a solution but I think it'll be best if the code can go over list of regions and AZs and try in each and every one of them. I did something similar in Jenkins pipeline

…

On Mon, Aug 30, 2021, 23:31 Jukka Palomäki ***@***.***> wrote: All needed capacity. AWS must be checking them anyhow, at instance launch time, to be able to distribute/optimize load. Looking at https://aws.amazon.com/premiumsupport/knowledge-center/ec2-insufficient-capacity-errors/ and then https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/troubleshooting-launch.html#troubleshooting-launch-capacity, the latter link does mention "not specifying AZ" as one solution. 🤷‍♂️ That said, one option could be to abstract the runner start job in a composite action <https://docs.github.com/en/actions/creating-actions/creating-a-composite-action> that tries ec2-github-runner start with different region/AZ settings (in consecutive, conditional steps) until it succeeds, returning the start mode outputs from the successful step? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#60 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAULLE4QPMTAGC4AXAZP56DT7PTADANCNFSM5BZWQ76A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[stas00]

And while we are at it, when it fails to start because of the region not having an instance I get a while bunch of errors on stopping:

[jpalomaki]

And while we are at it, when it fails to start because of the region not having an instance I get a while bunch of errors on stopping

That can probably be fixed by amending the if condition on the stop job (to also check the result of the start job)

[jpalomaki]

@stas00 would something like this work? (NOTE: untested and some details omitted for brevity):

jobs:
  start-runner:
    runs-on: ubuntu-20.04
    outputs:
      label: ${{ steps.result.outputs.label }}
      ec2-instance-id: ${{ steps.result.outputs.ec2-instance-id }}
    steps:
      ...
      - id: try-us-east-2a
        uses: machulav/ec2-github-runner@v2
        continue-on-error: true
        with:
          mode: start
          subnet-id: subnet-on-2a
          ...
      - id: try-us-east-2b
        if: steps.try-us-east-2a.outcome == 'failure'
        uses: machulav/ec2-github-runner@v2
        continue-on-error: true
        with:
          mode: start
          subnet-id: subnet-on-2b
          ...
      - id: try-us-east-2c
        if: steps.try-us-east-2a.outcome == 'failure' && steps.try-us-east-2b.outcome == 'failure'
        uses: machulav/ec2-github-runner@v2
        with:
          mode: start
          subnet-id: subnet-on-2c
          ...
      - id: result
        run: |
          if [ "${{ steps.try-us-east-2a.outcome }}" = "success" ]; then
            echo "::set-output name=label::${{ steps.try-us-east-2a.outputs.label }}"
            echo "::set-output name=ec2-instance-id::${{ steps.try-us-east-2a.outputs.ec2-instance-id }}"
          fi
          if [ "${{ steps.try-us-east-2b.outcome }}" = "success" ]; then
            echo "::set-output name=label::${{ steps.try-us-east-2b.outputs.label }}"
            echo "::set-output name=ec2-instance-id::${{ steps.try-us-east-2b.outputs.ec2-instance-id }}"
          fi
          if [ "${{ steps.try-us-east-2c.outcome }}" = "success" ]; then
            echo "::set-output name=label::${{ steps.try-us-east-2c.outputs.label }}"
            echo "::set-output name=ec2-instance-id::${{ steps.try-us-east-2c.outputs.ec2-instance-id }}"
          fi

Also, if we supported launch templates (see #65), then multi-region would be less awkward as we could have a template per region (also less inputs per ec2-github-runner step)?

EDIT: fixed the if condition on the try-us-east-2c step to only run if both previous tries failed.

[stas00]

Thank you for writing out a concrete example, @jpalomaki! I will try and report back.

So it's the same sub-region we are using the same AMI /security group, but only rotating though the subnet-ids. Got it.

[jpalomaki]

Thank you for writing out a concrete example, @jpalomaki! I will try and report back.

So it's the same sub-region we are using the same AMI /security group, but only rotating though the subnet-ids. Got it.

Np.

NOTE: I now fixed the if condition on the try-us-east-2c step so that it only runs if both previous steps failed. Also be sure to monitor how many instances/runners are actually spawned, to avoid excess costs.

[stas00]

So I have done quite a few experiments and your example works perfectly well, @jpalomaki - thank you so much.

I switched to us-east-1 since us-east-2 didn't even have 3 subnets that supported the instance type I needed. On us-east-1 I think there are only 3.

It'd be nice to be able to remove the repetition of aws-resource-tags and other sections if at all possible. If not, that's OK too.

The final config is here:

https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/9e14c02a1dd22e4d36e2ee9a33e44d33774b8de7/.github/workflows/main.yml#L17

[gilShin]

So is it going to support only different AZs but not different regions? Thanks

…

On Wed, Sep 1, 2021, 07:41 Stas Bekman ***@***.***> wrote: So I have done quite a few experiments and your example works perfectly well, @jpalomaki <https://github.com/jpalomaki> - thank you so much. I switched to us-east-1 since us-east-2 didn't even have 3 subnets that supported the instance type I needed. On us-east-1 I think there are only 3. It'd be nice to be able to remove the repetition of aws-resource-tags and other sections if at all possible. If not, that's OK too. The final config is here: https://github.com/bigscience-workshop/Megatron-DeepSpeed/blob/9e14c02a1dd22e4d36e2ee9a33e44d33774b8de7/.github/workflows/main.yml#L17 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#60 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAULLE6U5J7YFN65XDN6USTT7WVHNANCNFSM5BZWQ76A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[stas00]

this particular setup is just sub-regions.

I'm concerned this might not be enough still at times. So different regions would be even more fault tolerant, but this is already a great start.

[gilShin]

I agree that this is a good start but with the types we're using, unfortunately, it won't be enough :-(

…

On Wed, Sep 1, 2021, 08:24 Stas Bekman ***@***.***> wrote: this particular setup is just sub-regions. I'm concerned this might not be enough still at times. So different regions would be even more fault tolerant, but this is already a great start. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#60 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAULLE53MJIJFANPPGKODVTT7W2KBANCNFSM5BZWQ76A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[machulav]

@stas00, great that you've made it to work!
@jpalomaki, thank you a lot for your help and examples! I really appreciate your help here!

I hope that the launch templates from #65 will make selecting the AZ from the region less painful in the future. Maybe it will be helpful also for the region selection when you have one launch template in each region and then try to create instances in each region one by one using the example from @jpalomaki.

But for now, I close the issue as we have the solution. Let's monitor the situation and re-open it if required.

[mgaitan]

Hey, I've implemented the idea of @philschmid

We could make config.input.subnetId a list and try to iterate randomly over it

Check this
Shiphero@69a9f36