Skip to content
This repository was archived by the owner on Apr 14, 2023. It is now read-only.

Conversation

@gravesm
Copy link

@gravesm gravesm commented Oct 11, 2019

Adding autoscaling to ECS for the worker service requires a few changes
to how the cluster is redeployed. Previously, everything was stopped by
setting the desiredCount to 0 and then started back up by setting the
desiredCount to 1. With ECS controlling the scaling for the worker
service it's no longer possible to rely on the desiredCount of the
worker service remaining at 0 until things are ready to be started
again.

The change here sets the desiredCount of the scheduler to 0 first, then
does a --force-new-deployment for the web and worker services, then
finally does a --force-new-deployment for the scheduler. While
redeploying becomes more complicated, there's no need anymore to track
the current worker count. We can just set it to 1 and let ECS scale as
needed.

I've also added stop-scheduler and start-scheduler commands which can be
used in conjunction with deploying a new service definition with
Terraform.

How can a reviewer see these changes?

You can run:

$ pipenv run workflow redeploy

and verify that the command completes without blowing up.

Reviewer Checklist

  • The commit message is clear and follows our guidelines
  • There are tests covering any new functionality
  • The documentation has been updated if necessary
  • The changes, if applicable, have been verified

Adding autoscaling to ECS for the worker service requires a few changes
to how the cluster is redeployed. Previously, everything was stopped by
setting the desiredCount to 0 and then started back up by setting the
desiredCount to 1. With ECS controlling the scaling for the worker
service it's no longer possible to rely on the desiredCount of the
worker service remaining at 0 until things are ready to be started
again.

The change here sets the desiredCount of the scheduler to 0 first, then
does a --force-new-deployment for the web and worker services, then
finally does a --force-new-deployment for the scheduler. While
redeploying becomes more complicated, there's no need anymore to track
the current worker count. We can just set it to 1 and let ECS scale as
needed.

I've also added stop-scheduler and start-scheduler commands which can be
used in conjunction with deploying a new service definition with
Terraform.
@gravesm gravesm merged commit 5ed6dbc into master Oct 11, 2019
@gravesm gravesm deleted the worker-autoscale branch October 11, 2019 17:45
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants