Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor worker nodes into ECS #29

Closed
3 tasks
brietaylor opened this issue Apr 9, 2020 · 0 comments
Closed
3 tasks

Refactor worker nodes into ECS #29

brietaylor opened this issue Apr 9, 2020 · 0 comments
Labels
enhancement New feature or request Terraform

Comments

@brietaylor
Copy link
Collaborator

brietaylor commented Apr 9, 2020

There are a couple of nuisances with our current worker strategy, that I think would be helped by moving most of what we've done to an orchestration system like ECS.

  1. Log streams are currently generated by instance, so the logs from all N workers get interleaved (making it hard to find errors)
  2. When a worker crashes, it never gets replaced.
  3. Updating the container images is a right pain. docker kill, docker rm, docker pull, find / -name part-001, (cloud-init script) /path/to/part-001. vs. pushing a new launch template and having fresh images in a couple of minutes.
  4. Ugly names for the ASGs (means we have to "discover" the names to do adjust desired sizes), like tf-asg-tf-serratus-dl-20200304125312000001, this is currently necessary, so that all instances get replaced when we change the user_data in the launch configuration, ECS would deal with sending the correct arguments to our scripts.

There are a couple things to work out though, first:

  • will we use Daemon or Replication jobs? Daemon doesn't solve 1, but replication doesn't solve 4. We need a way to force all images to be replaced if we change them.
  • ECS + Cloudwatch Logs
  • ...and more, maybe?
@brietaylor brietaylor added Terraform enhancement New feature or request labels Apr 9, 2020
@mathemage mathemage added this to Open Tasks in TODO List via automation Apr 18, 2020
@ababaian ababaian closed this as completed Dec 9, 2020
TODO List automation moved this from Open Tasks to Completed Tasks Dec 9, 2020
@ababaian ababaian removed this from Completed Tasks in TODO List Dec 9, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Terraform
Projects
None yet
Development

No branches or pull requests

2 participants