Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Audit task memory/cpu reservations #120

Closed
cavis opened this issue Oct 2, 2017 · 5 comments
Closed

Audit task memory/cpu reservations #120

cavis opened this issue Oct 2, 2017 · 5 comments
Assignees
Labels
Milestone

Comments

@cavis
Copy link
Member

cavis commented Oct 2, 2017

Sanity check and update the Memory/CPU Reservations for all our tasks. Base on how many tasks we generally want running on an m4.large instance.

Also take into account disk usage by dovetail and the nginx-cache. Which may indicate we don't want > 2 dovetails per server in any case.

@cavis cavis changed the title Audit service DesiredCounts Audit task memory/cpu reservations Oct 6, 2017
@cavis cavis self-assigned this Oct 10, 2017
@cavis cavis added the high label Oct 10, 2017
@cavis cavis added this to the Sprint Uwe milestone Oct 10, 2017
@cavis
Copy link
Member Author

cavis commented Oct 10, 2017

Prerequisite for #10.

@cavis
Copy link
Member Author

cavis commented Oct 13, 2017

Current configurations:

Type CPU Capacity Memory Capacity
m4.large 2048 7985
t2.medium 1024 2001
Task Def CPU Reservation Memory Reservation
Castle 128 128
Cms (web/work) 128 778
Crier (web/work) 128 500
Dovetail 700 600
DTProxy 256 1000
FourOhFour 16 128
Feeder (web/work) 128 500
Id 128 512
Jingle 128 500
Metrics 128 500
Play 128 500
Publish 128 500

Based on this, i think it makes sense to:

  1. Switch all/most staging reservations to 128/400 (we don't really care about staging), and turn some of the desired-counts down to 1.
  2. Set prod reservations to:
    rails/elixir -> 200/1400
    node/frontend -> 100/400
    dovetail -> 700/1000
  3. Figure out auto-scaling rules based on that.

@cavis
Copy link
Member Author

cavis commented Oct 13, 2017

Some example full-instances based on that:

2x DT, 3x rails (cpu limit)
2x DT, 2x rails, 2x node (cpu limit)
1x DT, 4x rails, 3x node (mem limit)
1x DT, 3x rails, 6x node (mem limit)

@farski
Copy link
Member

farski commented Oct 13, 2017

That's all seems reasonable

@cavis
Copy link
Member Author

cavis commented Oct 17, 2017

Bumped staging rails memory up to 650, because they kept crashing when i tried to run a rails console. Appears to be better now. Fixed!

@cavis cavis closed this as completed Oct 17, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants