oauth and profile cpu is high for the actual request/sec

[jrgm]

The profile (and to a lesser degree, oauth) server doesn't serve that many rps, but cloudwatch metrics show cpu as running steady at 85% during peak demand. Not sure where the cycles are going.

Note though: it's odd because top, Cloudwatch and Datadog report significantly different levels of cpu, with top showing much lower levels.

[jbuck]

This is still a problem. oauth and profile currently use m3.mediums so an easy test would be to try different instance types to see how they perform under load.

Instance	Memory	Compute Units	vCPUs	Cost (per hour)	Cost (per month)
m3.medium	3.75 GB	3 units	1 vCPUs	$0.067	$50.25
t2.medium	4.0 GB	40% base performance	2 vCPUs for a 4h 48m burst	$0.047	$35.25
t2.large	8.0 GB	60% base performance	2 vCPUs for a 7h 12m burst	$0.094	$70.50
c4.large	3.75 GB	8 units	2 vCPUs	$0.100	$75.00
m4.large	8.0 GB	6.5 units	2 vCPUs	$0.108	$81.00

Simultaneously we could be looking at profiling the oauth or profile servers to see if there are any low-hanging fruit to fix. Someone suggested in the meeting that the profile server does 4 HTTP requests for each request it recieves - something to optimize there?

[jrgm]

[04-28 13:50:07] vladikoff jrgm, were there any good results when you
  switched the type of aws boxes for https://github.com/mozilla/fxa-profile-server/issues/181 ?
[04-28 13:52:50] jrgm vladikoff: ah, yes, there were. I didn't write
  them up at the time, but looked like a net cost and performance win
  using (I think it was) m4.large.
[04-28 13:53:22] jrgm vladikoff: but it's largely a sunk cost for this
  year (reserved instances).
[04-28 13:53:36] vladikoff ah yes
[04-28 13:54:46] jrgm vladikoff: Probably even more of a win if we used
  ran more than one daemon per box (using cluster, of some other scheme).

[vladikoff]

Let's reopen or create a new this if we need to do more work on optimizing this