load_gen: Investigate "failed to auth" errors showing up at regular intervals

[jbonacci]

Today, 12/20, started up a load test on Stage around 11:30am PST.

bin/load_gen -s https://stage-browserid.services.mozilla.com -o -m 500000 -u 1/500

After one initial spike of errors around 11:50am PST, I am starting to see authorization failures at fairly regular (consistent) intervals:
error: failed to authenticate: non-success response 200 - {"success":false,"reason":"no such user"}
501120.00 514036.19 792156.50 46 R, 232 S (2sp 1rs 1al 0rh 42sn 0iy 0cs) (1 ERRORS!)

or

error: failed to authenticate: non-success response 200 - {"success":false,"reason":"mismatch - pass:lxuGFYE3Bo - email:EnCnPlCG@OsPzlOvm.GQE - hash:$2a$12$dGsuShWCIRW1XuclkwP7EeN/cbiO5zarTYOYBVdqhqfSd4.v.u7T6"}
492480.00 501345.91 515317.68 63 R, 232 S (3sp 0rs 0al 3rh 57sn 0iy 0cs) (1 ERRORS!)

or similar

[lloyd]

reopening this bug, still seeing:

error: (signin) failed to authenticate: non-success response 200 - {"success":false,"reason":"no such user"}

Steps to reproduce? Turn up the load until things fall over (requests start failing), then back off and leave several clients running. They'll be left in this state where they have users in their database that the server doesn't know about.

[jbonacci]

We will verify this in Stage this week with load_gen.

[jbonacci]

Not really seeing this anymore...