small number of 500 Errors in production for FxOS

[jrgm]

There seems to be some number of 500 that are all listed as FxOS. In the most recent two cases, the stack was this one below, which offhand appears to be something failing Joi.

"stack":"Error: child \"email\" fails because [[object Object]]
    at Object.exports.create (/data/fxa-auth-server/node_modules/hapi/node_modules/boom/lib/index.js:21:17)
    at Object.exports.internal (/data/fxa-auth-server/node_modules/hapi/node_modules/boom/lib/index.js:252:92)
    at Object.exports.badImplementation (/data/fxa-auth-server/node_modules/hapi/node_modules/boom/lib/index.js:288:23)
    at postValidate (/data/fxa-auth-server/node_modules/hapi/lib/validation.js:194:26)
    at internals.Any._validateWithOptions (/data/fxa-auth-server/node_modules/hapi/node_modules/joi/lib/any.js:601:16)
    at root.validate (/data/fxa-auth-server/node_modules/hapi/node_modules/joi/lib/index.js:102:23)
    at exports.response (/data/fxa-auth-server/node_modules/hapi/lib/validation.js:213:20)
    at /data/fxa-auth-server/node_modules/hapi/lib/request.js:331:13
    at iterate (/data/fxa-auth-server/node_modules/hapi/node_modules/items/lib/index.js:35:13)
    at done (/data/fxa-auth-server/node_modules/hapi/node_modules/items/lib/index.js:27:25)
    at /data/fxa-auth-server/node_modules/hapi/node_modules/hoek/lib/index.js:796:22
    at process._tickDomainCallback (node.js:492:13)"}}

"agent":"Mozilla/5.0 (Mobile; Orange KLIFD; rv:32.0) Gecko/32.0 Firefox/32.0"

[jrgm]

Putting this next. Chronic 500 errors mask the existence of other bugs.

[rfk]

I haven't been able to find an obvious cause for this, but I suspect some error occurring in out custom validation code for email addresses. From kibana it looks like all the 500s are on GET /v1/recovery_email/status requests, @jrgm can you confirm?

[rfk]

I've been periodically checking out kibana since this was filed, and I haven't seen any further re-occurrences of this. All the 500s I could see have been on /v1/recovery_email/status and have had a variety of different user-agents, most recently a Mac on Oct 21st.

Interestingly, this is a read-only endpoint, so if there's an email validation failure it's occurring in the response schema checking, against a value that we already accepted and stored in the db.

Given the variety of user-agents that trigger this, the nature of the endpoint on which we're seeing it, and the fact that we haven't seen any more in the last seven days, I don't think there's much more we can do with this bug. I'm going to close it out, but @jrgm please re-open with additional tracebacks if we get more of them.