Treatment of errors in Orion

[staceytay]

Context

At present all errors from a service are treated as errors on Orion and so treated as errors on New Relic. However, for SSO, we have a use case where we would like to ignore HTTP 401s for the login endpoints (this is the case when a user gives incorrect login credentials) and treat that as a non-error.

I tried to do this by creating a custom decoder for login, but that didn't work because it's not possible to change the endpointError variable from the decoder. At present, I can't think of a clean way to send a HTTP 401 to clients and still not treat that as an error.

Impact

NR is reporting a 20% error rate for SSO on prod, but they're mostly rpc error: code = Unauthenticated desc = Unauthorized took 71.696791ms errors which are used to return a 401, making it hard to detect legitimate errors on NR.

Proposed Solution

One workaround that I can think of is to ignore such errors in the New Relic config, as in https://docs.newrelic.com/docs/agents/go-agent/configuration/go-agent-configuration#error-ignore-status. Hence, it'll be great if Orion could provide an API or some way for services to customise this.

[ankurs]

This is WIP!

[ankurs]

fixed,
call modifiers.DontLogError(ctx) from you function, before throwing error