Add a way to differentiate between server-returned and client-generated errors

[CAFxX]

Use case(s) - what problem will this feature solve?

We want to perform different telemetry actions (logging, metrics) depending on whether the error was generated by the client or the server.

Proposed Solution

Off the top of my head, add a function in status that returns different values based on whether the client or server generated the error, e.g. status.IsClientError(err) or status.ErrorSource(err) or anything else that we can use to distinguish the source of the error.

This is basically the same as #319 that was closed without resolution.

Alternatives Considered

Forcing all servers to include a well-known detail message in all responses. It's pretty impractical to enforce.

Additional Context

In #319 it looks like the argument was along the lines of "only Unknown can be returned by the client, and you should not return Unknown from the server". unfortunately this appear to be incorrect, since https://github.com/grpc/grpc/blob/master/doc/statuscodes.md has a big list of examples showing that almost all status codes (Unknown included) can be returned by both client and server for good reasons.

it would be ideal to further differentiate between errors returned by the server-library and server-application, but this may be done later.

[dfawley]

@ejona86 we talked about this awhile ago - does Java have a way to distinguish client-produced errors from server errors or connection errors?

[ejona86]

Java does not. There could be a bit included for locally-generated errors. It would take some thought but is probably feasible. I don't know if we want it though, as that level of source shouldn't matter. It is probably a symptom of another problem. I'll remind others that we don't know if a server generates a status, as proxies may generate errors.

We want to perform different telemetry actions (logging, metrics) depending on whether the error was generated by the client or the server.

I'd suggest explaining a bit more deeply at what you are trying to gain/understand by splitting these cases.

[CAFxX]

We want to perform different telemetry actions (logging, metrics) depending on whether the error was generated by the client or the server.

I'd suggest explaining a bit more deeply at what you are trying to gain/understand by splitting these cases.

We have gRPC proxies, and the team managing them has to respond very differently in case a proxied gRPC service is returning an error vs. if the proxy itself is generating errors when trying to communicate with the proxied gRPC services. In the former case, they 100% don't care at all. In the latter case it's an indication of a potential (not certain) misconfiguration somewhere. Obviously the latter case is orders of magnitude more rare than the former, so they must find a way to remove the noise.

To a lesser extent, the same applies to regular services as well, especially if these services have many dependencies, and especially if you are dealing with automated canary promotion.