Allow gRPC error code config similar to HTTP

[ghost]

The dd.http.client.error.statuses config allows you to set a range of status codes that won't be reported as errors and, by default, 4xx errors aren’t reported. A similar config for gRPC so codes like Canceled, NotFound, etc. don't report as errors should be generally helpful. dd-trace-go added support in March, '19, DataDog/dd-trace-go#414.

[jeremy-lq]

@rockywarren-vertex We have a couple questions about your use-case. Can you open a support ticket and we'll reach out? Thanks.

[ghost]

Received this response from support,

While we don't currently support customizing status codes for gRPC today, I've gone ahead and opened feature request on your behalf to discuss with engineering and product management. In most cases, feature requests are planned no sooner than the next quarter, but may be longer. While we cannot guarantee every request will be productized, we'll reach back out in the future if product and engineering are considering a related update.

[ghost]

Another response,

After submitting your feature request to the Engineering team they provided a way you accomplish this in the meantime. You could attempt setting up a Trace Interceptor with Mutable Span changing this to error or not an error with some kind of if statement.

[ghost]

The Scala code we ended up with,

class StatusInterceptor() extends TraceInterceptor {
  private val nonErrors = Set(CANCELLED, INVALID_ARGUMENT, NOT_FOUND).map(_.getCode.name)

  override def onTraceComplete(trace: util.Collection[_ <: MutableSpan]): util.Collection[_ <: MutableSpan] =
    trace.asScala
      .map(s =>
        if (s.isError && s.getTags.asScala
              .exists(t => t._1 == "status.code" && nonErrors.contains(t._2.toString)))
          s.setError(false)
        else s
      )
      .asJavaCollection

  override def priority(): Int = 10
}

[MaxMao]

@jeremy-lq Is TraceInterceptor still the recommended solution to filter out some kinds of errors? We built some datadog monitor based on the gRPC errors reported by dd-trace-java and there are too much noise cased by client errors.