Add throttling / backpressure and correct error handling to OTLP receiver

[tigrannajaryan]

OTLP must throttle the clients when it receives a backpressure from the pipeline (e.g. memory_limiter).

Must comply with OTLP spec and use gRPC status codes + RetryInfo (see the spec).

Also currently does not seem to correctly handle non-permanent errors returned by next consumer. We return 500, which according to OTLP spec means client will not retry. This means we will lose data.

Instead for retryable errors we must return 503 and for permanent errors must return 400 or similar.

[tigrannajaryan]

@pjanotti can you please add details here about how exactly receivers are informed about backpressure?

[tigrannajaryan]

This is a nice-to-have, removing from 1.0.

[tigrannajaryan]

We also need to test this for OTLP receiver's HTTP transport and ensure we return the correct backpressure code according to OTLP/HTTP spec.

Preferably these all should have automated tests.

[punya]

Is this specific to OTLP, or does it apply to other push-based receivers too?

[tigrannajaryan]

Briefly checked the source code and what we need to do is check that the error returned from the consumer is PermanentError (which can happen if connected directly to exporter) and return a BadData to the sender.

In all other cases we should return a "backoff" response to the sender (real-life use cases are either errors from memory limiter or errors from queued_retry when the queue is fill). If the error is throttleRetry error we can also extract and use the retry interval from throttleRetry.

All others receivers need to behave similarly. Not sure if we can implement some sort of helper (likely doable at least for all http-based receivers or for all grpc-based receivers).

[tigrannajaryan]

Related or duplicate of #4335 ?