[BUG] Sending traces with compression from OTel collector to OTel traces source in a pipeline results in 500 error

[oeyh]

Describe the bug
Sending traces with compression from OTel collector to OTel traces source in a pipeline results in 500 error.

To Reproduce
Steps to reproduce the behavior:

1. Change otel-collector-config.yaml in jaeger-hotrod example to use otlphttp exporter:

receivers:
  jaeger:
    protocols:
      grpc:

exporters:
  otlphttp:
    traces_endpoint: http://data-prepper:21890/v1/traces
#    compression: none
    tls:
      insecure: false
      insecure_skip_verify: true
  logging:

service:
  pipelines:
    traces:
      receivers: [jaeger]
      exporters: [logging, otlphttp]

and change the corresponding OTel trace source config in trace_analytics_no_ssl_2x.yml to:

entry-pipeline:
  delay: "100"
  source:
    otel_trace_source:
      path: "/v1/traces"
      ssl: false
      unframed_requests: true
... ...

2. Run the example
3. With compression: none option (compression disabled), everything works as expected; without compression: none option, we can see error messages in OTel collector like this:

2023-11-15T17:33:54.261Z	info	exporterhelper/queued_retry.go:426	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "data_type": "traces", "name": "otlphttp", "error": "error exporting items, request to http://data-prepper:21890/v1/traces responded with HTTP Status Code 500", "interval": "42.666367768s"}

With the recent improvement in logging (#3658), we can now see exception in Data Prepper logs as well:

2023-11-15T17:34:51,616 [armeria-common-worker-epoll-3-8] ERROR org.opensearch.dataprepper.GrpcRequestExceptionHandler - Unexpected exception handling gRPC request
io.grpc.StatusRuntimeException: INTERNAL: Invalid protobuf byte sequence
......
Caused by: com.google.protobuf.InvalidProtocolBufferException$InvalidWireTypeException: Protocol message tag had invalid wire type.

Expected behavior
Otel sources should work with compressed data from OTel collector

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: [e.g. Ubuntu 20.04 LTS]
• Version [e.g. 22]

Additional context
This might be a regression. We had compression supported through #2702

[oeyh]

My mistake. Didn't realize that there's a compression option on the source config. After setting that option, the example is working with compression enabled on the collector.