[BUG] otel-trace-source is not accepting traces in OTLP format via HTTP

[tmarszal]

Describe the bug
OTel Trace Source claims to support OTLP Protocol using both OTLP/grpc and OTLP/HTTP. I am not able to make the latter work. (It's a limitation of my runtime environment that I can only use HTTP for now.)

To Reproduce

1. Use jaeger-hotrod example from the repository
2. Edit otel-collector-config.yml:
   1. Change exporter from GRPC to HTTP: otlp to otlphttp on lines 7 and 16
   2. Add HTTP protocol to the endpoint: data-prepper:21890 to http://data-prepper:21890 on line 8
3. Run docker compose: docker-compose up -d --build
4. Go to http://localhost:8080/
5. Click a few buttons
6. Go to http://localhost:5601/app/observability-dashboards#/trace_analytics/home
7. See that no traces are coming through
8. Run docker logs otel-collector
9. See 404 errors:

2022-03-08T07:24:35.632Z	info	exporterhelper/queued_retry.go:314	Exporting failed. Will retry the request after interval.	{"kind": "exporter", "name": "otlphttp/2", "error": "error exporting items, request to http://data-prepper:21890/v1/traces responded with HTTP Status Code 404", "interval": "17.391133947s"}

10. (Optional) clean-up: docker-compose rm -s -v

Expected behavior
I expected the messages to come through, just as they do when using unmodified otel-collector-config.yml.

Environment (please complete the following information):

• OS: Ubuntu 20.04.4 LTS
• Version: 1.2.1-linux-64

Additional context
I can't see any implementation for OTLP/HTTP in https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/otel-trace-source/src/main/java/com/amazon/dataprepper/plugins/source/oteltrace

[graytaylor0]

Hi @tmarszal,

Thanks for creating this issue! I have recreated the 404 failure and am looking further into it.

[graytaylor0]

After doing some investigation, it was found that the Armeria servers support application/protobuf content type, while the default content type for otlphttp is application/x-protobuf. I am able to get data through Data Prepper to OpenSearch by making the following changes.

1. Modify the otel-collector-config.yml exporter to the following

exporters:
  otlphttp/2:
    traces_endpoint: http://data-prepper:21890/opentelemetry.proto.collector.trace.v1.TraceService/Export
    insecure: false
    insecure_skip_verify: true
    headers:
      "Content-type": "application/protobuf"

2. Change the otel-trace-source configuration in the pipeline configuration used for the jaeger-hotrod demo (https://github.com/opensearch-project/data-prepper/blob/main/examples/trace_analytics_no_ssl.yml) to enable unframed_requests to allow for http requests

otel_trace_source:
      ssl: false
      unframed_requests: true

Again, thanks for finding this. We should definitely have an example usage of otlphttp somewhere. Let me know if you run into any issues after making these configuration changes and retrying the jaeger-hotrod example.

[tmarszal]

Hi @graytaylor0 ,

Thank you for a quick reply. Your changes worked for me on the example. However, they did not when I tried migrating the changes to my environment. I keep getting HTTP 500 as a response to the request from opentelemetry-collector to data prepper, but there is no indicator of any problems in data prepper logs. What I noticed is that the example uses opentelemetry-collector:0.24.0, while the latest one is 0.46.0 (I am using 0.45.0 at the moment).

I am also getting the same error in example after bumping collector to 0.45.0 in docker-compose.yml. I slightly adjusted the collector configuration after the upgrade:

exporters:
  otlphttp/2:
    traces_endpoint: http://data-prepper:21890/opentelemetry.proto.collector.trace.v1.TraceService/Export
    tls:
      insecure: false
      insecure_skip_verify: true
    headers:
      "Content-type": "application/protobuf"

Tomorrow I am going to enable more detailed logs and/or debug data prepper to see what's the underlying issue there.

[tmarszal]

I inspected HTTP 500 response to opentelemetry-collector and found out there was INTERNAL: Invalid protobuf byte sequence in the body. When I turned off gzip compression, everything started working:

exporters:
  otlphttp/2:
    traces_endpoint: http://data-prepper:21890/opentelemetry.proto.collector.trace.v1.TraceService/Export
    compression: none
    tls:
      insecure: false
      insecure_skip_verify: true
    headers:
      "Content-type": "application/protobuf"

So the last question from me is if it is possible to enable compression on data-prepper side? I was not able to find anything regarding that in the documentation.

[graytaylor0]

Glad you were able to get it working. Right now, there is no way to enable compression on the data-prepper side. Would you like to have an issue created for this feature?

[tmarszal]

@graytaylor0 Thanks for confirming! I will create it myself as an enhancement suggestion 👍