Otel collector fails to establish a connection to jaeger

[mpbnkg]

Component(s)

No response

What happened?

When I try to export metrics to jaeger via OTLP collector it fails with the following error. What is causing this given the fact that it was working a few weeks back?

2024-07-09T18:35:39.864Z info exporterhelper/retry_sender.go:118 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 172.25.0.2:4317: connect: connection refused"", "interval": "3.136185958s"}

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317" # OTLP gRPC receiver
      - "9090:9090" # Prometheus exporter port
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "6831:6831/udp" # UDP port for Jaeger agent
      - "16686:16686" # Web UI
      - "14268:14268" # HTTP port for spans

otel-collector-config.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: otel-collector:4317

processors:

extensions:
  health_check: {}

exporters:
  logging:
    loglevel: debug

  otlp:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      exporters: [logging]
    logs:
      receivers: [otlp]
      exporters: [logging]

Collector version

latest

Environment information

Environment

OS: MAC

OpenTelemetry Collector configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: otel-collector:4317

processors:

extensions:
  health_check: {}

exporters:
  logging:
    loglevel: debug

  otlp:
    endpoint: jaeger:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      exporters: [logging]
    logs:
      receivers: [otlp]
      exporters: [logging]

Log output

2024-07-09T18:35:39.864Z info exporterhelper/retry_sender.go:118 Exporting failed. Will retry the request after interval. {"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 172.25.0.2:4317: connect: connection refused"", "interval": "3.136185958s"}

Additional context

No response

[PedemonteGiacomo]

I also obtain the same error and I'm not be able to fix it

[mpbnkg]

@PedemonteGiacomo It seems like it's an issue with the latest version. I'm sure this code was working a few weeks ago, but I haven't used it since then. When I re-ran it today, it started failing for reasons I can't understand. Hopefully, we will get a response from the maintainers soon.

[PedemonteGiacomo]

@mpbnkg Also for me it was working fine with the Jaeger:4317 declaration. The problem is that the connection refused error seems to be more Jaeger side, to try this, I've chaned the export to the endpoint Jaeger:14250 and the connection is reached but then if I send traces through the collector, the traces are dropped and not exported correctly.

[PedemonteGiacomo]

I've checked that even if I use older version of the image the error remain. So now my question is: is the error related to otel-collector?

[mpbnkg]

@PedemonteGiacomo Hopefully, we'll get some answers soon here :)

[crobert-1]

You may need to configure depends_on in your docker compose file. This may just be a timing issue of trying to connect to a container that's not ready yet.

For reference, I'm running the Collector demo which has a very similar format and setup to your example here, and it's working locally for me. You're welcome to try it out and see what could be the impacting differences if depends_on doesn't fix it.

[mpbnkg]

@crobert-1 Same thing when I even added depends_on jaeger in the otel collector section! Error:

info    exporterhelper/retry_sender.go:118      Exporting failed. Will retry the request after interval.  {"kind": "exporter", "data_type": "traces", "name": "otlp", "error": "rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 172.26.0.2:4317: connect: connection refused\"", "interval": "28.015591359s"}.

Docker-compose

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317" # OTLP gRPC receiver
      - "9090:9090" # Prometheus exporter port
    depends_on:
      - jaeger
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "6831:6831/udp" # UDP port for Jaeger agent
      - "16686:16686" # Web UI
      - "14268:14268" # HTTP port for spans

[PedemonteGiacomo]

As indicated by the error, the depends_on directive isn't the root cause of the problem since the pod or container IP is recognized successfully. The issue seems to be that Jaeger is refusing the connection.

To troubleshoot, I will attempt to use an older version of Jaeger and will update you on the results.

Additionally, @crobert-1 has raised an interesting point: another possible cause of failure could be a missing configuration in our otel-collector setup. The relevant configuration code is:

exporters:
  zipkin:
    endpoint: "http://zipkin-all-in-one:9411/api/v2/spans"
    format: proto

  otlp:
    endpoint: jaeger-all-in-one:4317
    tls:
      insecure: true

You can review the complete configuration code here

Please let me know if you have any further insights or suggestions.

[mpbnkg]

@PedemonteGiacomo Zipkin doesn't have to do anything with Jaeger. It's a completely different tool, so we don't need Zipkin for sure.

To troubleshoot, I will attempt to use an older version of Jaeger and will update you on the results.

Yep, let me know please, thanks!

[PedemonteGiacomo]

Yes @mpbnkg, you're absolutely right about zipkin, sorry for the confusion.

I changed the version of the jaeger image to an older one, more precisely, I've choose one image that was not pushed within the last 10 days to be sure to get older than the "born" of this issue that we are investigating on.

And I get my traces succesfully received by jaeger:

So, I think is a jager problem with the latest or one of the changes done in one of the latest jager-all-in-one image

[mpbnkg]

Oh, wow... Look at that, thanks :) Marking this ticket as resolved.

[crobert-1]

Thanks for following up, glad to hear this was figured out!