Unable to collect metrics for SPM

[Corfucinas]

Hi, I already went thought

1. https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/troubleshooting.md
2. https://www.jaegertracing.io/docs/1.38/spm/

troubleshooting, but it seems that I have something misconfigured or misunderstood which concludes in the monitor section being empty No data.

I'm using Python with Jaeger (traces) and Prometheus (metrics) and I'm able to successfully extract traces and metrics respectively.

For the sake of this question, and to keep it simple, let's suppose my Python program consumes from Kafka messages, processes its contents, and just relays them over to a HTTP destination. Kafka -> Logic -> Send

simple example of the pattern

tracer = trace.get_tracer(__name__)

async def send_post_request(url: str, data: dict[str, object]) -> None:
    with tracer.start_as_current_span('send_post_request'):
        async with self.create_http_client() as async_client:
             await async_client.post(url, data)

I'm using HTTPX to send the messages, which are POST.

Will the SPM be populated automatically or do I need to manually and explicitly tell it how to generate Latency, Error rate, and, Request rate?

Here's my current configuration, what am I missing to make the SPM functional?

Thanks

prometheus.yml

global:
  scrape_interval: 15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'my-program'
    query_log_file: 'prometheus.log'

# A scrape configuration containing exactly one endpoint to scrape:
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  # https://github.com/prometheus/node_exporter
  - job_name: 'node_exporter'
    honor_labels: true
    honor_timestamps: true
    static_configs:
      - targets: ['node_exporter:9100']
  - job_name: my_program
    scrape_interval: 15s
    static_configs:
      - targets: ['host.docker.internal:8000']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: '.*grpc_io.*'
        action: drop

otel-collector-config.yml

receivers:
  jaeger:
    protocols:
      grpc:

  # Dummy receiver that's never used, because a pipeline is required to have one.
  otlp/spanmetrics:
    protocols:
      grpc:
        endpoint: 'localhost:65535'

  otlp:
    protocols:
      grpc:

  prometheus:
    config:
      scrape_configs:
        - job_name: open-telemetry
          scrape_interval: 15s
          static_configs:
            - targets: ['localhost:8888']
          metric_relabel_configs:
            - source_labels: [__name__]
              regex: '.*grpc_io.*'
              action: drop

exporters:
  logging:
    loglevel: debug

  prometheus:
    endpoint: '0.0.0.0:8889'
    resource_to_telemetry_conversion:
      enabled: true

  jaeger:
    endpoint: 'jaeger:14250'
    tls:
      insecure: true

  otlp/spanmetrics:
    endpoint: 'localhost:55677'
    tls:
      insecure: true

processors:
  # https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md
  batch:
  # https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/README.md
  memory_limiter:
    check_interval: 5s
    limit_mib: 819
    spike_limit_mib: 256
  spanmetrics:
    metrics_exporter: prometheus
    latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 100ms, 250ms]
    dimensions_cache_size: 1500

    # Additional list of dimensions on top of:
    # - service.name
    # - operation
    # - span.kind
    # - status.code
    dimensions:
      # If the span is missing http.method, the processor will insert
      # the http.method dimension with value 'GET'.
      # For example, in the following scenario, http.method is not present in a span and so will be added as a dimension to the metric with value "GET":
      # - calls_total{http_method="GET",operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET"} 1
      - name: http.method
        default: GET

      # If a default is not provided, the http.status_code dimension will be omitted
      # if the span does not contain http.status_code.
      # For example, consider a scenario with two spans, one span having http.status_code=200 and another missing http.status_code. Two metrics would result with this configuration, one with the http_status_code omitted and the other included:
      # - calls_total{http_status_code="200",operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET"} 1
      # - calls_total{operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET"} 1
      - name: http.status_code

    # The aggregation temporality of the generated metrics.
    # Default: "AGGREGATION_TEMPORALITY_CUMULATIVE"
    aggregation_temporality: 'AGGREGATION_TEMPORALITY_CUMULATIVE'

extensions:
  health_check:
  memory_ballast:
  pprof:
    endpoint: :1888
  zpages:
    # http://localhost:55679/debug/tracez
    endpoint: :55679

service:
  extensions: [memory_ballast, health_check, zpages, pprof]
  telemetry:
    metrics:
      address: :8888
    logs:
      level: debug
  pipelines:
    traces:
      receivers: [jaeger]
      processors: [memory_limiter, batch, spanmetrics]
      exporters: [logging, jaeger]
    metrics:
      receivers: [prometheus]
      processors: [memory_limiter, batch]
      exporters: [logging]
    # The exporter name must match the metrics_exporter name.
    # The receiver is just a dummy and never used; added to pass validation requiring at least one receiver in a pipeline.
    metrics/spanmetrics:
      receivers: [otlp/spanmetrics]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [logging]

docker-compose

---
version: "3.9"
services:
  jaeger:
    networks:
      - backend
    container_name: jaeger
    image: jaegertracing/all-in-one:latest
    volumes:
      - "./jaeger-ui.json:/etc/jaeger/jaeger-ui.json"
    command: --query.ui-config /etc/jaeger/jaeger-ui.json
    # https://www.jaegertracing.io/docs/1.38/cli/
    # https://github.com/jaegertracing/jaeger/discussions/2834?sort=top
    environment:
      - METRICS_STORAGE_TYPE=prometheus
      - PROMETHEUS_SERVER_URL=http://prometheus:9090
      - COLLECTOR_ENABLE_SPAN_SIZE_METRICS=true
    ports:
      - 14269:14269 # admin port
      - 14250:14250 # gRPC
      - 14268:14268
      - 6831:6831/udp # Thrift compact, might delete
      - 16686:16686 # UI
      - 16685:16685 # Protobuf
      - 9411:9411 
      - 16687:16687 
      - 5775:5775/udp 
      - 6832:6832 
      - 5778:5778 
    restart: on-failure
  otel_collector:
    networks:
      - backend
    image: otel/opentelemetry-collector-contrib:latest
    container_name: otel_collector
    volumes:
      - "./otel-collector-config.yml:/etc/otelcol/otel-collector-config.yml"
    command: --config /etc/otelcol/otel-collector-config.yml
    depends_on:
      - jaeger
    #TODO!: Do I need these ports? probably..
    ports:
      - 1888:1888 # pprof extension
      - 8888:8888 # Prometheus metrics exposed by the collector
      - 8889:8889 # Prometheus exporter metrics
      - 13133:13133 # health_check extension
      - 4317:4317 # OTLP gRPC receiver
      - 55679:55679 # zpages extension
    restart: on-failure
  prometheus:
    container_name: prometheus
    networks:
      - backend
    image: prom/prometheus:latest
    volumes:
      # - ./prometheus:/etc/prometheus
      # - ./prometheus_data:/prometheus
      - "./prometheus.yml:/etc/prometheus/prometheus.yml"
      - "./prometheus-alerts.yml:/etc/prometheus/prometheus-alerts.yml"
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      #   - '--storage.tsdb.path=/prometheus'
      #  - '--storage.tsdb.retention.time=200h'
      - '--web.console.libraries=/etc/prometheus/console_libraries'
      - '--web.console.templates=/etc/prometheus/consoles'
      - '--web.enable-lifecycle'
    ports:
      - 9090:9090
    restart: on-failure
  node_exporter:
    image: prom/node-exporter:latest
    container_name: node_exporter
    restart: on-failure
    # Additional volumes?
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro # Docker
      # - '/:/host:ro,rslave' # Host machine
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs' # Docker
      # - '--path.rootfs=/host' # Host machine
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc|var/lib/docker/.+)($$|/)'
      # /roots/etc/os-release permission denied node exporter
    ports:
      - 9100:9100
    networks:
      - backend
  grafana:
    container_name: grafana
    networks:
      - backend
    image: grafana/grafana:latest
    volumes:
      # - grafana_data:/var/lib/grafana
      # - ./grafana/provisioning:/etc/grafana/provisioning
      - ./grafana.ini:/etc/grafana/grafana.ini
      - ./datasource.yml:/etc/grafana/provisioning/datasources/datasource.yaml
    # environment:
    #  - GF_SECURITY_ADMIN_USER=admin
    #  - GF_SECURITY_ADMIN_PASSWORD=admin
    #  - GF_USERS_ALLOW_SIGN_UP=false
    ports:
      - 3000:3000
    restart: on-failure

networks:
  backend:

pyproject.toml (telemetry dependencies)

[tool.poetry.group.telemetry.dependencies]
opentelemetry-distro = {extras = ["otlp"], version = "^0.34b0"}
opentelemetry-exporter-jaeger-proto-grpc = "^1.13.0"
opentelemetry-instrumentation-httpx = "^0.34b0"
opentelemetry-instrumentation-kafka-python = "^0.34b0"
opentelemetry-instrumentation-logging = "^0.34b0"
opentelemetry-instrumentation-sqlalchemy = "^0.34b0"
opentelemetry-instrumentation-system-metrics = "^0.34b0"
opentelemetry-exporter-prometheus = "^1.12.0rc1"
prometheus-client = "^0.14.1"

[yurishkuro]

Dummy receiver that's never used, because a pipeline is required to have one.

Isn't this your problem? If your OTEL collector is not receiving any spans there's nothing for the rest of the pipeline to do, so you'll get no metrics. Your application needs to export data to OTEL collector, not directly to Jaeger.

[Corfucinas]

I took that line and the comment from the official documentation

https://github.com/jaegertracing/jaeger/blob/main/docker-compose/monitor/otel-collector-config.yml

Could you copy paste the change I should make? I also tried using the .yml file as-is from that repo but it doesn't work.

[yurishkuro]

I don't think you need to change that config - you need to point your SDKs to export to OTEL Collector, not to Jaeger directly.

[albertteoh]

Agree with @yurishkuro, the first thing to check is that OTEL collector is receiving spans. I can see you've configured a logging exporter at debug level, so you should be able to see each span being logged as it arrives on the collector.

At a brief glance, assuming you're correctly sending spans to the OTEL collector, it looks like prometheus isn't configured to scrape from OTEL Collector. Try adding something like (example):

scrape_configs:
  - job_name: aggregated-trace-metrics
    static_configs:
    - targets: ['otel_collector:8889']

An easy way to check if you're seeing the raw RED metrics is to open up Grafana (which I see you've configured in docker-compose).

[Corfucinas]

@yurishkuro @albertteoh

After a few hours of troubleshooting here's the progress (read until the end)

otel-collector-config.yml

receivers:
  jaeger:
    protocols:
      grpc:

  otlp:
    protocols:
      grpc:

  prometheus:
    config:

exporters:
  logging:
    loglevel: debug

  prometheus:
    endpoint: '0.0.0.0:8889'
    resource_to_telemetry_conversion:
      enabled: true

  jaeger:
    endpoint: 'jaeger:14250'
    tls:
      insecure: true

  otlp:
    endpoint: otel_collector:4317
    tls:
      insecure: true

processors:
  # https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/batchprocessor/README.md
  batch:
  # https://github.com/open-telemetry/opentelemetry-collector/blob/main/processor/memorylimiterprocessor/README.md
  memory_limiter:
    check_interval: 5s
    limit_mib: 819
    spike_limit_mib: 256
  spanmetrics:
    metrics_exporter: prometheus
    # latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 100ms, 250ms]
    dimensions_cache_size: 1500
    # The aggregation temporality of the generated metrics.
    # Default: "AGGREGATION_TEMPORALITY_CUMULATIVE"
    aggregation_temporality: 'AGGREGATION_TEMPORALITY_CUMULATIVE'

    # Additional list of dimensions on top of:
    # - service.name
    # - operation
    # - span.kind
    # - status.code
    dimensions:
      # If the span is missing http.method, the processor will insert
      # the http.method dimension with value 'GET'.
      # For example, in the following scenario, http.method is not present in a span and so will be added as a dimension to the metric with value "GET":
      # - calls_total{http_method="GET",operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET"} 1
      - name: http.method
        default: GET

      # If a default is not provided, the http.status_code dimension will be omitted
      # if the span does not contain http.status_code.
      # For example, consider a scenario with two spans, one span having http.status_code=200 and another missing http.status_code. Two metrics would result with this configuration, one with the http_status_code omitted and the other included:
      # - calls_total{http_status_code="200",operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET"} 1
      # - calls_total{operation="/Address",service_name="shippingservice",span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_UNSET"} 1
      - name: http.status_code
        default: 200

extensions:
  health_check:
  memory_ballast:
  pprof:
    endpoint: :1888
  zpages:
    # http://localhost:55679/debug/tracez
    endpoint: :55679

service:
  extensions: [memory_ballast, health_check, zpages, pprof]
  telemetry:
    metrics:
      address: :8888
    logs:
      level: debug
  pipelines:
    traces:
      receivers: [otlp]
      # receivers: [jaeger] # This is creating a problem
      processors: [memory_limiter, spanmetrics, batch]
      exporters: [logging, otlp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [logging]

with this, I'm able to see latency bucket and call_total on Prometheus, however, I cannot see the graphs on Jaeger

main.py

import time

import httpx
from opentelemetry import metrics, trace
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import (
    OTLPMetricExporter,
)
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
    OTLPSpanExporter,
)
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.trace.status import Status, StatusCode


# Service name is required for most backends
resource = Resource(attributes={SERVICE_NAME: "jaeger-monitor-test"})

reader = PeriodicExportingMetricReader(
    OTLPMetricExporter(endpoint="localhost:4317", insecure=True)
    # OTLPMetricExporter(insecure=True)

)
provider = MeterProvider(resource=resource, metric_readers=[reader])
metrics.set_meter_provider(provider)


# Service name is required for most backends
resource = Resource(attributes={SERVICE_NAME: "jaeger-monitor-test"})

provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:14250", insecure=True))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

meter = metrics.get_meter(__name__)
tracer = trace.get_tracer(__name__)
HTTPXClientInstrumentor().instrument()


def send_request() -> None:
    with tracer.start_as_current_span('send_request') as span:
        with httpx.Client() as client:
            request = client.get("https://www.google.com")
            print(request.status_code)
            time.sleep(2)
            span.set_status(Status(StatusCode.OK)) # Hardcoded


if __name__ == "__main__":
    while True:
        send_request()

Right now if I use otlpas the reciever it works, but without any graphs on SPM.
If I change it to recievers: [jaeger]

it fails with

Failed to export traces, error code: StatusCode.UNIMPLEMENTED

What would be the reason?

I don't think you need to change that config - you need to point your SDKs to export to OTEL Collector, not to Jaeger directly.

Agree with @yurishkuro, the first thing to check is that OTEL collector is receiving spans. I can see you've configured a logging exporter at debug level, so you should be able to see each span being logged as it arrives on the collector.

At a brief glance, assuming you're correctly sending spans to the OTEL collector, it looks like prometheus isn't configured to scrape from OTEL Collector. Try adding something like (example):

scrape_configs:
  - job_name: aggregated-trace-metrics
    static_configs:
    - targets: ['otel_collector:8889']

An easy way to check if you're seeing the raw RED metrics is to open up Grafana (which I see you've configured in docker-compose).

[Corfucinas]

In case it helps you @yurishkuro @albertteoh to figure out where's the problem I'm posting a log from the docker containter.

[albertteoh]

The problem might be this:

processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="localhost:14250", insecure=True))
processor = BatchSpanProcessor(OTLPSpanExporter(insecure=True))

Since you're using OTLP exporters, then jaeger receivers won't know how to read spans from your python script, and that's probably the cause of the errors you see. The second issue is the second line seems to overwrite the first.

As I couldn't get your python script to work quickly, I just threw in a microsim container into your docker-compose file and, using a jaeger receiver, I could see metrics in SPM:

  microsim:
    networks:
      - backend
    image: yurishkuro/microsim:0.2.0
    command: "-j http://otel_collector:14278/api/traces -d 24h -s 500ms"
    depends_on:
      - otel_collector

[Corfucinas]

Hey @albertteoh, I tried flipping between using using JaegerExporter and OTLPSpanExporter and it does not work still....the code above is not rewritten, I accidentally left a # when pasting.

Is it possible for you to place a working example in python on how to extract the RED metrics? It's been a few days and I cannot get it to work properly.

For my example above, here's the dependencies.

Thank you a lot for the effort

[tool.poetry.dependencies]
python = "^3.10"
httpx = {extras = ["http2"], version = "^0.23.0"}
opentelemetry-exporter-otlp-proto-grpc = "^1.13.0"
opentelemetry-instrumentation-httpx = "^0.34b0"
opentelemetry-exporter-jaeger = "^1.13.0"
opentelemetry-exporter-prometheus = "^1.12.0rc1"
requests = "^2.28.1"
opentelemetry-exporter-otlp-proto-http = "^1.13.0"

[albertteoh]

Hey @Corfucinas, just an update on this, I've got a basic self-contained example working with python. I'll put it into a repo tonight so you can try it out.

[Corfucinas]

Great! @albertteoh Thanks a lot

[albertteoh]

The following is a very basic python script based on this tutorial: https://opentelemetry-python-kinvolk.readthedocs.io/en/latest/getting-started.html.

It emits spans to otel collector (that has a jaeger receiver), which will aggregate the spans into R.E.D metrics.

Prerequisites:

• Expose otel configured thrift http port #3986 needs to be merged first.

Instructions:

• git clone https://github.com/jaegertracing/jaeger
• cd jaeger/docker-compose/monitor
• docker-compose up
• python3 jaeger_example.py
  • Make sure to install the dependencies specified in https://opentelemetry-python-kinvolk.readthedocs.io/en/latest/getting-started.html
• open http://localhost:16686/monitor, select my_server in the service drop-down to confirm the R.E.D metrics are visible.

A few noteworthy points:

• In order to see R.E.D. metrics for these spans, its span kind must be "server"; I think it defaults to "undefined". This is because the SPM feature specifically queries on "server" spans. The reason for this decision is explained in: https://www.jaegertracing.io/docs/1.38/spm/#serviceoperation-missing-in-monitor-tab.
• The service.name is explicitly set in resources, otherwise it will default to unknown_service.
• Since the otel collector is configured to listen on the thrift_http port 14278, we need to configure collector_endpoint="http://localhost:14278/api/traces".

Here's the python code (@yurishkuro, if you feel it makes sense to add this script into Jaeger's docker-compose/monitor directory, I'm happy to do so):

# jaeger_example.py
from opentelemetry import trace
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.trace import SpanKind
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource

resource = Resource(attributes={
    "service.name": "my_service"
})

trace.set_tracer_provider(TracerProvider(resource=resource))

jaeger_exporter = JaegerExporter(
    collector_endpoint="http://localhost:14278/api/traces",
)

trace.get_tracer_provider().add_span_processor(
    BatchSpanProcessor(jaeger_exporter)
)

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("foo", kind=SpanKind.SERVER):
    with tracer.start_as_current_span("bar", kind=SpanKind.SERVER):
        with tracer.start_as_current_span("baz", kind=SpanKind.SERVER):
            print("Hello world from OpenTelemetry Python!")

Here's the requirements.txt:

Deprecated==1.2.13
googleapis-common-protos==1.56.2
grpcio==1.50.0
opentelemetry-api==1.13.0
opentelemetry-exporter-jaeger==1.13.0
opentelemetry-exporter-jaeger-proto-grpc==1.13.0
opentelemetry-exporter-jaeger-thrift==1.13.0
opentelemetry-sdk==1.13.0
opentelemetry-semantic-conventions==0.34b0
protobuf==3.20.3
six==1.16.0
thrift==0.16.0
typing_extensions==4.4.0
wrapt==1.14.1

[Corfucinas]

This works @yurishkuro ! Part of your changes are also using thrift over grpc, but happy with the results, it's showing up now in my code.