Enbaling ServiceGraph metrics thows an error

[mrmartan]

I am looking at https://github.com/grafana/k8s-monitoring-helm/blob/main/charts/k8s-monitoring/docs/destinations/otlp.md?plain=1

Having a this snippet in values:

destinations:
  - name: gc-otlp-endpoint
    metrics:
      enabled: true
    logs:
      enabled: true
    traces:
      enabled: true
    processors:
      serviceGraphMetrics:
        enabled: true
      tailSampling:
        enabled: true

throws an error:

Error: UPGRADE FAILED: template: k8s-monitoring/templates/alloy-servicegraph.yaml:24:34: executing "k8s-monitoring/templates/alloy-servicegraph.yaml" at <include "destinations.otlp.alloy.exporter.target" $dest>: error calling include: template: k8s-monitoring/templates/destinations/_destination_otlp.tpl:368:18: executing "destinations.otlp.alloy.exporter.target" at <.processors.batch.enabled>: nil pointer evaluating interface {}.enabled

The docs say the batch processor is enabled by default but changing the above to:

destinations:
  - name: gc-otlp-endpoint
    metrics:
      enabled: true
    logs:
      enabled: true
    traces:
      enabled: true
    processors:
      batch:
        enabled: true
      serviceGraphMetrics:
        enabled: true
      tailSampling:
        enabled: true

makes the update go though. (Take note of the processors.batch.enabled = true)

I am using the latest version of the chart. My goal is to have span matrics, service graph and tail sampling all enabled in Alloy so that I can have Grafana Cloud enabled with

After the change I have started to get the traces_service_graph_ metrics and the service map in APM.

I am still not getting the Outbound/Inbound shown for individual services.

I am using the OTEL Demo app with the Bring your own backend option:

opentelemetry-collector:
  config:
    exporters:
      otlphttp/grafana:
        endpoint: http://grafana-k8s-monitoring-alloy-receiver.grafana-cloud.svc.cluster.local:4318

    service:
      pipelines:
        traces:
          exporters: [spanmetrics, otlphttp/grafana]

[mrmartan]

It seems like the metrics are missing some of the labels APM is looking for. I have:

While the query running in APM is:

histogram_quantile(0.95, sum(rate(traces_service_graph_request_client_seconds_bucket{client="checkout", client_cluster=~".*", client_service_namespace="opentelemetry-demo"} [$__rate_interval])) by (le, server, server_service_namespace))

This might be different issue. I am happy to open a different issue. Let me know pls.

[petewall]

The label issue is different. Please open a new one for that.

As for requiring the batch processor, can you share how you're deploying the chart? Using the helm cli? which version? ArgoCD?

[mrmartan]

helm repo add grafana https://grafana.github.io/helm-charts &&
  helm repo update &&
  helm upgrade --install --atomic --timeout 300s grafana-k8s-monitoring grafana/k8s-monitoring \
    --namespace "grafana-cloud" --values values.yaml

[mrmartan]

I've opened a different issue regarding the labels #2085