Question: AD and missing tags from Unified Service Tagging

[Unibozu]

Describe what happened:

I am having some troubles getting metrics from a pod to be tagged with service: (as defined in the kubernetes pod labels). I followed the procedure for Unified Service Tagging, however had to improvise due to some complications with our setup:

• env: is set as a global tag on the agent configuration (in the helm chart values)
• service: is set as label on the kubernetes Deployment and Pod
• version: is set programmatically in our application (we are unable to get this value from our Deployment)

The metrics emitted from the pod do not show the service: tag. However, both env: and version: are present. Weirdly, the logs from the pod contain the service: tag, as well as APM.

This is an extract from the Deployment manifest - as explained above only the service label is applied here. Can it be the problem?

metadata:
  labels:
    tags.datadoghq.com/service: bit-api
spec:
  template:
    metadata:
      labels:
        tags.datadoghq.com/service: bit-api
  spec:
    containers:
      - env:
         - name: DD_SERVICE
           valueFrom:
             fieldRef:
               fieldPath: metadata.labels['tags.datadoghq.com/service']

The label is properly applied to the Pod (screenshot from GCP)

Describe what you expected:

Metrics sent from the relevant pods should have their service: tag available.

I have checked this is not the case using Datadog Metrics Summary on several unique metrics exported by our application. However, logs sent from the Pod are properly tagged with service:

Additional environment details (Operating System, Cloud provider, etc):

GKE. Kubernetes 1.17.14.
Datadog agent + cluster agents installed using this Helm chart.

The metrics are emitted by https://github.com/brightcove/hot-shots using Dogstatd Unix Socker (https://docs.datadoghq.com/developers/dogstatsd/unix_socket?tab=kubernetes).

---

Thanks

[ahmed-mez]

Hi @Unibozu

Thanks for opening this issue.

It appears that you're referring to custom metrics sent via dogstatsd.
In order to attach container and pod tags to dogstatsd metrics you need to enable Origin Detection. It can be done via the helm chart field datadog.dogstatsd.originDetection=true.

You can read more on Origin Detection here

I'm closing the issue as it isn't directly related to the helm chart but please let us know if you have any other questions!

Thank you!