Jaeger collector sampling configuration using strategies.json is not effective

[kavndu]

Hello folks, I'm configuring Jaeger collector sampling using a json file named strategies.json for a Jaeger distributed system. I've deployed all components successfully and the traces are working fine. I'm using a docker-compose for Jaeger collector and the query.

Here is the strategies.json file I used,

{
  "service_strategies": [
    {
      "service": "API-SERVICE",
      "type": "probabilistic",
      "param": 0.8,
      "operation_strategies": [
        {
          "operation": "API:API_MGT_Response_Handler",
          "type": "probabilistic",
          "param": 0.2
        },
        {
          "operation": "API:API_MGT_Usage_Handler",
          "type": "probabilistic",
          "param": 0.4
        }
      ]
    },
    {
      "service": "API:Latency",
      "type": "ratelimiting",
      "param": 5
    }
  ],
  "default_strategy": {
    "type": "probabilistic",
    "param": 0.5,
    "operation_strategies": [
      {
        "operation": "/get-the-raincoat",
        "type": "probabilistic",
        "param": 0.2
      },
      {
        "operation": "lookingForAvailability",
        "type": "probabilistic",
        "param": 0.2
      }      
    ]
  }
}

and here is the docker-compose file I used,

version: "3.7"
services:
  jaeger-collector:
    image: jaegertracing/jaeger-collector:1.21
    container_name: jaeger-collector
    networks:
      - applogs
    ports:
      - "14269:14269"
      - "14268:14268"
      - "14267:14267"
      - "9411:9411"
      - "14250:14250"
    volumes:   
      - /home/kavindu/strategies.json:/home/ubuntu/strategies.json
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
      - ES_SERVER_URLS=http://<ip_address>:9200 \  
    command:
      [
        "--es.server-urls=http://<ip_address>:9200",
        "--span-storage.type=elasticsearch",
        "--sampling.strategies-file=/home/ubuntu/strategies.json",
        "--log-level=error"
      ]
    restart: on-failure

 

  jaeger-query:
    image: jaegertracing/jaeger-query:1.21
    container_name: jaeger-query
    networks:
      - applogs
    environment:
      - SPAN_STORAGE_TYPE=elasticsearch
      - no_proxy=localhost
      - JAEGER_ENDPOINT=http://jaeger-collector:14268/api/traces
      - ES_SERVER_URLS=http://<ip_address>:9200 \
    ports:
      - "16686:16686"
      - "16687:16687"
    command:
      [
        "--es.server-urls=http://<ip_address>:9200",
        "--span-storage.type=elasticsearch",
        "--log-level=debug"
      ]
    restart: on-failure

 

networks:
  applogs:
    driver: bridge

This is how I mount the json file into collector container and use it in the docker-compose.

volumes:   
  - /home/kavindu/strategies.json:/home/ubuntu/strategies.json
command:
  [
    "--sampling.strategies-file=/home/ubuntu/strategies.json"
  ] 

This configuration is not effective for the traces. I can see the traces, same as before configuring the sampling.

Am I missing something ? How can i solve this ?

[yurishkuro]

try this:

• exec into container and check that the sampling file is indeed visible at the location you're pointing to via cmd line args
• query collector for the sampling strategies for your services: curl "http:/localhost:5778/sampling?service=foobar"

[yurishkuro]

this is not the sampling output, it's a healthcheck

[kavndu]

Sorry for that @yurishkuro Here is the output

[root@distributed-tracing-agent-app kavindu]# curl http://localhost:5778/sampling?service=foobar
{"strategyType":"PROBABILISTIC","probabilisticSampling":{"samplingRate":0.5},"operationSampling":{"defaultSamplingProbability":0.5,"defaultLowerBoundTracesPerSecond":0,"perOperationStrategies":[{"operation":"/get-the-raincoat","probabilisticSampling":{"samplingRate":0.2}},{"operation":"lookingForAvailability","probabilisticSampling":{"samplingRate":0.2}}],"defaultUpperBoundTracesPerSecond":0}}[root@distributed-tracing-agent-app kavindu]#

[root@distributed-tracing-agent-app kavindu]# curl http://localhost:5778/sampling?service=API-SERVICE
{"strategyType":"PROBABILISTIC","probabilisticSampling":{"samplingRate":0.8},"operationSampling":{"defaultSamplingProbability":0.8,"defaultLowerBoundTracesPerSecond":0,"perOperationStrategies":[{"operation":"API:API_MGT_Response_Handler","probabilisticSampling":{"samplingRate":0.2}},{"operation":"API:API_MGT_Usage_Handler","probabilisticSampling":{"samplingRate":0.4}},{"operation":"/get-the-raincoat","probabilisticSampling":{"samplingRate":0.2}},{"operation":"lookingForAvailability","probabilisticSampling":{"samplingRate":0.2}}],"defaultUpperBoundTracesPerSecond":0}}[root@distributed-tracing-agent-app kavindu]# 

[yurishkuro]

So the sampling service returns your values. Did you configure the SDK to use them via RemoteSampler? Can the sampler reach the service?

[kavndu]

I'm using a sample go application with Jaeger client that pass the traces to the Jaeger agent. Should i configure Jaeger client also ? I didn't configure any sampling configuration in application or client level.
Here is my Jaeger agent configuration.

docker run -d \
    -p 5775:5775/udp \
    -p 6831:6831/udp \
    -p 6832:6832/udp \
    -p 5778:5778/tcp \
    -p 14271:14271/tcp \
    jaegertracing/jaeger-agent:1.22 \
    --reporter.grpc.host-port=<ip_address>:14250

I've deployed application with the client and agent in one virtual machine and also I've deployed collector and the query in another virtual machine using above docker-compose file.

Sample application - https://github.com/yusufsyaifudin/go-opentracing-example | with-external-service branch

[yurishkuro]

you're using 100% sampler

		Sampler: &config.SamplerConfig{			Type:  "const",			Param: 1,		},

you need to configure remote sampler