Probabilistic sampling configuraiton does not apply to specified operations

[david-vavra]

Hi, I have a question regarding probabilistic sampling in jaeger-collector.

We have the following configuration:

    {
      "service_strategies": [
       ...
        {
           "service": "myservice",
           "type": "probabilistic",
           "param": 0.4,
           "operation_strategies": [
             {
                "operation": "healthz.liveness",
                "type": "probabilistic",
                "param": 0.0
             },
             {
                "operation": "healthz.readiness",
                "type": "probabilistic",
                "param": 0.0
             },
             {
                "operation": "healthz.metrics",
                "type": "probabilistic",
                "param": 0.0
             }
           ]
        },
...
      ],      
      "default_strategy": {
        "type": "probabilistic",
        "param": 0.01
      }
    }

The configuration is correctly mounted to jaeger-collector container, and seems to be correctly loaded:

root@jaeger-collector-5875447746-4rb6q:/# curl -s 'http://localhost:14268/api/sampling?service=myservice'  | jq
{
  "strategyType": "PROBABILISTIC",
  "probabilisticSampling": {
    "samplingRate": 0.4
  },
  "operationSampling": {
    "defaultSamplingProbability": 0.4,
    "defaultLowerBoundTracesPerSecond": 0,
    "perOperationStrategies": [
      {
        "operation": "healthz.liveness",
        "probabilisticSampling": {
          "samplingRate": 0
        }
      },
      {
        "operation": "healthz.readiness",
        "probabilisticSampling": {
          "samplingRate": 0
        }
      },
      {
        "operation": "healthz.metrics",
        "probabilisticSampling": {
          "samplingRate": 0
        }
      }
    ]
  }
}

However I can still see the traces with the above specified operations - healthz.liveness, healthz.readiness and healthz.liveness, including the tag sampler.param: 0.4, so the above configuration seems to be correctly used at least at the application level. However, I would expect that, the samplingRate: 0 applies and those traces will not be stored.

The jaeger-client within application is configured to use remote-sampling.

Any suggestions on what may be the issue, or how can I troubleshoot this further?

Thank you in advance. :)

[jpkrohling]

Which client are you using? Can you provide a reproducer for this?

[david-vavra]

We are using jaeger-client (4.5.0).

I will try to reproduce it locally using jaegertracing/all-in-one Docker image.

I also forgot to mention that the issue applies only for one of our components, the rest seems to honor the probabilistic sampling configuration also for specific operations.

[albertteoh]

Does the service/component have its sampling config set to remote? https://www.jaegertracing.io/docs/1.22/sampling/#client-sampling-configuration

[albertteoh]

Ah, apologies, I missed this in your original question:

The jaeger-client within application is configured to use remote-sampling.

[david-vavra]

I'm sorry but I'm not able to provide a simple reproducer which I could share publicly at github.

I'm closing the issue for now, eventually reopen with more details in future.