The randomSamplingPercentage field in the namespace level telemetry api cannot override the corresponding field at the mesh level

[fizos]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

According to the documentation：https://istio.io/latest/docs/tasks/observability/telemetry/#configuring-namespace-scoped-tracing-behavior, the sampling frequency of tracing can be override by the namespace-level or workload-level telemetry API. Currently, I configured the randomSamplingPercentage as 1% at the mesh level and 100% in the default namespace,
expecting traces in the default namespace to be sampled at a frequency of 100，however the actual observation did not meet the expectation.

namespace-scope telemetry

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: default-ns
  namespace: default
spec:
  tracing:
  - providers:
    - name: "otel-tracing"
    randomSamplingPercentage: 100

mesh-scope telemetry

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  tracing:
  - providers:
    - name: "otel-tracing"
    randomSamplingPercentage: 1

Version

istioctl version
client version: 1.21.2
control plane version: 1.21.2
data plane version: 1.21.2 (8 proxies)

kubectl version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.2

Additional Information

No response

[zirain]

I try to reproduce what you said, but worked as expected.

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  tracing:
    - providers:
        - name: "zipkin"
      randomSamplingPercentage: 100.00
---
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: ns-level
  namespace: default
spec:
  tracing:
    - providers:
        - name: "zipkin"
      randomSamplingPercentage: 99.00

output:

istioctl pc l sleep-7656cf8794-qsff2  --port 8000 -oyaml | grep randomSampling -A1
          randomSampling:
            value: 99

[fizos]

Thank you for your reply. It is true that the sampling frequency obtained according to the above command has been updated.
This is my output:

istioctl pc l productpage-v1-675fc69cf-qgmrw  --port 9080 -oyaml |grep randomSampling -A1
          randomSampling:
            value: 100

But according to the dashboard, not every request generates a trace record, and the sampling rate still seems to be 1%.

When I make a request with the x-envoy-force-trace header and force the generation of tracking records, I can see the records on the dashboard.

Is there something wrong with my configuration?

[zirain]

ingressgateway located in istio-system namespace, so only mesh-default take effect, which means 1%.

[fizos]

Can it be understood that the final sampling frequency of the entire trace depends on the smallest sampling frequency in the trace?