Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Telemetry: ALL_METRICS doesn't seem to work #43155

Closed
Stono opened this issue Feb 6, 2023 · 6 comments · Fixed by #43562
Closed

Telemetry: ALL_METRICS doesn't seem to work #43155

Stono opened this issue Feb 6, 2023 · 6 comments · Fixed by #43562
Labels
area/extensions and telemetry

Comments

@Stono
Copy link
Contributor

Stono commented Feb 6, 2023 •
edited by istio-policy-bot

Bug Description

I have the following telemetry v2 configuration:

spec:
  metrics:
  - providers:
    - name: prometheus
    overrides:
    - disabled: true
      match:
        metric: ALL_METRICS
        mode: SERVER
    - tagOverrides:
        destination_canonical_service:
          operation: REMOVE
        source_canonical_service:
          operation: REMOVE
        destination_principal:
          operation: REMOVE
        source_principal:
          operation: REMOVE
        connection_security_policy:
          operation: REMOVE
        grpc_response_status:
          operation: REMOVE
        source_version:
          operation: REMOVE
        destination_version:
          operation: REMOVE
        request_protocol:
          operation: REMOVE
        source_canonical_revision:
          operation: REMOVE
        destination_canonical_revision:
          operation: REMOVE
        source_cluster:
          operation: REMOVE
        destination_cluster:
          operation: REMOVE
        destination_app:
          operation: REMOVE

    - match:
        metric: REQUEST_DURATION
      tagOverrides:
        response_code:
          operation: REMOVE
        response_flags:
          operation: REMOVE
        source_cluster:
          operation: REMOVE
        destination_cluster:
          operation: REMOVE

    - match:
        metric: REQUEST_SIZE
      tagOverrides:
        response_code:
          operation: REMOVE
        response_flags:
          operation: REMOVE
        source_cluster:
          operation: REMOVE
        destination_cluster:
          operation: REMOVE

    - match:
        metric: RESPONSE_SIZE
      tagOverrides:
        response_code:
          operation: REMOVE
        response_flags:
          operation: REMOVE
        source_cluster:
          operation: REMOVE
        destination_cluster:
          operation: REMOVE

However with this configuration, I am still seeing reporter=destination metrics, despite the SERVER disabled.

If I change that first part to:

spec:
  metrics:
  - providers:
    - name: prometheus
    overrides:
    - disabled: true
      match:
        metric: REQUEST_COUNT
        mode: SERVER
    - disabled: true
      match:
        metric: REQUEST_DURATION
        mode: SERVER
    - disabled: true
      match:
        metric: REQUEST_SIZE
        mode: SERVER
    - disabled: true
      match:
        metric: RESPONSE_SIZE
        mode: SERVER

eg, explicitly disabling the SERVER for each of the metrics (instead of using ALL_METRICS), it works.

Version

1.16.2

Additional Information

No response
@Stono Stono changed the title Telemetry V2: ALL_METRICS doesn't seem to work Telemetry: ALL_METRICS doesn't seem to work Feb 6, 2023
@kyessenov
Copy link
Contributor

CC @lei-tang @zirain

@zirain
Copy link
Member

zirain commented Feb 7, 2023

will take a look later

@zirain
Copy link
Member

zirain commented Feb 7, 2023

@Stono can you share the output of running istioctl bug-report

@Stono
Copy link
Contributor Author

Stono commented Feb 11, 2023

@zirain will ping you on slack

@Stono
Copy link
Contributor Author

Stono commented Feb 16, 2023

Just checking, was this fixed by #43178 ?

@zirain
Copy link
Member

zirain commented Feb 23, 2023

please try with following configuration:

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: disable-server-metrics
  namespace: istio-system
spec:
  metrics:
    - providers:
        - name: prometheus
      overrides:
        - disabled: true
          match:
            metric: ALL_METRICS
            mode: SERVER
        - match:
            mode: CLIENT
          tagOverrides:
            destination_canonical_service:
              operation: REMOVE
            source_canonical_service:
              operation: REMOVE
            destination_principal:
              operation: REMOVE
            source_principal:
              operation: REMOVE
            connection_security_policy:
              operation: REMOVE
            grpc_response_status:
              operation: REMOVE
            source_version:
              operation: REMOVE
            destination_version:
              operation: REMOVE
            request_protocol:
              operation: REMOVE
            source_canonical_revision:
              operation: REMOVE
            destination_canonical_revision:
              operation: REMOVE
            source_cluster:
              operation: REMOVE
            destination_cluster:
              operation: REMOVE
            destination_app:
              operation: REMOVE
        - match:
            mode: CLIENT
            metric: REQUEST_DURATION
          tagOverrides:
            response_code:
              operation: REMOVE
            response_flags:
              operation: REMOVE
            source_cluster:
              operation: REMOVE
            destination_cluster:
              operation: REMOVE
        - match:
            mode: CLIENT
            metric: REQUEST_SIZE
          tagOverrides:
            response_code:
              operation: REMOVE
            response_flags:
              operation: REMOVE
            source_cluster:
              operation: REMOVE
            destination_cluster:
              operation: REMOVE
        - match:
            mode: CLIENT
            metric: RESPONSE_SIZE
          tagOverrides:
            response_code:
              operation: REMOVE
            response_flags:
              operation: REMOVE
            source_cluster:
              operation: REMOVE
            destination_cluster:
              operation: REMOVE

@zirain zirain mentioned this issue Feb 23, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/extensions and telemetry
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

telemetry: add metric ut zirain/istio
4 participants
@Stono @zirain @kyessenov @istio-policy-bot