-
Notifications
You must be signed in to change notification settings - Fork 7.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Customizing Istio Metrics with unexpected metrics #38277
Comments
I try that config local, everything looks good, did you restart the pod after you changes the IstioOperator CR? |
@zirain What istio version you are using |
do you have any istio annotations on the pod? |
For (1): are Generally, when we see funky exported prom metrics like that, it is an indication that the regexps configured to transform the raw metrics into prom metrics are incorrect (and missing these cases). That, typically, indicates that the proxy is not aware of the statsTag info at bootstrap (startup time). We recommend restarting the pods with the proxies to force regeneration of the bootstrap info from the control plane. If the MeshConfig looks correct and there are not annotations on the pod itself potentially overriding that configuration, maybe confirm there are not error logs in the control plane? For (2): It looks like there may be an issue with copy/paste in your examples. Both queries look the same to me. What is difference being expressed ? I think what you are suggesting is that your custom histogram is acting as a counter. Is that correct? if so, that is probably due to the use of the value of
Means that you are making an observation of 1 millisecond with each request. You will likely want to use some expression over For (3), adding customization around histogram buckets is something we would like to offer. IIRC, this is still a bootstrap setting in Envoy (and not under the focus of the Telemetry API at the moment, as it requires proxy restarts). I think there is progress that can be made, but it will require a decent amount of doing. |
@douglas-reid @zirain Thanks for your quick response For(1): I checked the pod, the annotation of pod is as following
Think this does not have relation to the customize metrics. Besides, the value of the pod ENV PROXY_CONFIG looks fine as
Also I checked
For (2) Yeah, @douglas-reid you are correct. Thanks for this. it works for me now For (3) It is a great news as there is progress. I know it is not an easy work for the customize of bucket. Waiting for the good news. Thanks for informing this job |
the config_dump file your shared seems is not complete? |
@zirain sorry for missing the message, redump the config and uploaded ,plz help check |
the configdump seems ok, does this pod still have wired metrics? |
@zirain yes, the pods with istio envoy still have the wired metrics |
@YaHuiSong can you post the non-prometheus dump of the stats from a proxy that matches that config dump? |
@douglas-reid What do you mean the non-prometheus dump of the stats?I used the |
Try: |
🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2022-04-28. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions. Created by the issue and PR lifecycle manager. |
Bug Description
situation:
I configured the customize COUNTER metric posta_total and HISTOGRAM metrics istio_posta_duration_millionseconds with dimensions url_path and response_status in istioOperator yaml file. Then I got some strange issue
use the PromQL with
sum(istio_request_duration_milliseconds_bucket{pod="xxxxxxxx"}) by (le)
result: 2 values
Use istio standard metric with the same pod
sum(istio_request_duration_milliseconds_bucket{pod="xxxxx"}) by (le)
result: 3 values
Is it possible If I want to override the existing HOSTGRAM metric dimensions le .
I know the telemetry V2 does not support customizing buckets for histogram type metrics from fqa https://istio.io/latest/about/faq/metrics-and-logs/#telemetry-v1-vs-v2.
But I am not sure if telemetry V2 support customizing buckets right now. While as the le default value is [0.5, 1, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000, 30000, 60000, 300000, 600000, 1800000, 3600000]
For reality service, the values almost locate less than 5000, the most of default le values is not used actually.
So do istio team has any plan to support customizing buckets for histogram type metrics to make it work better if it does not support now.
I am freshman in istio metric, correct me if I made any mistake. Thanks in advance
What I did:
Version
Additional Information
Running with the following config:
istio-namespace: istio-system
full-secrets: false
timeout (mins): 30
include: { }
exclude: { Namespaces: kube-system, kube-public, kube-node-lease, local-path-storage } AND { Namespaces: kube-system, kube-public, kube-node-lease, local-path-storage }
end-time: 2022-04-07 15:07:07.306058 +0800 CST
The following Istio control plane revisions/versions were found in the cluster:
Affected product area
Is this the right place to submit this?
The text was updated successfully, but these errors were encountered: