Repository navigation
Metrics not working following 1.5 -> 1.6 upgrade #27851
Description
Activity
@kyessenov did anything change between 1.5 and 1.6 with respect to supporting the
repeated MetricsConfig metricsor similar?Modifying the configuration to:
{ "debug": "false", "stat_prefix": "istio", "metrics": [ { "tags_to_remove": ["destination_canonical_service", "source_canonical_service", "destination_principal", "source_principal", "connection_security_policy", "grpc_response_status", "source_version", "destination_version", "request_protocol", "source_canonical_revision", "destination_canonical_revision"] }, { "name": "request_duration_milliseconds", "tags_to_remove": ["response_code", "response_flags"], }, { "name": "request_bytes", "tags_to_remove": ["response_code", "response_flags"], }, { "name": "response_bytes", "tags_to_remove": ["response_code", "response_flags"], } ] }(note the array). Resolved the issue.
@douglas-reid suggested istio/proxy#2807, in particular https://github.com/istio/proxy/pull/2807/files#diff-db83bcb3df7f25cfb88ff0a20bbd5540 was likely the cause.
As a user - this was extremely difficult to debug, and would expect a better UX than just no metrics and a silent failure.
@Stono Yes, the fix is right. I am entirely with you on the danger of EnvoyFilters.
RE: schema, I think we should do a better job in istio on enforcing JSON format. Protobuf is very lax so some gray areas like duplicate keys are accepted by protobuf parser, but not other common JSON parsers.@Stono is this OK to close now?
It depends, do we not want to tackle the UX issue here in some way? It's a breaking change between 1.5 and 1.6 which meant i was left with no metrics and silent failure. 🤷
@kyessenov how hard would it be to write a validator / converter across the parsing regimes ?
@douglas-reid Do you mean a validator for EnvoyFilters? There is a long list of tasks to make EnvoyFilter safe between upgrades (see #26627). It's not even possible to do it completely, but we want to carve out a subset that's reasonable.
For this issue specifically, I'm not sure where to put the tool. Istio does not have any custom code for extensions besides EnvoyFilters. The extension itself is not appropriate since we removed protobuf to make it smaller (the parser from protobuf easily takes a megabyte of binary, more than the extension itself).
This is the only place we use it
Should metrics not working be possible to consider as a failure, so that canary upgrades would cause a rollback?
Yes, In fact a traffic metrics based canary would automatically stop progress when it suddenly cannot find metrics.
@ssuchter I believe metrics not working is what @Stono caught in his staging environments when trying this upgrade. One thing we should be cautious of when thinking about automated rollbacks based on metrics is that they are entirely customizable. For instance, if we were looking for server-side metrics, none of @Stono's rollouts would ever succeed, as they are entirely disabled in their environment (iirc).
Agreed, canary progression should use the metrics the user expects to be present, so that mechanism should be pluggable as well.
@douglas-reid that's right, we only use
sourcemetrics.Is there anything to do on this issue, or is it just a learning piece that we can close off?
🚧 This issue or pull request has been closed due to not having had activity from an Istio team member since 2020-10-16. If you feel this issue or pull request deserves attention, please reopen the issue. Please see this wiki page for more information. Thank you for your contributions.
Created by the issue and PR lifecycle manager.
Bug description
As I've blogged about - it's really important for us that we're able to reduce the cardinality of metrics at the
Sidecar.We did this using an
EnvoyFilterconfigured with:This works fine on
1.5, but on1.6we get no metrics at all. Switching it back to the default:And we start getting metrics, but with all the dimensions we're not interested in.
I'm guessing something has changed about this config between 1.5 and 1.6 but I cannot see anything in documentation to tell me what/why.
It also really highlights how dangerous EnvoyFilters are for configuration rather than first class API's.
cc @howardjohn @douglas-reid
[ ] Docs
[ ] Installation
[ ] Networking
[ ] Performance and Scalability
[x] Extensions and Telemetry
[ ] Security
[ ] Test and Release
[x] User Experience
[x] Developer Infrastructure
Expected behavior
Metrics to work
Steps to reproduce the bug
Version (include the output of
istioctl version --remoteandkubectl version --shortandhelm versionif you used Helm)1.6
How was Istio installed?
Helm
Environment where bug was observed (cloud vendor, OS, etc)