status: wasteful re-runs of analyzers leaders to large allocations

[howardjohn]

istio.io/istio/pkg/config/schema/resource.(*schemaImpl).ValidateConfig is calling the validate function on telemetry Envoyfilter many many times, even though it doesn't ever change. As a result, a lot of allocations are made, since its not cheap to validate envoyfilter. This ends up accounting for 5% of all istiod allocations

[therealmitchconnors]

Can you help me understand how this relates to status? Can you provide stack traces, profiles, or other examples of this behavior?

[howardjohn]

It relates to status because the code only runs when status is enabled. This is the analysis. The Envoyfilters are not changing at all, so I am not sure why we are continuously running ValidateConfig on them - the result cannot change, so its just a huge waste of resources.

[jasonwzm]

I think this is because we run ValidateConfig on each resource everytime with a resource change in the snapshot. That was built mostly to replicate istioctl validate in CI/CD. I think we can either remove the analyzer or disable the analyzer for in-cluster analysis.

[howardjohn]

Sorry its not clear to me, what is the point of the status analyzer feature if its disabled? Doesn't that just mean the feature is not enabled?

[jasonwzm]

@howardjohn I meant just this specific analyzer: https://github.com/istio/istio/blob/master/galley/pkg/config/analysis/analyzers/schema/validation.go#L60

[howardjohn]

I see. It seems a bit of a workaround to disable the analyzers.. don't we already disable a lot of the for in-cluster analysis? We lose a lot of value if we just turn them all off, I am worried we are just turning them off and eventually will have none enabled, making the feature not worthwhile

[jasonwzm]

The schema analyzer is not worthwhile running in cluster. So we should disable it anyway.

[jasonwzm]

Also disable analyzer for alpha and deprecated annotations.

[howardjohn]

not stale

…

On Sun, Apr 25, 2021 at 10:19 PM Istio Policy Bot ***@***.***> wrote: 🧭 This issue or pull request has been automatically marked as stale because it has not had activity from an Istio team member since 2021-01-25. It will be closed on 2021-05-10 unless an Istio team member takes action. Please see this wiki page <https://github.com/istio/istio/wiki/Issue-and-Pull-Request-Lifecycle-Manager> for more information. Thank you for your contributions. *Created by the issue and PR lifecycle manager*. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#30200 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEYGXJ5UKS2MAG7SLDQSX3TKTZVVANCNFSM4WKAL7KA> .