stats: utilize expiration

[kyessenov]

Change-Id: I97c10eac08443566eca6f53dc73b86c2e51fa095

Replacing rotating scopes with the more accurate per-stat expiration implementation envoyproxy/envoy#40395

[istio-policy-bot]

😊 Welcome @kyessenov! This is either your first contribution to the Istio proxy repo, or it's been
a while since you've been here.

You can learn more about the Istio working groups, Code of Conduct, and contribution guidelines
by referring to Contributing to Istio.

Thanks for contributing!

Courtesy of your friendly welcome wagon.

[howardjohn]

qq - the envoy PR seems to suggest it only works with otlp (or other delta protocols) while I think the old istio approach worked with Prometheus as well. Is the Prometheus behavior changed by this pr?

[kyessenov]

The old behavior didn't work with prometheus either AFAICT, so it didn't change. The behavior is the same if you create all stats at once, and then wait some time.

[howardjohn]

I believe it did which is why there was the timer - to make sure it was sufficiently large to ensure Prometheus had time to scrape?

@zirain would know for sure

[zirain]

just for note, we need to remove all related stuffs(e.g. the env flags) from istio since it won't work anymore.

[kyessenov]

Yeah, that's a follow up. There's a difference in configuration since this is now a per-Envoy config, so maybe a pod annotation rather than telemetry API.

[zirain]

or just mentioned in custom bootstrap.

[zirain]

qq - the envoy PR seems to suggest it only works with otlp (or other delta protocols) while I think the old istio approach worked with Prometheus as well. Is the Prometheus behavior changed by this pr?

IIUC, this worked at the underlying mechanism of envoy. which means it works for both metric service and /stats/prometheus endpoint.

The old behavior didn't work with prometheus either AFAICT, so it didn't change. The behavior is the same if you create all stats at once, and then wait some time.

this new behavior will work if you enable stats_eviction in bootstrap, is that right?

[kyessenov]

qq - the envoy PR seems to suggest it only works with otlp (or other delta protocols) while I think the old istio approach worked with Prometheus as well. Is the Prometheus behavior changed by this pr?

IIUC, this worked at the underlying mechanism of envoy. which means it works for both metric service and /stats/prometheus endpoint.

It worked in the sense that prometheus would handle disappearing stats, even though that's not normal. The difference is that before a stat would disappear all the time, but now only when it's not being emitted.

The old behavior didn't work with prometheus either AFAICT, so it didn't change. The behavior is the same if you create all stats at once, and then wait some time.

this new behavior will work if you enable stats_eviction in bootstrap, is that right?

Yes, it requires that field to turn on. It does nothing without it.

[zirain]

LGTM, thanks!

This make things easier.

[ringerc]

this new behavior will work if you enable stats_eviction in bootstrap, is that right?

It sounds like that's what istio/api#3562 and istio/istio#57736 do, right?

[zirain]

this new behavior will work if you enable stats_eviction in bootstrap, is that right?

It sounds like that's what istio/api#3562 and istio/istio#57736 do, right?

bingo

[ringerc]

Docs update in istio/istio.io#17043