[Regression] Sample ingestion and latency worse after upgrade to 1.5.0

[RichiH]

After upgrading to 1.5.0 from 1.4.1, I see increased jitter in sample ingestion and higher internal latencies. The upgrade time should be obvious fromt the graphs. The drop-off at the end is due to a system reload after a kernel upgrade.

https://snapshot.raintank.io/dashboard/snapshot/US5WvmgcNdtwZjCZNPHgilvt86G2VutD
https://snapshot.raintank.io/dashboard/snapshot/4t87AZipjxQL4Ia2r7efusm8qjXzalN1

[brian-brazil]

Can you get a zoomed in graph with https://grafana.net/dashboards/1244 ?

[RichiH]

https://snapshot.raintank.io/dashboard/snapshot/tiZVFVsAOSUMs4TRd8rzXCucvBf0yE7k

[brian-brazil]

Are you sure that's of the same server? The ingestion rate isn't the same.

[RichiH]

Positive, yes. I can't say for sure when teams change their monitored targets, but these were stable over time:

sum(up) by (job)
count(up) by (job)

[RichiH]

Well, and not "by job"

[brian-brazil]

There's a difference of 2k/s between those two. Can you get me both graphs zoomed into a 30m period?

[RichiH]

New and correct time frame:
https://snapshot.raintank.io/dashboard/snapshot/7dJAdQ0VTo9oEQ06JUOE2HncWpqw2Y3X
https://snapshot.raintank.io/dashboard/snapshot/ylKqITu6MTjauGts54WxwtNABx1C5MLZ

[RichiH]

Your dashboard:
https://snapshot.raintank.io/dashboard/snapshot/agcwyaN1RGX7J1mOQkfE2xgbG7GyVGIs

[brian-brazil]

The ingestion rate looks fairly normal there, the effect you see in the initial report is due to aliasing.

I'm not sure exactly what's going on with the evals. My guess is that due to reduced memory usage, GC is happening more often and thus pushing out the tail. Can you graph the average latency with an irate?

[RichiH]

prometheus_notifications_latency_seconds only contains quantiles; what precisely do you want the average of?

[brian-brazil]

They're summaries, use the _sum and _count.

[RichiH]

https://snapshot.raintank.io/dashboard/snapshot/4XwIW6UZ7gOMFBq5CDddglSaB7sk3IUb

[RichiH]

That's irate([2m])

[brian-brazil]

Can you zoom that in and add the rules too?

[RichiH]

https://snapshot.raintank.io/dashboard/snapshot/GBATO8CWUV68cMURPzijeXINK9NtjDGd

[brian-brazil]

That looks less worrying. The only correlation I see for the spikes is with checkpoints completing. There's also a minor correlation with indexing, so maybe there's something around the rules creating new timeseries.

Have there been changes in your environment that'd effect what the rules were doing?

[RichiH]

TTBOMK, there have been no changes other than me compiling and restarting Prometheus.

[brian-brazil]

The graphs indicate some form of change to the indexing.

[RichiH]

I see no change in count(up) during that time.

[jeinwag]

I'm currently seeing a similar issue, I upgraded from 1.4.1 to 1.5.0 a little over an hour ago at ca. 15:30:
https://snapshot.raintank.io/dashboard/snapshot/h5F1k4mvnOaO1uvJCH2huebFc8wl5OyI

This isn't due to changes in our environment.

EDIT: Link to benchmark dashboard:
https://snapshot.raintank.io/dashboard/snapshot/PmQdXZ9lZZLAg77GoPzS8ukQyNph1gwJ

EDIT2: It looks like it's got to do with consul service discovery, at least in our case. The "up" count of jobs which use consul sd is fluctuating since the update, jobs with static target definitions are fine.

I'm now certain that my issue is rather related to #2377.

[beorn7]

I'm currently assessing performance impact at SC. So far, everything is pointing to "faster with less RAM". :)

[beorn7]

Wild guess; Could you try to set -storage.local.num-fingerprint-mutexes=100000 on the command line?

It could be that the 1.5.x creates more lock contention, and the flag above would help with that. And I'm genuinely curious if it does.

[brian-brazil]

This seems to have been the relabelling issue, combined with graph artifacts. 1.5.2 should work fine.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.