Commit d15adfc917b3d417980c6259c0c8f072f1f82082 increases system load from ~9 to ~16

[RichiH]

As per #1472

d15adfc is the next one

git bisect start
# good: [5b30bdb610a8fe7019edbf23bfc10412309ca5b8] Change TargetProvider interface.
git bisect good 5b30bdb610a8fe7019edbf23bfc10412309ca5b8
# bad: [fbe251c2df399b89bf87a11352ab26aff676d8d0] Fix scrape interval length calculation
git bisect bad fbe251c2df399b89bf87a11352ab26aff676d8d0
# skip: [05de8b7f8d43bc37b7198cad16e9b4760ba255ff] Extract target scraping into scrape loop.
git bisect skip 05de8b7f8d43bc37b7198cad16e9b4760ba255ff
# bad: [da99366f85bb9acc758ecc4ff0eeec43ac0310c7] Consolidate Target.Update into constructor.
git bisect bad da99366f85bb9acc758ecc4ff0eeec43ac0310c7
# bad: [d15adfc917b3d417980c6259c0c8f072f1f82082] Preserve target state across reloads.
git bisect bad d15adfc917b3d417980c6259c0c8f072f1f82082
# first bad commit: [d15adfc917b3d417980c6259c0c8f072f1f82082] Preserve target state across reloads.

[fabxc]

Could you elaborate?
Next one for what? If the issue started in the commit you bisected in #1472, it will likely be there for all subsequent ones.

[RichiH]

The last one went from load ~3 to ~9. This one went from ~9 to ~16.

[fabxc]

I understand. Generally it's super helpful to get the profiling SVG that nicely shows where time is spent.

Basically, run this:

go tool pprof --svg http://<prometheus>/debug/pprof/profile > prof.svg

[RichiH]

Good to know. Do you need that for those two issues?

[fabxc]

Got one for the first one, which is hopefully fixed via #1474.
Would be great for this one.

[RichiH]

Minor nitpick: GitHub does not allow SVG, only PNG, GIF, and JPG.

As they all look different, I made ten and had a sleep 10 in between.

[RichiH]

System load with different builds: https://snapshot.raintank.io/dashboard/snapshot/fN9YwxGe8X9lxmQOwiK42pi6D6XQIo7L

[RichiH]

Longer term data, with added ingestion rates: https://snapshot.raintank.io/dashboard/snapshot/Xd63j36O1R1RHFMutWw6trQwxBp438yg

[brian-brazil]

Is this resolved?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.