Commit 5b30bdb610a8fe7019edbf23bfc10412309ca5b8 increases system load from ~3 to ~9

[RichiH]

On my system as per IRC:

5b30bdb is the culprit.

git bisect start
# bad: [2f151d02eb2092ebdd84778944533ddb57b56103] Merge pull request #1456 from prometheus/validate-alertmanager-url
git bisect bad 2f151d02eb2092ebdd84778944533ddb57b56103
# good: [e11fab35d76d19c5c49b7d85e28275f894d3ada4] Bump version to 0.17.0
git bisect good e11fab35d76d19c5c49b7d85e28275f894d3ada4
# good: [8fa1560e489eaa39eec31f2f930a49b9d443c230] Fix a very special case of handling the checkpoint timer
git bisect good 8fa1560e489eaa39eec31f2f930a49b9d443c230
# good: [73399f826a11a2032c36a5346d176ae8d2d9edfe] Merge pull request #1427 from prometheus/fix-scrape-timeout
git bisect good 73399f826a11a2032c36a5346d176ae8d2d9edfe
# skip: [ddc74f712b05a26af1c7463fbb6003c78a53329b] Add sortable target list
git bisect skip ddc74f712b05a26af1c7463fbb6003c78a53329b
# good: [7763bbd99316681bca87930fdde0f1739a2a4d74] Validate alertmanager URL
git bisect good 7763bbd99316681bca87930fdde0f1739a2a4d74
# bad: [50c2f20756aa37afc75e5ccd2871a3238e0e2d8e] Add targetScraper tests
git bisect bad 50c2f20756aa37afc75e5ccd2871a3238e0e2d8e
# bad: [fbe251c2df399b89bf87a11352ab26aff676d8d0] Fix scrape interval length calculation
git bisect bad fbe251c2df399b89bf87a11352ab26aff676d8d0
# bad: [cebba3efbb143d35a9ffdb01930cc648235beaa3] Simplify and fix TargetManager reloading
git bisect bad cebba3efbb143d35a9ffdb01930cc648235beaa3
# bad: [5b30bdb610a8fe7019edbf23bfc10412309ca5b8] Change TargetProvider interface.
git bisect bad 5b30bdb610a8fe7019edbf23bfc10412309ca5b8
# skip: [5bfa4cdd46b9d2ff843f6ffe85e15fc87c614737] Simplify target update handling.
git bisect skip 5bfa4cdd46b9d2ff843f6ffe85e15fc87c614737
# good: [bb6dc3ff783463e0f9fdc915232b546cc0a116b3] Remove old tests
git bisect good bb6dc3ff783463e0f9fdc915232b546cc0a116b3
# first bad commit: [5b30bdb610a8fe7019edbf23bfc10412309ca5b8] Change TargetProvider interface.

[fabxc]

@beorn7 just did some profiling that showed performance consumption in the target provider handling. Found the probable issue for that and it adds up with the commit you bisected.

Fix incoming.

[RichiH]

Cross-reference: #1474

[RichiH]

System load with different builds: https://snapshot.raintank.io/dashboard/snapshot/fN9YwxGe8X9lxmQOwiK42pi6D6XQIo7L

[RichiH]

Longer term data, with added ingestion rates: https://snapshot.raintank.io/dashboard/snapshot/Xd63j36O1R1RHFMutWw6trQwxBp438yg

[brian-brazil]

Is this resolved?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.