Reduce constrainLabels allocations #1272

khasanovbi · 2023-05-13T17:29:47Z

Hi, in my application prometheus allocates a lot of memory in constrainLabels.
I suggest replacing labels inplace instead of creating a new map.

bwplotka

Thanks for this!

Hm, this seems great initially, but we need to double check if it's not breaking change or problematic for people or our code. Generally reusing input memory is considered as a side effect of the function, which can cause hard to find bugs for people (modifying in place).

cc @beorn7 do you know from the top of your head if this semantics trigger some bad memories ? (((:

bwplotka · 2023-05-26T11:48:33Z

prometheus/vec.go

 	}
-	return constrainedValues
 }

 func constrainLabelValues(desc *Desc, lvs []string, curry []curriedLabelValue) []string {


Should we do similar here for consistency?

Signed-off-by: Bulat Khasanov <afti@yandex.ru>

khasanovbi · 2023-05-26T12:26:44Z

Generally reusing input memory is considered as a side effect of the function, which can cause hard to find bugs for people (modifying in place).

Ok, looks like it better to use sync.Pool to prevent side effects.

khasanovbi · 2023-05-26T12:28:28Z

I added a benchmark.

Results in main branch:

goos: linux
goarch: amd64
pkg: github.com/prometheus/client_golang/prometheus
cpu: Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
BenchmarkMetricVecWithBasic
BenchmarkMetricVecWithBasic-12    	 1982983	       628.3 ns/op	     400 B/op	       4 allocs/op

In PR:

goos: linux
goarch: amd64
pkg: github.com/prometheus/client_golang/prometheus
cpu: Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz
BenchmarkMetricVecWithBasic
BenchmarkMetricVecWithBasic-12    	 3176462	       389.7 ns/op	      64 B/op	       2 allocs/op

PS. Currently maps are not shrinking in Go golang/go#20135

beorn7 · 2023-05-28T22:11:50Z

We dave certainly done a bunch of quite daring optimizations in client_golang in the past. It is, after all, often in a very hot code path of a lot of programs.

I have to leave the detailed code review to you, but at first glance, I don't see any red flags.

bwplotka

This is great iteration and safe. We can experiment with more aggressive label reuse, but this is already a big improvement.

Also, I just noticed: Constrained labels is very rarely used feature (new one). I believe there is room to never allocate anything and do it only if metric HAS some constrained labels defined. WDYT? It's essentially a little bit of regression and thanks for noticing this!

Thanks 💪🏽

* Add bench Signed-off-by: Bulat Khasanov <afti@yandex.ru> * Reduce constrainLabels allocations Signed-off-by: Bulat Khasanov <afti@yandex.ru> --------- Signed-off-by: Bulat Khasanov <afti@yandex.ru> Signed-off-by: krishnaduttPanchagnula <krishnadutt123@gmail.com>

….0 (#709) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/prometheus/client_golang](https://togithub.com/prometheus/client_golang) | require | minor | `v1.15.1` -> `v1.16.0` | --- ### Release Notes <details> <summary>prometheus/client_golang</summary> ### [`v1.16.0`](https://togithub.com/prometheus/client_golang/releases/tag/v1.16.0) [Compare Source](https://togithub.com/prometheus/client_golang/compare/v1.15.1...v1.16.0) #### What's Changed - \[BUGFIX] api: Switch to POST for LabelNames, Series, and QueryExemplars. [#1252](https://togithub.com/prometheus/client_golang/issues/1252) - \[BUGFIX] api: Fix undefined execution order in return statements. [#1260](https://togithub.com/prometheus/client_golang/issues/1260) - \[BUGFIX] native histograms: Fix bug in bucket key calculation. [#1279](https://togithub.com/prometheus/client_golang/issues/1279) - \[ENHANCEMENT] Reduce constrainLabels allocations for all metrics. [#1272](https://togithub.com/prometheus/client_golang/issues/1272) - \[ENHANCEMENT] promhttp: Add process start time header for scrape efficiency. [#1278](https://togithub.com/prometheus/client_golang/issues/1278) - \[ENHANCEMENT] promlint: Improve metricUnits runtime. [#1286](https://togithub.com/prometheus/client_golang/issues/1286) <details> <summary> Commits </summary> - Merge v1.15 to main by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1250 - Switch to POST for LabelNames, Series, and QueryExemplars to DoGetFallback by [@jacksontj](https://togithub.com/jacksontj) in [prometheus/client_golang#1252 - ✏️ \[collectors]: fix typo in test assertion by [@vegerot](https://togithub.com/vegerot) in [prometheus/client_golang#1153 - Added interactive tutorial \[kubeCon] by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1255 - Fixed tutorial. by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1256 - Bump golang.org/x/sys from 0.6.0 to 0.7.0 by [@dependabot](https://togithub.com/dependabot) in [prometheus/client_golang#1265 - Cleanup proto use in tests by [@SuperQ](https://togithub.com/SuperQ) in [prometheus/client_golang#1264 - Fix tutorial on WSL-based systems by [@marevers](https://togithub.com/marevers) in [prometheus/client_golang#1257 - Fix undefined execution order in return statements by [@PiotrLewandowski323](https://togithub.com/PiotrLewandowski323) in [prometheus/client_golang#1260 - Merge release 1.15.1 to main by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1267 - GitHub Workflows security hardening by [@sashashura](https://togithub.com/sashashura) in [prometheus/client_golang#1180 - add process start time header to client_golang prometheus by [@logicalhan](https://togithub.com/logicalhan) in [prometheus/client_golang#1278 - Fix bug in bucket key calculation by [@beorn7](https://togithub.com/beorn7) in [prometheus/client_golang#1279 - Bump github.com/prometheus/procfs from 0.9.0 to 0.10.1 by [@dependabot](https://togithub.com/dependabot) in [prometheus/client_golang#1283 - Reduce constrainLabels allocations by [@khasanovbi](https://togithub.com/khasanovbi) in [prometheus/client_golang#1272 - added circleci as gh action YAML by [@krishnaduttPanchagnula](https://togithub.com/krishnaduttPanchagnula) in [prometheus/client_golang#1281 - Improve metricUnits runtime by [@avlitman](https://togithub.com/avlitman) in [prometheus/client_golang#1286 - Moving fully to GH actions. by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1288 - Fix docstring references to renamed native histogram fields / functions. by [@juliusv](https://togithub.com/juliusv) in [prometheus/client_golang#1290 - Fixed README & CHANGELOG; Added fmt makefile command (+bingo) for easier contributions. by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1289 - </details> #### New Contributors * @vegerot made their first contributi[prometheus/client_golang#1153 * @marevers made their first contributi[prometheus/client_golang#1257 * @PiotrLewandowski323 made their first contributi[prometheus/client_golang#1260 * @sashashura made their first contributi[prometheus/client_golang#1180 * @logicalhan made their first contributi[prometheus/client_golang#1278 * @khasanovbi made their first contributi[prometheus/client_golang#1272 * @krishnaduttPanchagnula made their first contributi[prometheus/client_golang#1281 * @avlitman made their first contributi[prometheus/client_golang#1286 **Full Changelog**: prometheus/client_golang@v1.15.1...v1.16.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/open-feature/flagd).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

) [![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/prometheus/client_golang](https://togithub.com/prometheus/client_golang) | require | minor | `v1.15.1` -> `v1.16.0` | --- ### Release Notes <details> <summary>prometheus/client_golang</summary> ### [`v1.16.0`](https://togithub.com/prometheus/client_golang/releases/tag/v1.16.0) [Compare Source](https://togithub.com/prometheus/client_golang/compare/v1.15.1...v1.16.0) #### What's Changed - \[BUGFIX] api: Switch to POST for LabelNames, Series, and QueryExemplars. [#1252](https://togithub.com/prometheus/client_golang/issues/1252) - \[BUGFIX] api: Fix undefined execution order in return statements. [#1260](https://togithub.com/prometheus/client_golang/issues/1260) - \[BUGFIX] native histograms: Fix bug in bucket key calculation. [#1279](https://togithub.com/prometheus/client_golang/issues/1279) - \[ENHANCEMENT] Reduce constrainLabels allocations for all metrics. [#1272](https://togithub.com/prometheus/client_golang/issues/1272) - \[ENHANCEMENT] promhttp: Add process start time header for scrape efficiency. [#1278](https://togithub.com/prometheus/client_golang/issues/1278) - \[ENHANCEMENT] promlint: Improve metricUnits runtime. [#1286](https://togithub.com/prometheus/client_golang/issues/1286) <details> <summary> Commits </summary> - Merge v1.15 to main by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1250 - Switch to POST for LabelNames, Series, and QueryExemplars to DoGetFallback by [@jacksontj](https://togithub.com/jacksontj) in [prometheus/client_golang#1252 - ✏️ \[collectors]: fix typo in test assertion by [@vegerot](https://togithub.com/vegerot) in [prometheus/client_golang#1153 - Added interactive tutorial \[kubeCon] by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1255 - Fixed tutorial. by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1256 - Bump golang.org/x/sys from 0.6.0 to 0.7.0 by [@dependabot](https://togithub.com/dependabot) in [prometheus/client_golang#1265 - Cleanup proto use in tests by [@SuperQ](https://togithub.com/SuperQ) in [prometheus/client_golang#1264 - Fix tutorial on WSL-based systems by [@marevers](https://togithub.com/marevers) in [prometheus/client_golang#1257 - Fix undefined execution order in return statements by [@PiotrLewandowski323](https://togithub.com/PiotrLewandowski323) in [prometheus/client_golang#1260 - Merge release 1.15.1 to main by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1267 - GitHub Workflows security hardening by [@sashashura](https://togithub.com/sashashura) in [prometheus/client_golang#1180 - add process start time header to client_golang prometheus by [@logicalhan](https://togithub.com/logicalhan) in [prometheus/client_golang#1278 - Fix bug in bucket key calculation by [@beorn7](https://togithub.com/beorn7) in [prometheus/client_golang#1279 - Bump github.com/prometheus/procfs from 0.9.0 to 0.10.1 by [@dependabot](https://togithub.com/dependabot) in [prometheus/client_golang#1283 - Reduce constrainLabels allocations by [@khasanovbi](https://togithub.com/khasanovbi) in [prometheus/client_golang#1272 - added circleci as gh action YAML by [@krishnaduttPanchagnula](https://togithub.com/krishnaduttPanchagnula) in [prometheus/client_golang#1281 - Improve metricUnits runtime by [@avlitman](https://togithub.com/avlitman) in [prometheus/client_golang#1286 - Moving fully to GH actions. by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1288 - Fix docstring references to renamed native histogram fields / functions. by [@juliusv](https://togithub.com/juliusv) in [prometheus/client_golang#1290 - Fixed README & CHANGELOG; Added fmt makefile command (+bingo) for easier contributions. by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1289 - </details> #### New Contributors * @vegerot made their first contributi[prometheus/client_golang#1153 * @marevers made their first contributi[prometheus/client_golang#1257 * @PiotrLewandowski323 made their first contributi[prometheus/client_golang#1260 * @sashashura made their first contributi[prometheus/client_golang#1180 * @logicalhan made their first contributi[prometheus/client_golang#1278 * @khasanovbi made their first contributi[prometheus/client_golang#1272 * @krishnaduttPanchagnula made their first contributi[prometheus/client_golang#1281 * @avlitman made their first contributi[prometheus/client_golang#1286 **Full Changelog**: prometheus/client_golang@v1.15.1...v1.16.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/hetznercloud/hcloud-cloud-controller-manager).  Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>

[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [github.com/prometheus/client_golang](https://togithub.com/prometheus/client_golang) | require | minor | `v1.15.1` -> `v1.16.0` | --- ### Release Notes <details> <summary>prometheus/client_golang</summary> ### [`v1.16.0`](https://togithub.com/prometheus/client_golang/releases/tag/v1.16.0) #### What's Changed - \[BUGFIX] api: Switch to POST for LabelNames, Series, and QueryExemplars. [#1252](https://togithub.com/prometheus/client_golang/issues/1252) - \[BUGFIX] api: Fix undefined execution order in return statements. [#1260](https://togithub.com/prometheus/client_golang/issues/1260) - \[BUGFIX] native histograms: Fix bug in bucket key calculation. [#1279](https://togithub.com/prometheus/client_golang/issues/1279) - \[ENHANCEMENT] Reduce constrainLabels allocations for all metrics. [#1272](https://togithub.com/prometheus/client_golang/issues/1272) - \[ENHANCEMENT] promhttp: Add process start time header for scrape efficiency. [#1278](https://togithub.com/prometheus/client_golang/issues/1278) - \[ENHANCEMENT] promlint: Improve metricUnits runtime. [#1286](https://togithub.com/prometheus/client_golang/issues/1286) <details> <summary> Commits </summary> - Merge v1.15 to main by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1250 - Switch to POST for LabelNames, Series, and QueryExemplars to DoGetFallback by [@jacksontj](https://togithub.com/jacksontj) in [prometheus/client_golang#1252 - ✏️ \[collectors]: fix typo in test assertion by [@vegerot](https://togithub.com/vegerot) in [prometheus/client_golang#1153 - Added interactive tutorial \[kubeCon] by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1255 - Fixed tutorial. by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1256 - Bump golang.org/x/sys from 0.6.0 to 0.7.0 by [@dependabot](https://togithub.com/dependabot) in [prometheus/client_golang#1265 - Cleanup proto use in tests by [@SuperQ](https://togithub.com/SuperQ) in [prometheus/client_golang#1264 - Fix tutorial on WSL-based systems by [@marevers](https://togithub.com/marevers) in [prometheus/client_golang#1257 - Fix undefined execution order in return statements by [@PiotrLewandowski323](https://togithub.com/PiotrLewandowski323) in [prometheus/client_golang#1260 - Merge release 1.15.1 to main by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1267 - GitHub Workflows security hardening by [@sashashura](https://togithub.com/sashashura) in [prometheus/client_golang#1180 - add process start time header to client_golang prometheus by [@logicalhan](https://togithub.com/logicalhan) in [prometheus/client_golang#1278 - Fix bug in bucket key calculation by [@beorn7](https://togithub.com/beorn7) in [prometheus/client_golang#1279 - Bump github.com/prometheus/procfs from 0.9.0 to 0.10.1 by [@dependabot](https://togithub.com/dependabot) in [prometheus/client_golang#1283 - Reduce constrainLabels allocations by [@khasanovbi](https://togithub.com/khasanovbi) in [prometheus/client_golang#1272 - added circleci as gh action YAML by [@krishnaduttPanchagnula](https://togithub.com/krishnaduttPanchagnula) in [prometheus/client_golang#1281 - Improve metricUnits runtime by [@avlitman](https://togithub.com/avlitman) in [prometheus/client_golang#1286 - Moving fully to GH actions. by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1288 - Fix docstring references to renamed native histogram fields / functions. by [@juliusv](https://togithub.com/juliusv) in [prometheus/client_golang#1290 - Fixed README & CHANGELOG; Added fmt makefile command (+bingo) for easier contributions. by [@bwplotka](https://togithub.com/bwplotka) in [prometheus/client_golang#1289 - </details> #### New Contributors * @vegerot made their first contributi[prometheus/client_golang#1153 * @marevers made their first contributi[prometheus/client_golang#1257 * @PiotrLewandowski323 made their first contributi[prometheus/client_golang#1260 * @sashashura made their first contributi[prometheus/client_golang#1180 * @logicalhan made their first contributi[prometheus/client_golang#1278 * @khasanovbi made their first contributi[prometheus/client_golang#1272 * @krishnaduttPanchagnula made their first contributi[prometheus/client_golang#1281 * @avlitman made their first contributi[prometheus/client_golang#1286 **Full Changelog**: prometheus/client_golang@v1.15.1...v1.16.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - "before 6am on Monday" in timezone Europe/Paris, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/cozy/cozy-stack).

bboreham · 2023-08-21T15:47:45Z

Historical note: this code was changed again in #1296

bwplotka · 2023-08-21T18:28:08Z

Thanks for bringing my reviewing mistake again =D (we missed we allocate a lot of memory even if constrained feature is not used)

bboreham · 2023-08-21T21:42:06Z

No shade intended; just took me a little time to trace the history so I added a breadcrumb.

khasanovbi force-pushed the main branch from 53e710e to f06b30a Compare May 13, 2023 17:31

khasanovbi mentioned this pull request May 18, 2023

constrainLabels allocates too many memory #1276

Closed

bwplotka reviewed May 26, 2023

View reviewed changes

khasanovbi added 2 commits May 26, 2023 15:23

Add bench

912d48b

Signed-off-by: Bulat Khasanov <afti@yandex.ru>

Reduce constrainLabels allocations

d46f436

Signed-off-by: Bulat Khasanov <afti@yandex.ru>

khasanovbi force-pushed the main branch from f06b30a to d46f436 Compare May 26, 2023 12:30

sukunrt mentioned this pull request Jun 5, 2023

chore: update deps libp2p/go-libp2p#2332

Merged

This comment was marked as duplicate.

Sign in to view

bwplotka approved these changes Jun 7, 2023

View reviewed changes

bwplotka merged commit a09a1d3 into prometheus:main Jun 7, 2023
7 checks passed

AnshulMalik mentioned this pull request Mar 20, 2024

Allocate constrain label values on stack, not heap for performance #1465

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce constrainLabels allocations #1272

Reduce constrainLabels allocations #1272

khasanovbi commented May 13, 2023

bwplotka left a comment •

edited

bwplotka May 26, 2023 •

edited

khasanovbi commented May 26, 2023 •

edited

khasanovbi commented May 26, 2023 •

edited

beorn7 commented May 28, 2023

This comment was marked as duplicate.

bwplotka left a comment

bboreham commented Aug 21, 2023

bwplotka commented Aug 21, 2023

bboreham commented Aug 21, 2023

Reduce constrainLabels allocations #1272

Reduce constrainLabels allocations #1272

Conversation

khasanovbi commented May 13, 2023

bwplotka left a comment • edited

Choose a reason for hiding this comment

bwplotka May 26, 2023 • edited

Choose a reason for hiding this comment

khasanovbi commented May 26, 2023 • edited

khasanovbi commented May 26, 2023 • edited

beorn7 commented May 28, 2023

This comment was marked as duplicate.

bwplotka left a comment

Choose a reason for hiding this comment

bboreham commented Aug 21, 2023

bwplotka commented Aug 21, 2023

bboreham commented Aug 21, 2023

bwplotka left a comment •

edited

bwplotka May 26, 2023 •

edited

khasanovbi commented May 26, 2023 •

edited

khasanovbi commented May 26, 2023 •

edited