Skip to content

Conversation

SungJin1212
Copy link
Member

@SungJin1212 SungJin1212 commented Aug 17, 2025

This PR adds a benchmark to measure OTLP push performance.

The Benchmark results are:

goos: darwin
goarch: arm64
pkg: github.com/cortexproject/cortex/pkg/util/push
cpu: Apple M1 Max
BenchmarkOTLPWriteHandlerPush
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:10,_numHistograms:1
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:10,_numHistograms:1-10         	   26020	     46032 ns/op	   86703 B/op	     534 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:100,_numHistograms:1
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:100,_numHistograms:1-10        	    6900	    179708 ns/op	  232596 B/op	    2638 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1000,_numHistograms:1
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1000,_numHistograms:1-10       	     855	   1391819 ns/op	 1707477 B/op	   23364 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:10
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:10-10         	    7507	    154604 ns/op	  265258 B/op	    2129 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:100
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:100-10        	     871	   1377683 ns/op	 2248653 B/op	   20196 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:1000
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:1000-10       	      73	  14949837 ns/op	22246728 B/op	  200325 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:10,_samplesPerSeries:1,_numHistograms:1
BenchmarkOTLPWriteHandlerPush/numSeries:10,_samplesPerSeries:1,_numHistograms:1-10         	    7959	    153782 ns/op	  231220 B/op	    2070 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:100,_samplesPerSeries:1,_numHistograms:1
BenchmarkOTLPWriteHandlerPush/numSeries:100,_samplesPerSeries:1,_numHistograms:1-10        	     906	   1321136 ns/op	 1718853 B/op	   19181 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1000,_samplesPerSeries:1,_numHistograms:1
BenchmarkOTLPWriteHandlerPush/numSeries:1000,_samplesPerSeries:1,_numHistograms:1-10       	      86	  13179998 ns/op	17437157 B/op	  189383 allocs/op

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
@dosubot dosubot bot added the type/chore Something that needs to be done; not a bug or a feature label Aug 17, 2025
@SungJin1212
Copy link
Member Author

SungJin1212 commented Aug 17, 2025

Currently, our OTLPHandler uses a Prometheus converter, which appears to cause significant memory allocation during the conversion to the Cortex format.

I was thinking we could manage a dedicated Cortex converter, which can directly convert OTLP to Cortex format. This would allow us to skip the intermediate conversion from Prometheus format to Cortex. If the case, the memory allocation will be reduced, but this is expected to come at the cost of tracking overhead.

@yeya24 @friedrichg
WDYT?

@yeya24
Copy link
Contributor

yeya24 commented Aug 17, 2025

Hey @SungJin1212, I think that makes sense. But at the same time, do we have a benchmark to understand OTLP performance? If not let's create one so that we can compare the performance impact and its baseline

@SungJin1212
Copy link
Member Author

@yeya24
Yeah, let me add a Benchmark to show current memory allocations.

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
@SungJin1212
Copy link
Member Author

@yeya24
I added a Benchmark to evaluate push performance.

@SungJin1212 SungJin1212 changed the title Chore: add small optimization to otlp Add OTLP Benchmark to measure push performance Aug 18, 2025
Copy link
Contributor

@yeya24 yeya24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
Copy link
Member

@friedrichg friedrichg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change is correct in idiomatic go to avoid unnecessary slice growth.
But I can't see visible improvement. See my run below, first old change and later new change.

root@e517fff15516:/workspace# go test -benchtime=5s -benchmem -run=^$ -bench ^BenchmarkOTLPWriteHandlerPush$ github.com/cortexproject/cortex/pkg/util/push
goos: linux
goarch: arm64
pkg: github.com/cortexproject/cortex/pkg/util/push
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:10,_numHistograms:1-8         	  201042	     29349 ns/op	   86701 B/op	     538 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:100,_numHistograms:1-8        	   53223	    112784 ns/op	  236191 B/op	    2651 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1000,_numHistograms:1-8       	    6716	    880607 ns/op	 1505077 B/op	   23341 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:10-8         	   59271	    102093 ns/op	  266269 B/op	    2136 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:100-8        	    6506	    923257 ns/op	 2250522 B/op	   20203 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:1000-8       	     613	   9852447 ns/op	22470090 B/op	  200310 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:10,_samplesPerSeries:1,_numHistograms:1-8         	   59464	    101015 ns/op	  232933 B/op	    2078 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:100,_samplesPerSeries:1,_numHistograms:1-8        	    7021	    854859 ns/op	 1737350 B/op	   19197 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1000,_samplesPerSeries:1,_numHistograms:1-8       	     699	   8628978 ns/op	17653098 B/op	  189474 allocs/op
PASS
ok  	github.com/cortexproject/cortex/pkg/util/push	59.630s
root@e517fff15516:/workspace# go test -benchtime=5s -benchmem -run=^$ -bench ^BenchmarkOTLPWriteHandlerPush$ github.com/cortexproject/cortex/pkg/util/push
goos: linux
goarch: arm64
pkg: github.com/cortexproject/cortex/pkg/util/push
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:10,_numHistograms:1-8         	  199512	     29838 ns/op	   86566 B/op	     534 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:100,_numHistograms:1-8        	   51776	    116531 ns/op	  236514 B/op	    2657 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1000,_numHistograms:1-8       	    6588	    912950 ns/op	 1504125 B/op	   23335 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:10-8         	   58074	    105867 ns/op	  264960 B/op	    2129 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:100-8        	    6310	    945764 ns/op	 2241711 B/op	   20196 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1,_samplesPerSeries:1,_numHistograms:1000-8       	     600	  10040917 ns/op	22242130 B/op	  200293 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:10,_samplesPerSeries:1,_numHistograms:1-8         	   58162	    103052 ns/op	  232660 B/op	    2073 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:100,_samplesPerSeries:1,_numHistograms:1-8        	    6865	    870462 ns/op	 1734374 B/op	   19186 allocs/op
BenchmarkOTLPWriteHandlerPush/numSeries:1000,_samplesPerSeries:1,_numHistograms:1-8       	     682	   8783210 ns/op	17645120 B/op	  189453 allocs/op
PASS
ok  	github.com/cortexproject/cortex/pkg/util/push	59.905s

Nevertheless the benchmark is appreciated. Thanks!

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Sep 1, 2025
Signed-off-by: SungJin1212 <tjdwls1201@gmail.com>
@friedrichg friedrichg merged commit 0b9befb into cortexproject:master Sep 1, 2025
18 checks passed
@SungJin1212 SungJin1212 mentioned this pull request Sep 5, 2025
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lgtm This PR has been approved by a maintainer size/L type/chore Something that needs to be done; not a bug or a feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants