Removed yolo String usage for remote write; Renamed yolo Lables to ZLabels; Added further benchmarks. #3279

bwplotka · 2020-10-06T11:35:47Z

Reason for leak: It was not leaking (: It was just over allocating a lot for every new series that comes, causing overall more memory used then usual. This is the consequence of using yolo string (no copy for string during deserialize). When we do so, we reuse some of bytes buffered in order to parse full protobuf. When we keep just labels (e.g in TSDB for new series) we actually started to keep full protobuf bytes, not just what is used for labels string. I checked and it's not worth to copy manually when needed. Not much benefits seen on benchmarks and more complex code, so removed ZLabel usage from remote write.

(Base: nocpy labels vs normal unmarshal)

benchstat -delta-test=none _dev/diffd.out _dev/diffe.out
name                                                                                   old time/op    new time/op    delta
HandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/OK-12                         19.5µs ± 0%    20.2µs ± 0%   +3.44%
HandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/conflict_errors-12            27.2µs ± 0%    26.1µs ± 0%   -4.29%
HandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/OK-12                        25.4ms ± 0%    27.5ms ± 0%   +8.09%
HandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/conflict_errors-12           60.2ms ± 0%    56.3ms ± 0%   -6.58%
HandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/OK-12                           25.9µs ± 0%    22.0µs ± 0%  -14.97%
HandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/conflict_errors-12              28.3µs ± 0%    28.4µs ± 0%   +0.32%
HandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/OK-12                          25.2ms ± 0%    27.3ms ± 0%   +8.22%
HandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/conflict_errors-12             51.9ms ± 0%    61.3ms ± 0%  +18.05%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/OK-12                 14.3ms ± 0%    14.5ms ± 0%   +1.29%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/conflict_errors-12    12.2ms ± 0%    13.1ms ± 0%   +6.85%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/OK-12                   44.5ms ± 0%    42.6ms ± 0%   -4.34%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/conflict_errors-12      64.7ms ± 0%    66.3ms ± 0%   +2.37%

name                                                                                   old alloc/op   new alloc/op   delta
HandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/OK-12                         6.65kB ± 0%    6.33kB ± 0%   -4.84%
HandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/conflict_errors-12            9.66kB ± 0%    9.32kB ± 0%   -3.44%
HandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/OK-12                        15.5MB ± 0%    13.8MB ± 0%  -10.84%
HandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/conflict_errors-12           45.4MB ± 0%    45.4MB ± 0%   -0.00%
HandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/OK-12                           7.05kB ± 0%    6.73kB ± 0%   -4.57%
HandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/conflict_errors-12              10.1kB ± 0%     9.8kB ± 0%   -3.05%
HandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/OK-12                          14.1MB ± 0%    13.5MB ± 0%   -3.81%
HandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/conflict_errors-12             45.4MB ± 0%    45.4MB ± 0%   -0.01%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/OK-12                 22.5MB ± 0%    22.5MB ± 0%   +0.04%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/conflict_errors-12    22.5MB ± 0%    22.5MB ± 0%   +0.00%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/OK-12                   42.8MB ± 0%    43.9MB ± 0%   +2.53%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/conflict_errors-12      67.8MB ± 0%    67.8MB ± 0%   +0.00%

name                                                                                   old allocs/op  new allocs/op  delta
HandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/OK-12                           68.0 ± 0%      67.0 ± 0%   -1.47%
HandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/conflict_errors-12               111 ± 0%       110 ± 0%   -0.90%
HandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/OK-12                         5.14k ± 0%     5.14k ± 0%   -0.10%
HandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/conflict_errors-12             500k ± 0%      500k ± 0%   -0.00%
HandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/OK-12                             68.0 ± 0%      67.0 ± 0%   -1.47%
HandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/conflict_errors-12                 111 ± 0%       110 ± 0%   -0.90%
HandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/OK-12                           5.16k ± 0%     5.14k ± 0%   -0.41%
HandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/conflict_errors-12               500k ± 0%      500k ± 0%   -0.00%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/OK-12                   47.0 ± 0%      45.0 ± 0%   -4.26%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/conflict_errors-12      89.0 ± 0%      87.0 ± 0%   -2.25%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/OK-12                    5.11k ± 0%     5.11k ± 0%    0.00%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/conflict_errors-12        500k ± 0%      500k ± 0%   -0.00%

Keeping for rest though.

Changes

Optimized Hash functions for both Prometheus and Thanos (used on every write)
Optimized conflict error write path
Renamed Label to ZLabel for more explicit naming
FullCopyLabel is just Label now.
LabelSet is now ZLabelSet for explicit naming.
Added benchmarks.

Signed-off-by: Bartlomiej Plotka bwplotka@gmail.com

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP-12 2 7898665921 ns/op 16280636468 B/op 28550 allocs/op PASS On top of that 15GB per writing single lab?! Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

brancz · 2020-10-07T08:29:54Z

Let us know when/if this is ready for a review :)

/tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive -test.v -test.bench ^\QBenchmarkHandlerReceiveHTTP\E$ -test.run ^$ -test.benchtime 10s goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP-12 2 9450654058 ns/op 21524818740 B/op 30866 allocs/op PASS Process finished with exit code 0 More allocs, but really also leaking. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

/tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive -test.v -test.bench ^\QBenchmarkHandlerReceiveHTTP\E$ -test.run ^$ -test.benchtime 10s goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/OK BenchmarkHandlerReceiveHTTP/OK-12 620 20486199 ns/op 43042842 B/op 61 allocs/op BenchmarkHandlerReceiveHTTP/conflict_errors BenchmarkHandlerReceiveHTTP/conflict_errors-12 72 166820856 ns/op 90261692 B/op 123 allocs/op PASS Crazy consumption on large (2x) labels. Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

/tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive -test.v -test.bench ^\QBenchmarkHandlerReceiveHTTP\E$ -test.run ^$ -test.benchtime 10s goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/OK handler_test.go:1031: skip --- SKIP: BenchmarkHandlerReceiveHTTP/OK BenchmarkHandlerReceiveHTTP/conflict_errors BenchmarkHandlerReceiveHTTP/conflict_errors-12 602 19169514 ns/op 43048217 B/op 109 allocs/op PASS (.String()) can be evaled later on only for debug levels.

/tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive -test.v -test.bench ^\QBenchmarkHandlerReceiveHTTP\E$ -test.run ^$ -test.benchtime 10s goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/OK handler_test.go:1031: skip --- SKIP: BenchmarkHandlerReceiveHTTP/OK BenchmarkHandlerReceiveHTTP/conflict_errors BenchmarkHandlerReceiveHTTP/conflict_errors-12 608 19246095 ns/op 43492837 B/op 102 allocs/op PASS Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

oarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB/OK-12 16 77844680 ns/op 20103398 B/op 5145 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB/conflict_errors-12 12 98204422 ns/op 45388960 B/op 500129 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB/OK BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB/OK-12 14 91293983 ns/op 13586843 B/op 5096 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB/conflict_errors BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB/conflict_errors-12 12 115822519 ns/op 45388295 B/op 500126 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB/OK handler_test.go:1106: handler_test.go:1106:got non 200 error: context deadline exceeded exp: 200 got: 500 --- FAIL: BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB/OK handler_test.go:1144: handler_test.go:1144:got non 200 error: context deadline exceeded exp: 200 got: 500 --- FAIL: BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB --- FAIL: BenchmarkHandlerReceiveHTTP FAIL Process finished with exit code 1 Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

/tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive -test.v -test.bench ^\QBenchmarkHandlerReceiveHTTP\E$ -test.run ^$ goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/OK-12 56775 19741 ns/op 5369 B/op 47 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/conflict_errors-12 56403 26993 ns/op 8375 B/op 90 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/OK-12 12 87911345 ns/op 16402942 B/op 5149 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/conflict_errors-12 12 103342964 ns/op 45391968 B/op 500133 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/OK BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/OK-12 60825 18656 ns/op 5447 B/op 47 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/conflict_errors BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/conflict_errors-12 60274 23760 ns/op 8464 B/op 90 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/OK BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/OK-12 12 91463233 ns/op 16401870 B/op 5145 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/conflict_errors BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/conflict_errors-12 9 123664888 ns/op 45390874 B/op 500135 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/OK-12 97 12629487 ns/op 12033129 B/op 42 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/conflict_errors-12 97 11787359 ns/op 11993940 B/op 85 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/OK handler_test.go:1142: handler_test.go:1142:got non 200 error: context deadline exceeded exp: 200 got: 500 --- FAIL: BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/OK handler_test.go:1180: handler_test.go:1180:got non 200 error: context deadline exceeded exp: 200 got: 500 --- FAIL: BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples --- FAIL: BenchmarkHandlerReceiveHTTP FAIL Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

GOROOT=/home/bwplotka/.gvm/gos/go1.15 #gosetup GOPATH=/home/bwplotka/Repos/thanosgopath #gosetup /home/bwplotka/.gvm/gos/go1.15/bin/go test -c -o /tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive github.com/thanos-io/thanos/pkg/receive #gosetup /tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive -test.v -test.bench ^\QBenchmarkHandlerReceiveHTTP\E$ -test.run ^$ goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/OK-12 73584 18173 ns/op 5375 B/op 47 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/conflict_errors-12 60162 19739 ns/op 8374 B/op 89 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/OK-12 56 23332356 ns/op 16629633 B/op 5132 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/conflict_errors-12 28 48608661 ns/op 45387223 B/op 500127 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/OK BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/OK-12 63535 23560 ns/op 5441 B/op 47 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/conflict_errors BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/conflict_errors-12 61261 19384 ns/op 8467 B/op 90 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/OK BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/OK-12 54 21951054 ns/op 15404871 B/op 5135 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/conflict_errors BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/conflict_errors-12 31 43516277 ns/op 45381626 B/op 500120 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/OK-12 102 11764308 ns/op 12033237 B/op 43 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/conflict_errors-12 115 10119612 ns/op 11992988 B/op 84 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/OK-12 36 32795889 ns/op 27302400 B/op 5101 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/conflict_errors-12 20 54012380 ns/op 57343256 B/op 500114 allocs/op PASS Process finished with exit code 0 Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

GOROOT=/home/bwplotka/.gvm/gos/go1.15 #gosetup GOPATH=/home/bwplotka/Repos/thanosgopath #gosetup /home/bwplotka/.gvm/gos/go1.15/bin/go test -c -o /tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive github.com/thanos-io/thanos/pkg/receive #gosetup /tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive -test.v -test.bench ^\QBenchmarkHandlerReceiveHTTP\E$ -test.run ^$ goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/OK-12 64332 17726 ns/op 6650 B/op 68 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/conflict_errors-12 51852 20598 ns/op 9638 B/op 111 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/OK-12 50 23595737 ns/op 15370905 B/op 5145 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/conflict_errors-12 19 54317241 ns/op 45386242 B/op 500143 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/OK BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/OK-12 60478 24363 ns/op 7056 B/op 68 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/conflict_errors BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/conflict_errors-12 53396 24496 ns/op 10067 B/op 111 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/OK BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/OK-12 52 22248700 ns/op 13965277 B/op 5147 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/conflict_errors BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/conflict_errors-12 26 52803484 ns/op 45388146 B/op 500144 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/OK-12 88 12405059 ns/op 22526723 B/op 47 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/conflict_errors-12 51 31550792 ns/op 22485243 B/op 91 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/OK-12 31 36376900 ns/op 40740138 B/op 5101 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/conflict_errors-12 19 54598755 ns/op 67828469 B/op 500118 allocs/op PASS Process finished with exit code 0 Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

GOROOT=/home/bwplotka/.gvm/gos/go1.15 #gosetup GOPATH=/home/bwplotka/Repos/thanosgopath #gosetup /home/bwplotka/.gvm/gos/go1.15/bin/go test -c -o /tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive github.com/thanos-io/thanos/pkg/receive #gosetup /tmp/___BenchmarkHandlerReceiveHTTP_in_github_com_thanos_io_thanos_pkg_receive -test.v -test.bench ^\QBenchmarkHandlerReceiveHTTP\E$ -test.run ^$ goos: linux goarch: amd64 pkg: github.com/thanos-io/thanos/pkg/receive BenchmarkHandlerReceiveHTTP BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/OK-12 60322 17516 ns/op 6336 B/op 67 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_single_sample/conflict_errors-12 62701 21705 ns/op 9330 B/op 110 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/OK BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/OK-12 50 22807733 ns/op 14876916 B/op 5144 allocs/op BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/conflict_errors BenchmarkHandlerReceiveHTTP/typical_labels_under_1KB,_2MB_of_samples/conflict_errors-12 28 43593475 ns/op 45386012 B/op 500142 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/OK BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/OK-12 61938 18519 ns/op 6727 B/op 67 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/conflict_errors BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_single_sample/conflict_errors-12 54048 22010 ns/op 9758 B/op 110 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/OK BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/OK-12 52 24057139 ns/op 15770874 B/op 5159 allocs/op BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/conflict_errors BenchmarkHandlerReceiveHTTP/bigger_labels_over_1KB,_2MB_of_samples/conflict_errors-12 27 47096262 ns/op 45385205 B/op 500140 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/OK-12 88 13340206 ns/op 22526945 B/op 46 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_single_sample/conflict_errors-12 96 12885251 ns/op 22479263 B/op 87 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/OK BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/OK-12 38 34173124 ns/op 39127665 B/op 5102 allocs/op BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/conflict_errors BenchmarkHandlerReceiveHTTP/extremely_large_label_value_10MB,_2MB_samples/conflict_errors-12 19 58231264 ns/op 67827717 B/op 500117 allocs/op PASS Process finished with exit code 0 Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

bwplotka · 2020-10-15T23:21:07Z

Ready for review. I decided to change naming which increased size of this PR a bit...

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

bwplotka · 2020-10-15T23:26:06Z

pkg/receive/writer.go

-			_, err = app.Add(lset, s.Timestamp, s.Value)
+		for i, s := range t.Samples {
+			if i == 0 {
+				ref, err = app.Add(lset, s.Timestamp, s.Value)


Major improvement here.

bwplotka · 2020-10-15T23:26:29Z

pkg/receive/hashring.go

+	// TODO(bwplotka): Labels should be sorted already, consider removing this.
+	sort.Slice(ts.Labels, func(i, j int) bool { return ts.Labels[i].Name < ts.Labels[j].Name })
+
+	return s[(labelpb.HashWithPrefix(tenant, ts.Labels)+n)%uint64(len(s))], nil


New optimized function for hashing.

bwplotka · 2020-10-15T23:27:15Z

pkg/store/labelpb/label.go

-}
-
-func (m *Label) Unmarshal(data []byte) error {
+// Unmarshal unmarshalls gRPC protobuf into ZLabel struct. ZLabel string is directly using bytes passed in `data`.


Noted the behaviour which can be surprising.

bwplotka · 2020-10-15T23:30:27Z

The "leak" removal can be seen on remote write benchmark profiles of inuse space in correct moment ;p

Before:

After (note, no snappy):

bwplotka · 2020-10-15T23:36:10Z

Since it's a big change with extra optimizations, let's fix this against master.

Will fix release with quicker fix disabling all yolo strings for now.

bwplotka · 2020-10-15T23:40:06Z

Most likely to split as well

bwplotka added 2 commits September 28, 2020 17:25

Cut 0.16.0-rc.0 (#3247)

d1435f7

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

Fixed yolo String leak; Added further benchmarks.

e13431e

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

pstibrany mentioned this pull request Oct 6, 2020

Update Thanos to latest master. cortexproject/cortex#3271

Closed

3 tasks

bwplotka added 2 commits October 7, 2020 06:20

Another failed attempt.

f3fc98b

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

bwplotka added 17 commits October 7, 2020 09:33

Lot's to improve, but no leak. Let's check forwarding now.

7a0102b

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

After cleaning it works just fine, no leak.

6a995cc

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

Optimized hash.

90d7c5e

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

Further bench.

24fb5fc

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

Fixed tests.

abaa4b0

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

Large number of samples makes receive stuck.

6afc146

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

Switched to ZLabels.

76bca8c

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

Rename.

18048c1

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

bwplotka requested a review from brancz October 15, 2020 23:20

bwplotka force-pushed the yolo-string-leak branch from f3e91a1 to 18048c1 Compare October 15, 2020 23:20

bwplotka requested review from GiedriusS and kakkoyun October 15, 2020 23:20

bwplotka marked this pull request as ready for review October 15, 2020 23:20

Removed debug code.

136bca6

Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>

bwplotka changed the title ~~Fixed yolo String leak; Added further benchmarks.~~ Removed yolo String usage for remote write; Renamed yolo Lables to ZLabels; Added further benchmarks. Oct 15, 2020

bwplotka commented Oct 15, 2020

View reviewed changes

bwplotka changed the base branch from release-0.16 to master October 15, 2020 23:35

bwplotka mentioned this pull request Oct 15, 2020

Quick fix for remote write "leak". #3327

Merged

bwplotka closed this Oct 16, 2020

bwplotka mentioned this pull request Oct 16, 2020

labelbp: Renamed Label->Zlabel, and FullCopyLabel -> Label. #3330

Merged

bwplotka mentioned this pull request Mar 18, 2021

receive: v0.18.0 memory leak (v0.16.0 regression) #3726

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Removed yolo String usage for remote write; Renamed yolo Lables to ZLabels; Added further benchmarks. #3279

Removed yolo String usage for remote write; Renamed yolo Lables to ZLabels; Added further benchmarks. #3279

bwplotka commented Oct 6, 2020 •

edited

Loading

brancz commented Oct 7, 2020

bwplotka commented Oct 15, 2020

bwplotka Oct 15, 2020

bwplotka Oct 15, 2020

bwplotka Oct 15, 2020

bwplotka commented Oct 15, 2020

bwplotka commented Oct 15, 2020

bwplotka commented Oct 15, 2020

Removed yolo String usage for remote write; Renamed yolo Lables to ZLabels; Added further benchmarks. #3279

Removed yolo String usage for remote write; Renamed yolo Lables to ZLabels; Added further benchmarks. #3279

Conversation

bwplotka commented Oct 6, 2020 • edited Loading

Changes

brancz commented Oct 7, 2020

bwplotka commented Oct 15, 2020

bwplotka Oct 15, 2020

Choose a reason for hiding this comment

bwplotka Oct 15, 2020

Choose a reason for hiding this comment

bwplotka Oct 15, 2020

Choose a reason for hiding this comment

bwplotka commented Oct 15, 2020

bwplotka commented Oct 15, 2020

bwplotka commented Oct 15, 2020

bwplotka commented Oct 6, 2020 •

edited

Loading