receive: Improved efficiency of multitsdb appends, upgraded Prometheus deps.

[bwplotka]

Finally porting all improvements from https://github.com/thanos-io/thanos/pull/3334/files:

• Prometheus upgrade (no perf change, but required for new GetRef interface)
• ReadAll to Grow + Copy (20% less allocation in worse alloc case)
• Moved hashring to the optimized hash function (up to 31% less mem, up to 20% less allocations)
• Used Prometheus GetRef to avoid reallocating the same series. (up to 70% less allocs, 45% less mem)

Result:

benchstat -delta-test=none ../_dev/thanos/2021/receive2/impr1-go1.16.3.txt ../_dev/thanos/2021/receive2/impr4-go1.16.3.txt
name                                                                                old time/op    new time/op    delta
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12                        1.54ms ± 0%    1.15ms ± 0%  -25.44%
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12           6.96ms ± 0%    5.57ms ± 0%  -19.95%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12                       16.1ms ± 0%    11.5ms ± 0%  -28.35%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12          65.4ms ± 0%    58.8ms ± 0%  -10.10%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12                  168ms ± 0%     114ms ± 0%  -32.00%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12     1.69s ± 0%     1.43s ± 0%  -15.33%

name                                                                                old alloc/op   new alloc/op   delta
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12                        1.75MB ± 0%    1.15MB ± 0%  -34.01%
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12           4.89MB ± 0%    4.29MB ± 0%  -12.32%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12                       18.8MB ± 0%    12.8MB ± 0%  -31.97%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12          50.1MB ± 0%    44.1MB ± 0%  -12.13%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12                  344MB ± 0%     120MB ± 0%  -65.16%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12     816MB ± 0%     592MB ± 0%  -27.44%

name                                                                                old allocs/op  new allocs/op  delta
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12                         15.6k ± 0%      3.6k ± 0%  -76.97%
HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12            35.6k ± 0%     23.6k ± 0%  -33.71%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12                         155k ± 0%       35k ± 0%  -77.22%
HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12            355k ± 0%      235k ± 0%  -33.79%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12                    170 ± 0%        79 ± 0%  -53.53%
HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12       439 ± 0%       360 ± 0%  -18.00%

Signed-off-by: Bartlomiej Plotka bwplotka@gmail.com

[onprem]

I am not seeing any changes other then upgrading dependencies and some changes in the test files. Did you forgot to push some changes (or maybe I am just dumb :P)?

[bwplotka]

Yea, sorry, updated 🤦🏽

[kakkoyun]

Looking so good. Thanks for the benchmarks and detailed code comments.
I have a couple of questions though.

[bwplotka]

I addressed comments @kakkoyun and redid the benchmarks (and edited description)