receive: Improved efficiency of multitsdb appends, upgraded Prometheus deps. #4078
Commits on Apr 19, 2021
-
receive: Improved efficiency of multitsdb appends.
Release vs current main looks the same: ``` benchstat -delta-test=none ../_dev/thanos/2021/receive/5.txt ../_dev/thanos/2021/receive2/main-go1.15.txt name old time/op new time/op delta HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 1.56ms ± 0% 1.45ms ± 0% -7.12% HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 6.49ms ± 0% 7.14ms ± 0% +9.92% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 16.0ms ± 0% 16.4ms ± 0% +2.79% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 71.7ms ± 0% 69.4ms ± 0% -3.20% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 138ms ± 0% 131ms ± 0% -4.79% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 1.58s ± 0% 1.68s ± 0% +6.11% name old alloc/op new alloc/op delta HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 1.70MB ± 0% 1.70MB ± 0% +0.12% HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 4.84MB ± 0% 4.84MB ± 0% +0.04% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 18.3MB ± 0% 18.2MB ± 0% -0.19% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 49.6MB ± 0% 49.6MB ± 0% +0.00% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 331MB ± 0% 331MB ± 0% -0.00% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 804MB ± 0% 804MB ± 0% +0.00% name old allocs/op new allocs/op delta HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 15.6k ± 0% 15.6k ± 0% +0.04% HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 35.6k ± 0% 35.6k ± 0% +0.01% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 155k ± 0% 155k ± 0% -0.08% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 355k ± 0% 355k ± 0% +0.00% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 147 ± 0% 145 ± 0% -1.36% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 417 ± 0% 421 ± 0% +0.96% ``` Unfortunately go1.16 introduces more allocs overall (not that much more): ``` benchstat -delta-test=none ../_dev/thanos/2021/receive2/main-go1.15.txt ../_dev/thanos/2021/receive2/main-go1.16.3.txt name old time/op new time/op delta HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 1.45ms ± 0% 1.62ms ± 0% +11.87% HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 7.14ms ± 0% 6.47ms ± 0% -9.40% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 16.4ms ± 0% 15.8ms ± 0% -3.87% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 69.4ms ± 0% 66.4ms ± 0% -4.35% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 131ms ± 0% 141ms ± 0% +7.59% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 1.68s ± 0% 1.67s ± 0% -0.49% name old alloc/op new alloc/op delta HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 1.70MB ± 0% 1.75MB ± 0% +2.50% HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 4.84MB ± 0% 4.89MB ± 0% +0.88% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 18.2MB ± 0% 18.8MB ± 0% +3.07% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 49.6MB ± 0% 50.1MB ± 0% +1.09% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 331MB ± 0% 343MB ± 0% +3.63% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 804MB ± 0% 816MB ± 0% +1.50% name old allocs/op new allocs/op delta HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 15.6k ± 0% 15.6k ± 0% -0.01% HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 35.6k ± 0% 35.6k ± 0% +0.01% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 155k ± 0% 155k ± 0% +0.08% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 355k ± 0% 355k ± 0% +0.00% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 145 ± 0% 166 ± 0% +14.48% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 421 ± 0% 440 ± 0% +4.51% ``` Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
-
No difference. ``` benchstat -delta-test=none ../_dev/thanos/2021/receive2/main-go1.16.3.txt ../_dev/thanos/2021/receive2/impr-go1.16.3-promup.txt name old time/op new time/op delta HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 1.62ms ± 0% 1.77ms ± 0% +9.57% HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 6.47ms ± 0% 5.71ms ± 0% -11.76% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 15.8ms ± 0% 15.2ms ± 0% -3.83% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 66.4ms ± 0% 59.5ms ± 0% -10.37% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 141ms ± 0% 129ms ± 0% -8.60% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 1.67s ± 0% 1.41s ± 0% -15.58% name old alloc/op new alloc/op delta HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 1.75MB ± 0% 1.75MB ± 0% +0.04% HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 4.89MB ± 0% 4.89MB ± 0% +0.02% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 18.8MB ± 0% 18.8MB ± 0% -0.05% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 50.1MB ± 0% 50.1MB ± 0% +0.00% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 343MB ± 0% 344MB ± 0% +0.00% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 816MB ± 0% 816MB ± 0% -0.00% name old allocs/op new allocs/op delta HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 15.6k ± 0% 15.6k ± 0% +0.01% HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 35.6k ± 0% 35.6k ± 0% +0.00% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 155k ± 0% 155k ± 0% -0.06% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 355k ± 0% 355k ± 0% +0.00% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 166 ± 0% 169 ± 0% +1.81% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 440 ± 0% 435 ± 0% -1.14% ``` Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
-
Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
-
Moved hashring to optimized hash function.
``` benchstat -delta-test=none ../_dev/thanos/2021/receive2/impr1-go1.16.3.txt ../_dev/thanos/2021/receive2/impr3-go1.16.3.txt name old time/op new time/op delta HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 1.54ms ± 0% 1.64ms ± 0% +6.54% HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 6.96ms ± 0% 8.02ms ± 0% +15.23% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 16.1ms ± 0% 16.5ms ± 0% +2.77% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 65.4ms ± 0% 65.1ms ± 0% -0.49% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 168ms ± 0% 119ms ± 0% -29.49% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 1.69s ± 0% 1.37s ± 0% -19.05% name old alloc/op new alloc/op delta HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 1.75MB ± 0% 1.63MB ± 0% -6.43% HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 4.89MB ± 0% 4.77MB ± 0% -2.50% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 18.8MB ± 0% 17.6MB ± 0% -6.55% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 50.1MB ± 0% 48.9MB ± 0% -2.55% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 344MB ± 0% 225MB ± 0% -34.63% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 816MB ± 0% 697MB ± 0% -14.59% name old allocs/op new allocs/op delta HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 15.6k ± 0% 13.6k ± 0% -12.85% HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 35.6k ± 0% 33.6k ± 0% -5.64% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 155k ± 0% 135k ± 0% -12.95% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 355k ± 0% 335k ± 0% -5.64% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 170 ± 0% 101 ± 0% -40.59% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 439 ± 0% 372 ± 0% -15.26% ``` Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
-
Used Prometheus GetRef to avoid reallocating the same series.
``` benchstat -delta-test=none ../_dev/thanos/2021/receive2/impr3-go1.16.3.txt ../_dev/thanos/2021/receive2/impr4-go1.16.3.txt name old time/op new time/op delta HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 1.64ms ± 0% 1.15ms ± 0% -30.02% HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 8.02ms ± 0% 5.57ms ± 0% -30.53% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 16.5ms ± 0% 11.5ms ± 0% -30.28% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 65.1ms ± 0% 58.8ms ± 0% -9.66% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 119ms ± 0% 114ms ± 0% -3.56% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 1.37s ± 0% 1.43s ± 0% +4.58% name old alloc/op new alloc/op delta HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 1.63MB ± 0% 1.15MB ± 0% -29.48% HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 4.77MB ± 0% 4.29MB ± 0% -10.07% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 17.6MB ± 0% 12.8MB ± 0% -27.20% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 48.9MB ± 0% 44.1MB ± 0% -9.82% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 225MB ± 0% 120MB ± 0% -46.70% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 697MB ± 0% 592MB ± 0% -15.05% name old allocs/op new allocs/op delta HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/OK-12 13.6k ± 0% 3.6k ± 0% -73.58% HandlerReceiveHTTP/typical_labels_under_1KB,_500_of_them/conflict_errors-12 33.6k ± 0% 23.6k ± 0% -29.75% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/OK-12 135k ± 0% 35k ± 0% -73.84% HandlerReceiveHTTP/typical_labels_under_1KB,_5000_of_them/conflict_errors-12 335k ± 0% 235k ± 0% -29.84% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/OK-12 101 ± 0% 79 ± 0% -21.78% HandlerReceiveHTTP/extremely_large_label_value_10MB,_10_of_them/conflict_errors-12 372 ± 0% 360 ± 0% -3.23% ``` Signed-off-by: Bartlomiej Plotka <bwplotka@gmail.com>
Commits on Apr 20, 2021
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.