Add TEA Benchmarks #2438

PROFeNoM · 2023-12-28T08:17:56Z

Description

(TEA) Benchmarks request startup & spindown using Google benchmarks.

These benchmarks can be run using make benchmarks_tea from the root of the repository. Sample results are provided below.

This PR adds GBench as a submodule rather than installing it globally in the buster images. The choice of GBench over Catch2 was made for the following reasons:

GBench is already supported by the benchmark analyzer, while Catch2's isn't (XML Output)
GBench is objectively more powerful than Catch2 when it comes to microbenchmarks, it being its core purpose
Using GBench, benchmarking memory usage is theoretically possible, while it isn't with Catch2
CPU Time

On the benchmarking-platform side, I'll cherry-pick these two commits: commit1, commit2. It adds missing requirements to build tea and analyze results.

Sample JSON Results

{
  "context": {
    "date": "2024-01-11T08:38:04+00:00",
    "host_name": "19feafdf3b17",
    "executable": "/home/circleci/app/tmp/build_tea/benchmarks/tea_benchmarks",
    "num_cpus": 10,
    "mhz_per_cpu": 48,
    "cpu_scaling_enabled": false,
    "caches": [
    ],
    "load_avg": [0.73,0.55,0.52],
    "library_build_type": "debug"
  },
  "benchmarks": [
    {
      "name": "BM_TeaSapiSpinup",
      "family_index": 0,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpinup",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 0,
      "threads": 1,
      "iterations": 204,
      "real_time": 3.5789468489454914e+00,
      "cpu_time": 3.5788549558823552e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpinup",
      "family_index": 0,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpinup",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 1,
      "threads": 1,
      "iterations": 204,
      "real_time": 4.0908328988806195e+00,
      "cpu_time": 4.0913859656862597e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpinup",
      "family_index": 0,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpinup",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 2,
      "threads": 1,
      "iterations": 204,
      "real_time": 4.8347900485489790e+00,
      "cpu_time": 4.8365735196078363e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpinup",
      "family_index": 0,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpinup",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 3,
      "threads": 1,
      "iterations": 204,
      "real_time": 4.0736395252786357e+00,
      "cpu_time": 4.0736938333333512e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpinup",
      "family_index": 0,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpinup",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 4,
      "threads": 1,
      "iterations": 204,
      "real_time": 4.1406070395525747e+00,
      "cpu_time": 4.1399689166666525e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpinup",
      "family_index": 0,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpinup",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 5,
      "threads": 1,
      "iterations": 204,
      "real_time": 4.3746989885774239e+00,
      "cpu_time": 4.3738346960783856e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpinup",
      "family_index": 0,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpinup",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 6,
      "threads": 1,
      "iterations": 204,
      "real_time": 4.1305049594751946e+00,
      "cpu_time": 4.1304384950980024e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpinup",
      "family_index": 0,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpinup",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 7,
      "threads": 1,
      "iterations": 204,
      "real_time": 3.7166019159770918e+00,
      "cpu_time": 3.7058204803921715e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpinup",
      "family_index": 0,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpinup",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 8,
      "threads": 1,
      "iterations": 204,
      "real_time": 3.5813267690280233e+00,
      "cpu_time": 3.5811268039215545e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpinup",
      "family_index": 0,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpinup",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 9,
      "threads": 1,
      "iterations": 204,
      "real_time": 3.6689225872738889e+00,
      "cpu_time": 3.6688674264705479e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpinup_mean",
      "family_index": 0,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpinup",
      "run_type": "aggregate",
      "repetitions": 10,
      "threads": 1,
      "aggregate_name": "mean",
      "aggregate_unit": "time",
      "iterations": 10,
      "real_time": 4.0190871581537930e+00,
      "cpu_time": 4.0180565093137117e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpinup_median",
      "family_index": 0,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpinup",
      "run_type": "aggregate",
      "repetitions": 10,
      "threads": 1,
      "aggregate_name": "median",
      "aggregate_unit": "time",
      "iterations": 10,
      "real_time": 4.0822362120796276e+00,
      "cpu_time": 4.0825398995098050e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpinup_stddev",
      "family_index": 0,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpinup",
      "run_type": "aggregate",
      "repetitions": 10,
      "threads": 1,
      "aggregate_name": "stddev",
      "aggregate_unit": "time",
      "iterations": 10,
      "real_time": 3.9813532009242131e-01,
      "cpu_time": 3.9940834109605772e-01,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpinup_cv",
      "family_index": 0,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpinup",
      "run_type": "aggregate",
      "repetitions": 10,
      "threads": 1,
      "aggregate_name": "cv",
      "aggregate_unit": "percentage",
      "iterations": 10,
      "real_time": 9.9061131153798790e-02,
      "cpu_time": 9.9403365823811443e-02,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpindown",
      "family_index": 1,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpindown",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 0,
      "threads": 1,
      "iterations": 798,
      "real_time": 1.2446454198643528e+00,
      "cpu_time": 1.2446149536340700e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpindown",
      "family_index": 1,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpindown",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 1,
      "threads": 1,
      "iterations": 798,
      "real_time": 1.2334602004393262e+00,
      "cpu_time": 1.2332810438597051e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpindown",
      "family_index": 1,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpindown",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 2,
      "threads": 1,
      "iterations": 798,
      "real_time": 1.3291927340529732e+00,
      "cpu_time": 1.3292008182958002e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpindown",
      "family_index": 1,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpindown",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 3,
      "threads": 1,
      "iterations": 798,
      "real_time": 1.3511481225568764e+00,
      "cpu_time": 1.3510073558897415e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpindown",
      "family_index": 1,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpindown",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 4,
      "threads": 1,
      "iterations": 798,
      "real_time": 1.0602376491247436e+00,
      "cpu_time": 1.0602266666665683e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpindown",
      "family_index": 1,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpindown",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 5,
      "threads": 1,
      "iterations": 798,
      "real_time": 1.2656754436176201e+00,
      "cpu_time": 1.2654911516291578e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpindown",
      "family_index": 1,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpindown",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 6,
      "threads": 1,
      "iterations": 798,
      "real_time": 1.2791834575127585e+00,
      "cpu_time": 1.2779725827067279e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpindown",
      "family_index": 1,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpindown",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 7,
      "threads": 1,
      "iterations": 798,
      "real_time": 1.2985168717918862e+00,
      "cpu_time": 1.3052286240602167e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpindown",
      "family_index": 1,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpindown",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 8,
      "threads": 1,
      "iterations": 798,
      "real_time": 1.2179460049502406e+00,
      "cpu_time": 1.2179112130325631e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpindown",
      "family_index": 1,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpindown",
      "run_type": "iteration",
      "repetitions": 10,
      "repetition_index": 9,
      "threads": 1,
      "iterations": 798,
      "real_time": 1.1755986322116638e+00,
      "cpu_time": 1.1755502243107603e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpindown_mean",
      "family_index": 1,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpindown",
      "run_type": "aggregate",
      "repetitions": 10,
      "threads": 1,
      "aggregate_name": "mean",
      "aggregate_unit": "time",
      "iterations": 10,
      "real_time": 1.2455604536122440e+00,
      "cpu_time": 1.2460484634085311e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpindown_median",
      "family_index": 1,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpindown",
      "run_type": "aggregate",
      "repetitions": 10,
      "threads": 1,
      "aggregate_name": "median",
      "aggregate_unit": "time",
      "iterations": 10,
      "real_time": 1.2551604317409863e+00,
      "cpu_time": 1.2550530526316139e+00,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpindown_stddev",
      "family_index": 1,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpindown",
      "run_type": "aggregate",
      "repetitions": 10,
      "threads": 1,
      "aggregate_name": "stddev",
      "aggregate_unit": "time",
      "iterations": 10,
      "real_time": 8.3421037255346581e-02,
      "cpu_time": 8.3856127344118014e-02,
      "time_unit": "ms"
    },
    {
      "name": "BM_TeaSapiSpindown_cv",
      "family_index": 1,
      "per_family_instance_index": 0,
      "run_name": "BM_TeaSapiSpindown",
      "run_type": "aggregate",
      "repetitions": 10,
      "threads": 1,
      "aggregate_name": "cv",
      "aggregate_unit": "percentage",
      "iterations": 10,
      "real_time": 6.6974699632938423e-02,
      "cpu_time": 6.7297645161193728e-02,
      "time_unit": "ms"
    }
  ]
}

Reviewer checklist

Test coverage seems ok.
Appropriate labels assigned.

pr-commenter · 2024-01-10T14:19:27Z

Benchmarks

Benchmark execution time: 2024-01-10 14:18:30

Comparing candidate commit e7ecc45 in PR branch alex/bench/gbenchmark with baseline commit 4568c41 in branch master.

Found 1 performance improvements and 0 performance regressions! Performance is the same for 38 metrics, 51 unstable metrics.

scenario:ContextPropagationBench/benchExtractHeaders128Bit

🟩 execution_time [-1317.948ns; -682.052ns] or [-17.573%; -9.094%]

bwoebi

Thanks, looks good to me, let's merge this :-)

PROFeNoM added the cat:benchmarks label Dec 28, 2023

PROFeNoM self-assigned this Dec 28, 2023

PROFeNoM mentioned this pull request Dec 28, 2023

Add basic PHP MicroBenchmarks #2435

Merged

7 tasks

PROFeNoM added the ☠️ do-not-merge/WIP label Dec 28, 2023

PROFeNoM force-pushed the alex/bench/gbenchmark branch from f68c3c5 to e7ecc45 Compare January 10, 2024 13:43

PROFeNoM marked this pull request as ready for review January 11, 2024 09:58

PROFeNoM requested a review from a team as a code owner January 11, 2024 09:58

PROFeNoM force-pushed the alex/bench/gbenchmark branch from b4c6c92 to ef3088f Compare January 12, 2024 08:55

PROFeNoM removed the ☠️ do-not-merge/WIP label Jan 16, 2024

PROFeNoM force-pushed the alex/bench/gbenchmark branch 2 times, most recently from 239d3fc to a7aaa2d Compare January 17, 2024 07:43

bwoebi approved these changes Feb 7, 2024

View reviewed changes

PROFeNoM added 8 commits February 16, 2024 10:47

Add GBench skeleton

6d9c57a

Clean dead code & Fix Compilation issues

fbf1979

Add Google Benchmark as a submodule

4a1eee1

Use Google Benchmark as a submodule

9e5c41c

Remove GBench global buster installation

59fdff9

Change results output dir

5373f30

Run benchmarks_tea during runall.sh

babba8b

Trigger pipeline run on tea changes

b590053

PROFeNoM force-pushed the alex/bench/gbenchmark branch from a7aaa2d to b590053 Compare February 16, 2024 09:48

PROFeNoM merged commit 346de4d into master Feb 16, 2024
367 of 387 checks passed

PROFeNoM deleted the alex/bench/gbenchmark branch February 16, 2024 10:19

github-actions bot added this to the 0.98.0 milestone Feb 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add TEA Benchmarks #2438

Add TEA Benchmarks #2438

PROFeNoM commented Dec 28, 2023 •

edited

pr-commenter bot commented Jan 10, 2024

bwoebi left a comment

Add TEA Benchmarks #2438

Add TEA Benchmarks #2438

Conversation

PROFeNoM commented Dec 28, 2023 • edited

Description

Reviewer checklist

pr-commenter bot commented Jan 10, 2024

Benchmarks

scenario:ContextPropagationBench/benchExtractHeaders128Bit

bwoebi left a comment

Choose a reason for hiding this comment

PROFeNoM commented Dec 28, 2023 •

edited