Prints benchmark results in a neat table and attempts to run every benchmark

[akhildevelops]

Pull Request Template

Checklist

• Confirmed that run-checks all script has been executed.
• Made sure the book is up to date with changes in this PR.

Related Issues/PRs

Fixes #1440

Changes

Benchmark results will be displayed as below based on benchmarks X backends

Benchmark process will continue to run even if there's a failure in any one of the combination of benchmarks and backends

Testing

There's a test case included and ran this feature across all the available benchmarks and backends.

[codecov]

Codecov Report

Attention: Patch coverage is 61.53846% with 80 lines in your changes are missing coverage. Please review.

Project coverage is 85.75%. Comparing base (53eb3ec) to head (8f2fbd9).
Report is 1 commits behind head on main.

Files	Patch %	Lines
backend-comparison/src/burnbenchapp/base.rs	0.00%	47 Missing ⚠️
backend-comparison/src/persistence/base.rs	79.50%	33 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1464      +/-   ##
==========================================
- Coverage   85.79%   85.75%   -0.04%     
==========================================
  Files         646      647       +1     
  Lines       72051    72300     +249     
==========================================
+ Hits        61813    61998     +185     
- Misses      10238    10302      +64     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[akhildevelops]

Is the code ok to get merged ?

[antimora]

I haven't reviewed closely - leaving it up to @syl20bnr but I had small suggestion. It'd be convenient if we make the table Markdown compatible so we can copy the table output into any MD renderer (Discord, Github, other docs).

[akhildevelops]

Intersting idea, that can be done. 🤝

[akhildevelops]

❯ cargo run --release --bin burnbench -- run --benches unary --backends wgpu candle-cuda tch-cpu        
    Finished release [optimized] target(s) in 0.22s
     Running `target/release/burnbench run --benches unary --backends wgpu candle-cuda tch-cpu`
Executing the following benchmark and backend combinations (Total: 3):
Running benchmarks...
==========Benchmarking unary on wgpu==========
    Finished bench [optimized] target(s) in 0.14s
     Running benches/unary.rs (target/benchmarks/release/deps/unary-14750bb533c606ad)

        Timestamp: 1710384072278
        Git Hash: 24db2c25023771b5d4a856734457f232dcafcc91
        Benchmarking - unary
―――――――― Result ―――――――――
  Samples     10
  Mean        1.660ms
  Variance    81.000ns
  Median      1.545ms
  Min         1.380ms
  Max         2.271ms
―――――――――――――――――――――――――
        
==========Benchmarking unary on candle-cuda==========
    Finished bench [optimized] target(s) in 0.15s
     Running benches/unary.rs (target/benchmarks/release/deps/unary-216654a32a33583a)

        Timestamp: 1710384072664
        Git Hash: 24db2c25023771b5d4a856734457f232dcafcc91
        Benchmarking - unary
―――――――― Result ―――――――――
  Samples     10
  Mean        8.667ms
  Variance    23.000ns
  Median      8.611ms
  Min         8.488ms
  Max         8.981ms
―――――――――――――――――――――――――
        
==========Benchmarking unary on tch-cpu==========
    Finished bench [optimized] target(s) in 0.15s
     Running benches/unary.rs (target/benchmarks/release/deps/unary-26a7ed056fa878c2)

        Timestamp: 1710384073296
        Git Hash: 24db2c25023771b5d4a856734457f232dcafcc91
        Benchmarking - unary
―――――――― Result ―――――――――
  Samples     10
  Mean        15.416ms
  Variance    34.880µs
  Median      13.681ms
  Min         8.867ms
  Max         24.696ms
―――――――――――――――――――――――――
        
==========Benchmark Results==========

| Benchmark      | Backend                            | Runtime        |
|----------------|------------------------------------|----------------|
| unary          | jit<wgpu>-BestAvailable            | 1.660ms        |
| unary          | candle-Cuda(0)                     | 8.666ms        |
| unary          | tch-Cpu                            | 15.415ms       |

[akhildevelops]

Below is the rendering in this github's comment box

==========Benchmark Results==========

Benchmark	Backend	Runtime
unary	jit-BestAvailable	1.660ms
unary	candle-Cuda(0)	8.666ms
unary	tch-Cpu	15.415ms

[syl20bnr]

Awesome ! Nothing to say on the actual implementation. I have just minor change requests on the form. For the intermediary file we should create it in the cache directory explicitly.

[akhildevelops]

Hi @syl20bnr, I've made changes according to the suggestions you've provided.

[syl20bnr]

Thank you !

[syl20bnr]

@akhildevelops I forget to tell you in case you did not know it that you should not use the main branch of your fork when you submit a pull request and instead you should create a dedicate branch for each PR. This way you can more easily work on multiple PRs at once and keep the main branch up to date with the upstream repository.

[akhildevelops]

Sure Thanks 🤝