Add performance test for handle.py / BinaryValue

[aubindetrez]

These tests are measuring how long it takes to read signals, read one specific bit, write signals in cocotb.

This is part of the "Replace BinaryValue with LogicArray" effort #3244, these tests will help us better compare different implementations and their pros/cons.

Example of output:

100025000.00ns INFO     cocotb.dff8                        Little Endian, setting input signal: 3.1270687309988716s
200035000.00ns INFO     cocotb.dff8                        Big Endian, setting input signal: 3.1491839480004273s
300045000.00ns INFO     cocotb.dff8                        Little Endian, reading the output signal: 2.52074205900135
400055000.00ns INFO     cocotb.dff8                        Big Endian, reading the output signal: 2.66596318899974
500065000.00ns INFO     cocotb.dff8                        Little Endian, reading part of the output signal: 2.780234088000725
600075000.00ns INFO     cocotb.dff8                        Big Endian Endian, reading part of the output sign: 2.776298863000193
700085000.00ns INFO     cocotb.dff8                        Little Endian, reading single bit from the output signal: 2.6840617939997173
800095000.00ns INFO     cocotb.dff8                        Big Endian, reading single bit from the output signal: 2.7572237929998664
800115000.00ns INFO     cocotb.dff8                        Little Endian, setting part of the input signal: Unsupported
800125000.00ns INFO     cocotb.dff8                        Big Endian, setting part of the input signal: Unsupported
900125000.00ns INFO     cocotb.dff8                        Little Endian, setting single bit of the input signal: 3.229350190000332
1000135000.00ns INFO     cocotb.dff8                       Big Endian, setting single bit of the input signal: 3.3048068640000565
1100155000.00ns INFO     cocotb.dff8                       Little Endian, read + write signals: 3.469675129999814
1200175000.00ns INFO     cocotb.dff8                       Big Endian, read + write signals: 3.5726902460010024

[ktbarrett]

It's probably easier to gather performance metrics using pytest-benchmark and the test_binary_value.py in the pytest directory.

[aubindetrez]

The benchmark introduced by this PR is meant to be implementation independent.
test_binary_value.py only tests BinaryValue, however I would like an end-to-end test that does not depend on any specific implementation.
I would use it to check how different implementations of LogicArray perform as a replacement for BinaryValue.

What alternative do we have (that would ideally not require a simulator to run)?
I considered a pytest for handle.py, but how much of handle.py require a simulator to run anyway?

[themperek]

Example benchmark code is here: https://github.com/cocotb/cocotb/blob/master/tests/benchmark.py
It is running in CI: https://github.com/cocotb/cocotb/actions/workflows/benchmark.yml

It needs fixing (authentication settings) to actually export and make plots.

[aubindetrez]

Thanks, I have already been using this matrix multiplier benchmark for my BinaryValue vs LogicArray comparison.

But I would also need to benchmark other aspects of the BinaryValue/LogicArray API, like what if a signal contains X or Z, or if we only access a few bits?
Dedicated benchmarks would be very helpful to compare different implementations and optimization as we are doing refactoring...

[themperek]

But I would also need to benchmark other aspects of the BinaryValue/LogicArray API, like what if a signal contains X or Z, or if we only access a few bits?

Feel free to add.

I will try to figure out how to get plotting/authentication working.

[themperek]

@aubindetrez https://cocotb.github.io/cocotb-benchmark-results/dev/bench/