feat: add test to view batch speedup amount

[drbh]

This PR adds a very simple test with the goal of giving more insight into speedups from batching vs a single input.

The test outputs the timing of a single batch op vs iterating over the individual items and helps give insight into rough expected timing for various batch sizes.

pytest -vvv server/tests/utils/test_qkv_batch_speedup.py

ouput

============================================ test session starts ============================================
platform linux -- Python 3.11.1, pytest-7.4.0, pluggy-1.5.0 -- /home/ubuntu/Projects/text-generation-inference/.venv/bin/python
cachedir: .pytest_cache
rootdir: /home/ubuntu/Projects/text-generation-inference/server
configfile: pyproject.toml
plugins: asyncio-0.23.7, anyio-4.3.0, syrupy-4.0.1
asyncio: mode=Mode.STRICT
collected 1 item

server/tests/utils/test_qkv_batch_speedup.py::test_qkv_batch_speedup Testing with sequence length: 3
Batch Size Batch Time (ms)      Sequential Time (ms)      Speedup
-----------------------------------------------------------------
0          0.17                 0.00                      0.00
1          47.00                0.58                      0.01
2          0.56                 0.99                      1.78
3          0.52                 1.49                      2.85
4          0.52                 1.96                      3.76
5          0.53                 2.47                      4.67
6          0.52                 3.03                      5.77
7          0.66                 3.44                      5.20
8          0.55                 3.98                      7.21
9          0.56                 4.39                      7.90
10         0.56                 4.92                      8.79
11         0.57                 5.55                      9.76
12         0.58                 5.94                      10.25
13         0.61                 6.40                      10.55
14         0.58                 7.05                      12.22
15         0.62                 7.56                      12.20
Testing with sequence length: 35
Batch Size Batch Time (ms)      Sequential Time (ms)      Speedup
-----------------------------------------------------------------
0          0.08                 0.00                      0.00
1          0.57                 0.54                      0.94
2          0.73                 1.08                      1.48
3          0.75                 1.63                      2.19
4          0.93                 2.15                      2.32
5          1.20                 2.80                      2.34
6          1.28                 3.35                      2.62
7          1.28                 3.91                      3.05
8          1.70                 4.42                      2.59
9          1.32                 4.90                      3.72
10         1.66                 5.46                      3.29
11         1.96                 6.05                      3.08
12         1.94                 6.65                      3.44
13         2.15                 7.15                      3.32
14         2.05                 7.68                      3.74
15         2.43                 8.17                      3.36
Testing with sequence length: 365
Batch Size Batch Time (ms)      Sequential Time (ms)      Speedup
-----------------------------------------------------------------
0          0.06                 0.00                      0.00
1          1.64                 1.92                      1.17
2          3.04                 4.30                      1.41
3          4.08                 6.38                      1.56
4          5.53                 8.23                      1.49
5          6.18                 10.16                     1.64
6          7.45                 12.09                     1.62
7          8.04                 13.94                     1.73
8          9.36                 16.24                     1.73
9          10.57                18.27                     1.73
10         11.84                20.00                     1.69
11         13.12                21.91                     1.67
12         13.75                23.77                     1.73
13         15.00                26.04                     1.74
14         15.56                28.10                     1.81
15         17.35                30.00                     1.73
Testing with sequence length: 3501
Batch Size Batch Time (ms)      Sequential Time (ms)      Speedup
-----------------------------------------------------------------
0          0.08                 0.00                      0.00
1          11.17                11.64                     1.04
2          21.86                23.75                     1.09
3          32.52                34.84                     1.07
4          43.39                46.50                     1.07
5          53.92                59.33                     1.10
6          64.55                70.75                     1.10
7          75.40                82.64                     1.10
8          85.56                94.23                     1.10
9          96.05                105.05                    1.09
10         105.86               117.54                    1.11
11         116.41               130.89                    1.12
12         126.60               140.40                    1.11
13         136.51               153.14                    1.12
14         147.00               164.14                    1.12
15         157.39               175.03                    1.11
Testing with sequence length: 11111
Batch Size Batch Time (ms)      Sequential Time (ms)      Speedup
-----------------------------------------------------------------
0          0.07                 0.00                      0.00
1          34.76                34.39                     0.99
2          67.31                70.45                     1.05
3          100.14               106.35                    1.06
4          133.37               142.14                    1.07
5          167.20               176.10                    1.05
6          199.45               211.42                    1.06
7          231.29               246.77                    1.07
8          264.17               281.65                    1.07
9          297.42               315.85                    1.06
10         329.85               354.24                    1.07
11         362.14               388.87                    1.07
12         394.93               422.40                    1.07
13         428.10               457.95                    1.07
14         461.78               496.16                    1.07
15         494.73               531.75                    1.07
PASSED                           [100%]

------------------------------------------ snapshot report summary ------------------------------------------