Directly benchmark build directories

[jeremyfowers]

Closes #19 by adding a new command, turnkey cache benchmark.

Note that this command is temporary, and will be replaced by turnkey benchmark BUILD_DIRS soon (see #115).

(tklocal) PS C:\work> turnkey cache benchmark -h
usage: turnkey cache benchmark [-h] [-d CACHE_DIR] [--all] [--skip {attempted,failed,successful,none}]
                               [--timeout TIMEOUT] [--runtime {ort,trt,torch-eager,torch-compiled,vitisep}]    
                               [--iterations ITERATIONS] [--rt-args [RT_ARGS [RT_ARGS ...]]]
                               [build_names]

positional arguments:
  build_names           Name of the specific build(s) to be benchmarked, within the cache directory

optional arguments:
  -h, --help            show this help message and exit
  -d CACHE_DIR, --cache-dir CACHE_DIR
                        Search path for builds (defaults to C:\Users\jefowers/.cache/turnkey)
  --all                 Benchmark all builds in the cache directory
  --skip {attempted,failed,successful,none}
                        Sets the policy for skipping benchmark attempts (defaults to attempted).`attempted`    
                        means to skip any previously-attempted benchmark, whether it succeeded or
                        failed.`failed` skips benchmarks that have already failed once.`successful` skips      
                        benchmarks that have already succeeded.`none` will attempt all benchmarks, regardless  
                        of whether they were previously attempted.
  --timeout TIMEOUT     Benchmark timeout, in seconds, after which each benchmark will be canceled
                        (default: no timeout).
  --runtime {ort,trt,torch-eager,torch-compiled,vitisep}
                        Software runtime that will be used to collect the benchmark. Must be compatible with   
                        the device chosen for the build. If this argument is not set, the default runtime of   
                        the selected device will be used.
  --iterations ITERATIONS
                        Number of execution iterations of the model to capture the benchmarking performance    
                        (e.g., mean latency)
  --rt-args [RT_ARGS [RT_ARGS ...]]
                        Optional arguments provided to the runtime being used

Design Considerations

• There is a fair amount of overlap in logic between the benchmarking capability in turnkeyml.analyze.script.explore_invocation() and the new benchmark_build() function.
  • If left as-is, we will need to keep these functions in sync, which may be annoying and error-prone
  • We could refactor so that there is only one code path for benchmarking. However, that would involve substantially moving functionality around and it will delay this PR and other work.
• turnkey cache benchmark always runs its benchmarks in process isolation mode, with no option to turn that off. I did it this way since the purpose of the function is mass-benchmarking, which should always happen in isolated processes.

Demo

Basic Operation

On a cache with 3 efficientnet* builds:

(tklocal) PS C:\work\turnkeyml\models\timm> turnkey cache benchmark --all --skip none
  0%|                                                                                   | 0/3 [00:00<?, ?it/s]
Info: Attempting to benchmark: efficientnet_b0_timm_94d604de

Info: Performance of build efficientnet_b0_timm_94d604de on AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics (ort v1.15.1) is:
        Mean Latency: 6.168 milliseconds (ms)
        Throughput: 162.1 inferences per second (IPS)


Woohoo! Done benchmarking: efficientnet_b0_timm_94d604de
 33%|█████████████████████████                                                  | 1/3 [00:23<00:47, 23.85s/it]
Info: Attempting to benchmark: efficientnet_b1_timm_47686f1f

Info: Performance of build efficientnet_b1_timm_47686f1f on AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics (ort v1.15.1) is:
        Mean Latency: 10.051 milliseconds (ms)
        Throughput: 99.5 inferences per second (IPS)


Woohoo! Done benchmarking: efficientnet_b1_timm_47686f1f
 67%|██████████████████████████████████████████████████                         | 2/3 [00:48<00:24, 24.53s/it]
Info: Attempting to benchmark: efficientnet_b2_timm_72dbc375

Info: Performance of build efficientnet_b2_timm_72dbc375 on AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics (ort v1.15.1) is:
        Mean Latency: 15.435 milliseconds (ms)
        Throughput: 64.8 inferences per second (IPS)


Woohoo! Done benchmarking: efficientnet_b2_timm_72dbc375
100%|███████████████████████████████████████████████████████████████████████████| 3/3 [01:14<00:00, 24.82s/it]

HardwareError Raised

If benchmarking a build results in a HardwareError (e.g., computer needs a restart), that build will be marked as benchmark_status: error and error_log will get content like this:

evaluations:
  3b643be8:
    benchmark_status: error
    build_name: efficientnet_b0_timm_94d604de
    build_status: successful
    device: AMD Ryzen 7 PRO 7840U w/ Radeon 780M Graphics
    device_type: x86
    error_log: "Traceback (most recent call last):\n  File \"C:\\work\\turnkeyml\\\
      src\\turnkeyml\\run\\benchmark_build.py\", line 29, in run\n    multiprocessing.Process.run(self)\n\
      \  File \"C:\\Users\\jefowers\\AppData\\Local\\miniconda3\\envs\\tklocal\\lib\\\
      multiprocessing\\process.py\", line 108, in run\n    self._target(*self._args,\
      \ **self._kwargs)\n  File \"C:\\work\\turnkeyml\\src\\turnkeyml\\run\\benchmark_build.py\"\
      , line 72, in benchmark_build\n    raise exp.HardwareError(\nturnkeyml.common.exceptions.HardwareError:\
      \ You have the correct driver installed (12121). However, Microsoft PnP Utility\
      \ shows that your device has a problem. Please check Device Manager for\
      \ more information about this issue.\n\n\n"

[danielholanda]

Please make sure that you show a reasonable error message when the user provides an invalid build name. :)

[danielholanda]

Approved as experimental feature.
Agree that this should be deprecated and replaced by the approach suggested in #115