You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As of #112, turnkey is able to benchmark build directories directly via the turnkey cache benchmark BUILD_NAME command.
#112 implements the functionality as a completely new code path. This is a technical debt because now there are 2 ways to do benchmarking (turnkey benchmark being the other), they may produce slightly different results (e.g., turnkey cache benchmark does a better job reporting hardware failures), and any changes to benchmarking policy now need to be repeated in two places.
Ideally, a single code path would be used for benchmarking anything, regardless of whether it is a python script, ONNX file, or build directory. However, at the time of #112 there were blockers that need to be resolved before integration can take place:
Open design question: once we introduce turnkey cache benchmark would we ever deprecate it in favor of turnkey benchmark cache/*?
Once the mass-benchmarking logic is mainlined into the Files API it would be trivial to enable turnkey benchmark cache/*, its just that would be a breaking change pretty soon after introducing turnkey cache benchmark.
As of #112, turnkey is able to benchmark build directories directly via the
turnkey cache benchmark BUILD_NAME
command.#112 implements the functionality as a completely new code path. This is a technical debt because now there are 2 ways to do benchmarking (
turnkey benchmark
being the other), they may produce slightly different results (e.g.,turnkey cache benchmark
does a better job reporting hardware failures), and any changes to benchmarking policy now need to be repeated in two places.Ideally, a single code path would be used for benchmarking anything, regardless of whether it is a python script, ONNX file, or build directory. However, at the time of #112 there were blockers that need to be resolved before integration can take place:
--rebuild
to--skip
#114Proposed solution:
cc @danielholanda
The text was updated successfully, but these errors were encountered: