Align InferenceEngine to store ms in _model_times #3501
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When using cuda_events, the measured model time is stored in ms.
When not using cuda_events, the measured model time was stored in seconds.
This commit fixes the units and aligns them to store ms, the same as elapsed() function.
This was observed when running the following pytest:
unit/inference/test_model_profiling.py::TestModelProfiling::test[False-True-roberta-base-fill-mask]
Returned values were:
count=0 e2e_t=895.174312 model_t=0.8529715538024902
count=1 e2e_t=7.500252 model_t=0.0041310787200927734
count=2 e2e_t=3.887346 model_t=0.0018568038940429688
count=3 e2e_t=3.577845 model_t=0.0016334056854248047
count=4 e2e_t=3.43976 model_t=0.0016703605651855469
count=5 e2e_t=3.310903 model_t=0.0016107559204101562
count=6 e2e_t=3.299556 model_t=0.001603841781616211
count=7 e2e_t=3.605722 model_t=0.0015969276428222656
count=8 e2e_t=3.273741 model_t=0.0015516281127929688
count=9 e2e_t=3.46306 model_t=0.0016617774963378906
The units difference is observed here, when model_t is in ther order of 10e-3 comparing to e2e_t
HolyFalafel
requested review from
RezaYazdaniAminabadi,
jeffra,
mrwyattii,
awan-10,
cmikeh2 and
arashb
as code owners
molly-smith
pushed a commit
that referenced
this pull request
Jun 23, 2023
* Align InferenceEngine to store ms in _model_times
When using cuda_events, the measured model time is stored in ms.
When not using cuda_events, the measured model time was stored in seconds.
This commit fixes the units and aligns them to store ms, the same as elapsed() function.
This was observed when running the following pytest:
unit/inference/test_model_profiling.py::TestModelProfiling::test[False-True-roberta-base-fill-mask]
Returned values were:
count=0 e2e_t=895.174312 model_t=0.8529715538024902
count=1 e2e_t=7.500252 model_t=0.0041310787200927734
count=2 e2e_t=3.887346 model_t=0.0018568038940429688
count=3 e2e_t=3.577845 model_t=0.0016334056854248047
count=4 e2e_t=3.43976 model_t=0.0016703605651855469
count=5 e2e_t=3.310903 model_t=0.0016107559204101562
count=6 e2e_t=3.299556 model_t=0.001603841781616211
count=7 e2e_t=3.605722 model_t=0.0015969276428222656
count=8 e2e_t=3.273741 model_t=0.0015516281127929688
count=9 e2e_t=3.46306 model_t=0.0016617774963378906
The units difference is observed here, when model_t is in ther order of 10e-3 comparing to e2e_t
* Update engine.py
---------
Co-authored-by: Michael Wyatt <michaelwyatt@microsoft.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When using cuda_events, the measured model time is stored in ms.
When not using cuda_events, the measured model time was stored in seconds.
This commit fixes the units and aligns them to store ms, the same as elapsed() function.
This was observed when running the following pytest:
unit/inference/test_model_profiling.py::TestModelProfiling::test[False-True-roberta-base-fill-mask]
Returned values were:
count=0 e2e_t=895.174312 model_t=0.8529715538024902
count=1 e2e_t=7.500252 model_t=0.0041310787200927734
count=2 e2e_t=3.887346 model_t=0.0018568038940429688
count=3 e2e_t=3.577845 model_t=0.0016334056854248047
count=4 e2e_t=3.43976 model_t=0.0016703605651855469
count=5 e2e_t=3.310903 model_t=0.0016107559204101562
count=6 e2e_t=3.299556 model_t=0.001603841781616211
count=7 e2e_t=3.605722 model_t=0.0015969276428222656
count=8 e2e_t=3.273741 model_t=0.0015516281127929688
count=9 e2e_t=3.46306 model_t=0.0016617774963378906
The units difference is observed here, when model_t is in ther order of 10e-3 comparing to e2e_t