Remove the manual model conversion during benchmark #953
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
RunningLeon
reviewed
Jan 16, 2024
lvhan028
force-pushed
the
update-benchmark
branch
from
c32233e
to
568a9f3
Compare
RunningLeon
reviewed
Jan 18, 2024
Update benchmark to v0.2.0 cli style
lvhan028
commented
Feb 4, 2024
grimoire
reviewed
Feb 5, 2024
grimoire
reviewed
Feb 5, 2024
grimoire
reviewed
Feb 5, 2024
|
pytorch benchmark might leads to over sized memory usage. One solution is clearing caches after gathering. lmdeploy/lmdeploy/pytorch/engine/engine.py Line 600 in c332efa del tmp_out
torch.cuda.empty_cache()Another solution is using a small Line 170 in c332efa |
grimoire
approved these changes
Feb 19, 2024
grimoire
pushed a commit
to grimoire/lmdeploy
that referenced
this pull request
Feb 22, 2024
* tmp * update tuning gemm * modify profile_throughput * update profile_generation * fix profile generation * fix benchmark bash script * update * update * update * fix conflicts * fix error * pass cache count param * remove benchmark config * fix according to reviewer comments * update * update * update * update * update * fix profile_generation * update user guide * change title * fix profile_generation * fix gemm_tune according to reviewer comments * pass session_len and cache_max_entry_count to engines --------- Co-authored-by: RunningLeon <mnsheng@yeah.net>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
As tiltled
Modification
Tests
profile request throughput
profile api_server
profile static inference