Skip to content

Commit

Permalink
Merge pull request #21 from sgwhat/warm-up-refactor
Browse files Browse the repository at this point in the history
Remove redundant warm-up for to optimize chat speed
  • Loading branch information
sgwhat committed Mar 22, 2024
2 parents f9cfbdc + b7ad3d9 commit e21b499
Showing 1 changed file with 0 additions and 5 deletions.
5 changes: 0 additions & 5 deletions modules/text_generation.py
Original file line number Diff line number Diff line change
Expand Up @@ -403,11 +403,6 @@ def generate_with_callback(callback=None, *args, **kwargs):
def generate_with_streaming(**kwargs):
return Iteratorize(generate_with_callback, [], kwargs, callback=None)

# warm-up
with torch.no_grad():
shared.model.generate(**generate_params)
torch.xpu.synchronize()

with generate_with_streaming(**generate_params) as generator:
cumulative_reply = ''
starting_from = 0 if shared.is_seq2seq else len(input_ids[0])
Expand Down

0 comments on commit e21b499

Please sign in to comment.