Updated llm demo to report number of generated tokens in the last response #2373
Conversation
demos/python_demos/llm_text_generation/servable_stream/model.py
Outdated
Show resolved
Hide resolved
| def generate(): | ||
| ov_model_exec.generate(**tokens, **generate_kwargs) | ||
| result = ov_model_exec.generate(**tokens, **generate_kwargs) |
There was a problem hiding this comment.
What's the reason we do not include special tokens? What are those special tokens?
There was a problem hiding this comment.
In this case special tokens are used for left/right padding. They are stripped to assure accurate token counting when batch_size > 1.
|
include output changes in readme |
| @@ -181,6 +181,7 @@ def generate(): | |||
| for partial_result in streamer: | |||
| yield serialize_completions(batch_size, partial_result) | |||
| t1.join() | |||
| token_count[0] -= len(tokens["input_ids"].flatten()) | |||
| yield [Tensor("token_count", np.array([token_count[0]], dtype=np.int32))] | |||
There was a problem hiding this comment.
why only 1st element of token_count? Was it tested with bs>1 with variable response size?
There was a problem hiding this comment.
First element of token_count represents final number of tokens. This snipped uses list instead of single variable due to the mutable property of python's list type.
There was a problem hiding this comment.
- why are you re-creating numpy array here? token_count is an array anyway
There was a problem hiding this comment.
I currently do not understand the value in having token reporting fragmented per batch, it would be a simple change
|
|
||
| return serialize_completions(batch_size, completions) | ||
| t1.join() | ||
| return serialize_completions(batch_size, completions, token_count[0]) |
There was a problem hiding this comment.
I'm not sure why are working on a numpy array, just to return a number in line 191. Then, in serialize_completions you create numpy array anyway Tensor("token_count", np.array([token_count], dtype=np.int32)) @bstrzele
bstrzele
force-pushed
the
llm_token_reporting
branch
2 times, most recently
from
22b0135
to
a899698
Compare
bstrzele
force-pushed
the
llm_token_reporting
branch
from
3c84f44
to
94f2cc8
Compare
CVS-135176