[Misc] Include matched stop string/token in responses #2976
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
njhill
force-pushed
the
return-stop-str
branch
2 times, most recently
from
0700062
to
b9c7685
Compare
njhill
changed the title
njhill
force-pushed
the
return-stop-str
branch
2 times, most recently
from
84c73da
to
bb6f831
Compare
|
@simon-mo would be good to get your thoughts on this simple addition. It would be useful for us even if it's not exposed in the openai API responses. Also not sure what the best field name is, perhaps |
njhill
force-pushed
the
return-stop-str
branch
from
c0ad972
to
ab8ba66
Compare
|
I think this is very simple and useful! Naming it as |
|
Thanks @simon-mo!
Will defer to your judgment on this! I like the simplicity of a single field but agree that more explicit could be better. |
joerunde
pushed a commit
to IBM/vllm
that referenced
this pull request
Mar 11, 2024
Include matched stop string/token in responses [Cherry-picked from open upstream PR vllm-project/vllm#2976] Currently a finish_reason of "stop" is returned if any of the following are encountered: - One of the provided stop strings - One of the provided stop tokens - The EOS token It can be useful to know specifically which of these caused the sequence generation to stop, especially since by default the stop strings/tokens are omitted from the output text (and output token_ids?). This PR adds a "stop_reason" field to the CompletionOutput class which will contain the matched stop string or integer token id. It will be None otherwise, including the EOS token case. This means in particular that EOS can be inferred by (finish_reason=="stop" and stop_reason=None). I've also added to the openai server responses but not sure whether or not this should be included since it isn't part of the official API. Signed-off-by: Joe Runde <joseph.runde@ibm.com>
joerunde
pushed a commit
to IBM/vllm
that referenced
this pull request
Mar 11, 2024
Include matched stop string/token in responses [Cherry-picked from open upstream PR vllm-project/vllm#2976] Currently a finish_reason of "stop" is returned if any of the following are encountered: - One of the provided stop strings - One of the provided stop tokens - The EOS token It can be useful to know specifically which of these caused the sequence generation to stop, especially since by default the stop strings/tokens are omitted from the output text (and output token_ids?). This PR adds a "stop_reason" field to the CompletionOutput class which will contain the matched stop string or integer token id. It will be None otherwise, including the EOS token case. This means in particular that EOS can be inferred by (finish_reason=="stop" and stop_reason=None). I've also added to the openai server responses but not sure whether or not this should be included since it isn't part of the official API. Signed-off-by: Joe Runde <joseph.runde@ibm.com> Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
joerunde
pushed a commit
to IBM/vllm
that referenced
this pull request
Mar 11, 2024
joerunde
pushed a commit
to IBM/vllm
that referenced
this pull request
Mar 11, 2024
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
joerunde
pushed a commit
to IBM/vllm
that referenced
this pull request
Mar 12, 2024
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
njhill
force-pushed
the
return-stop-str
branch
2 times, most recently
from
2e129bd
to
560f090
Compare
njhill
changed the title
Currently a finish_reason of "stop" is returned if any of the following are encountered: - One of the provided stop strings - One of the provided stop tokens - The EOS token It can be useful to know specifically which of these caused the sequence generation to stop, especially since by default the stop strings/tokens are omitted from the output text (and output token_ids?). This PR adds a "stop_reason" field to the CompletionOutput class which will contain the matched stop string or integer token id. It will be None otherwise, including the EOS token case. This means in particular that EOS can be inferred by (finish_reason=="stop" and stop_reason=None). I've also added to the openai server responses but not sure whether or not this should be included since it isn't part of the official API.
|
What you have is totally fine as long as it's well documented. |
|
Thanks @simon-mo, I just pushed a small update to add descriptions to the new |
simon-mo
approved these changes
Mar 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently a
finish_reasonof "stop" is returned if any of the following are encountered:It can be useful to know specifically which of these caused the sequence generation to stop, especially since by default the stop strings/tokens are omitted from the output text (and output token_ids?).
This PR adds a
stop_reasonfield to theCompletionOutputclass which will contain the matched stop string or integer token id. It will be None otherwise, including the EOS token case. This means in particular that EOS can be inferred ifffinish_reason=="stop" and stop_reason=None.I've also added to the openai server responses but not sure whether or not this should be included since it isn't part of the official API.
Thanks @sahilsuneja1 for adding a test