Logprobs suggestions #948
Merged
Logprobs suggestions #948
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
enochcheung
commented
Dec 21, 2023
| @@ -8,11 +8,12 @@ | |||
| "\n", | |||
| "This notebook demonstrates the use of the `logprobs` parameter in the Chat Completions API. When `logprobs` is enabled, the API returns the log probabilities of each output token, along with a limited number of the most likely tokens at each token position and their log probabilities. The relevant request parameters are:\n", | |||
| "* `logprobs`: Whether to return log probabilities of the output tokens or not. If true, returns the log probabilities of each output token returned in the content of message. This option is currently not available on the `gpt-4-vision-preview` model.\n", | |||
| "* `top_logprobs`: An integer between 0 and 5 specifying the number of most likely tokens to return at each token position, each with an associated log probability. logprobs must be set to true if this parameter is used.\n", | |||
| "* `top_logprobs`: An integer between 0 and 5 specifying the number of most likely tokens to return at each token position, each with an associated log probability. `logprobs` must be set to true if this parameter is used.\n", | |||
There was a problem hiding this comment.
just adding backticks
enochcheung
commented
Dec 21, 2023
| "Log probabilities of output tokens indicate the likelihood of each token occurring in the sequence given the context. To simplify, a logprob is `log(p)`, where `p` = probability of a token occurring at a specific position based on the other tokens in the sentence. Some key points about `logprobs`:\n", | ||
| "* Higher log probabilities suggest a higher likelihood of the token in that context. This allows users to gauge the model's confidence in its output or explore alternative responses the model considered. \n", | ||
| "* It also allow us to compute the overall probability of a sequence as the sum of the log probs of the individual tokens. This is useful for scoring and ranking model outputs. It's pretty common to take the average logprob of a sentence to choose the best generation.\n", | ||
| "Log probabilities of output tokens indicate the likelihood of each token occurring in the sequence given the context. To simplify, a logprob is `log(p)`, where `p` = probability of a token occurring at a specific position based on the previous tokens in the context. Some key points about `logprobs`:\n", |
There was a problem hiding this comment.
Slightly clarified
- causal masking, distribution only depend on previous tokens
- replace "sentence" with "context"
enochcheung
commented
Dec 21, 2023
| "* Higher log probabilities suggest a higher likelihood of the token in that context. This allows users to gauge the model's confidence in its output or explore alternative responses the model considered. \n", | ||
| "* It also allow us to compute the overall probability of a sequence as the sum of the log probs of the individual tokens. This is useful for scoring and ranking model outputs. It's pretty common to take the average logprob of a sentence to choose the best generation.\n", | ||
| "Log probabilities of output tokens indicate the likelihood of each token occurring in the sequence given the context. To simplify, a logprob is `log(p)`, where `p` = probability of a token occurring at a specific position based on the previous tokens in the context. Some key points about `logprobs`:\n", | ||
| "* Higher log probabilities suggest a higher likelihood of the token in that context. This allows users to gauge the model's confidence in its output or explore alternative responses the model considered.\n", |
There was a problem hiding this comment.
removed trailing space
enochcheung
commented
Dec 21, 2023
| "* It also allow us to compute the overall probability of a sequence as the sum of the log probs of the individual tokens. This is useful for scoring and ranking model outputs. It's pretty common to take the average logprob of a sentence to choose the best generation.\n", | ||
| "Log probabilities of output tokens indicate the likelihood of each token occurring in the sequence given the context. To simplify, a logprob is `log(p)`, where `p` = probability of a token occurring at a specific position based on the previous tokens in the context. Some key points about `logprobs`:\n", | ||
| "* Higher log probabilities suggest a higher likelihood of the token in that context. This allows users to gauge the model's confidence in its output or explore alternative responses the model considered.\n", | ||
| "* Logprob can be any negative number or `0.0`. `0.0` corresponds to 100% probability.\n", |
There was a problem hiding this comment.
I think it is sometimes surprising that logprobs are negative. I think it's important to point out, to contextualize the next point to explain why summing results in the joint probablity which should be lower than the individual logprobs
enochcheung
commented
Dec 21, 2023
| "Log probabilities of output tokens indicate the likelihood of each token occurring in the sequence given the context. To simplify, a logprob is `log(p)`, where `p` = probability of a token occurring at a specific position based on the previous tokens in the context. Some key points about `logprobs`:\n", | ||
| "* Higher log probabilities suggest a higher likelihood of the token in that context. This allows users to gauge the model's confidence in its output or explore alternative responses the model considered.\n", | ||
| "* Logprob can be any negative number or `0.0`. `0.0` corresponds to 100% probability.\n", | ||
| "* Logprobs allow us to compute the joint probability of a sequence as the sum of the logprobs of the individual tokens. This is useful for scoring and ranking model outputs. Another common approach is to take the average per-token logprob of a sentence to choose the best generation.\n", |
There was a problem hiding this comment.
overall -> joint. "overall" can sometimes mean the total probability which is kind of the opposite.
slight re-wording to point out that summing and averaging are two different approaches, where you choose one or the other not both
shyamal-anadkat
approved these changes
Dec 21, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Some minor re-wording, to clarify some minor things
Motivation
Why are these changes necessary? How do they improve the cookbook?
I'll comment individually
For new content
When contributing new content, read through our contribution guidelines, and mark the following action items as completed:
We will rate each of these areas on a scale from 1 to 4, and will only accept contributions that score 3 or higher on all areas. Refer to our contribution guidelines for more details.