feat: Log probabilities support #221
Conversation
There was a problem hiding this comment.
@ruivieira I added some in-place comments
In addition, please add logprobs support for requests with streaming=true (sendStreamingResponse), please take into consideration comments in PR #215 relevant to streaming (don't pass/use request in streaming.go).
Please add tests for the simulator with logprobs:
completions and chat completions requests with logprobs with and without streaming, and check the response
ruivieira
force-pushed
the
logprobs-simple
branch
from
4f49952 to
52204d2
Compare
There was a problem hiding this comment.
Hi @ruivieira , I added some comments.
Please pay attention that there are changes in tests utility functions.
ruivieira
force-pushed
the
logprobs-simple
branch
from
029ab1a to
1332201
Compare
|
/lgtm @ruivieira - approved the PR, you just need to fix merge conflicts |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
ruivieira
dismissed stale reviews from mayabar and github-actions[bot]
via
5762aa3
ruivieira
force-pushed
the
logprobs-simple
branch
3 times, most recently
from
0ccf03d to
d201bd2
Compare
Add support for log probabilities in both chat completions and text completions APIs: - Add logprobs parameter to chat completions with top_logprobs support - Add logprobs parameter to text completions with configurable count - Implement streaming and non-streaming logprobs functionality - Add comprehensive test coverage for all logprobs scenarios - Add utility functions for logprobs calculation and validation - Support both echo and random modes for logprobs generation - Include proper token-level probability information in responses - Fix undefined variables in tests after upstream merge Signed-off-by: Rui Vieira <ruidevieira@googlemail.com>
ruivieira
force-pushed
the
logprobs-simple
branch
from
d201bd2 to
8b80d67
Compare
|
@mayabar thank for you for the comments. The conflict has been solved. |
|
Fixes #213 |
2 participants
This PR implements log probabilities support in llm-d-inference-sim enabling compatibility with evaluation frameworks and other API clients that require token-level probability.
API Coverage
/v1/completions: logprobs parameter withTextLogprobsresponse structure/v1/chat/completions: logprobs boolean + top_logprobs integer withChatLogprobsresponse structureRefer to #213
Replaces #215