Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

logits API support #67

Merged
merged 25 commits into from
Sep 12, 2024
Merged

logits API support #67

merged 25 commits into from
Sep 12, 2024

Conversation

qmeng222
Copy link
Collaborator

@qmeng222 qmeng222 commented Sep 4, 2024 •
edited
Loading

No description provided.

@qmeng222
implement logits and logprobs output, add debug print statements
d4e88d3
@qmeng222
debug (in progress)
f833649
@qmeng222
Merge branch 'main' of https://github.com/NexaAI/nexa-sdk into qingyi…
946da23 
…ng-logits
@qmeng222
implement the expected structure in response body
e69c81c
@qmeng222
debug null logprobs in Nexa SDK server
20f10a6
@qmeng222
add additional intermediate print statements for enhanced debugging
e79a043
@qmeng222
handle and accumulate logprobs for completion format
3bad104
@qmeng222
clean code
09f0d5c
@qmeng222
update NexaTextInference class to handle logprobs
12c1904
@qmeng222
add logprobs functionality to Llama class
7c15aa0
@qmeng222
troubleshoot the CI issue
c4cd6fb
@qmeng222
solve pr conflict: update completion chunk format for streaming
5a090c9
@qmeng222
Merge remote-tracking branch 'origin/main' into qingying-logits
9fc5b98
@qmeng222
fix logprobs handling consistency in llama.py
fcdfc6c
@qmeng222
implement logprobs and top_logprobs functionality
21d5cbc
@qmeng222
fix TypeError in create_completion by conditionally passing top_logprobs
89780b9
@qmeng222
add top_logprobs parameter to Llama.create_completion method
9ba5684
@qmeng222
ensure logprobs is being passed as a boolean throughout the call chain
49615ec
@qmeng222
set logprobs: true and top_logprobs: 4 by default for both the /v1/co…
1c77636 
…mpletions and /v1/chat/completions endpoints
@qmeng222
remove redundant logprobs CLI argument
a37f501
@qmeng222
comment out top_logprobs CLI argument
329993f
@zhiyuan8 zhiyuan8 changed the title Qingying logits logits API support Sep 10, 2024
@JoyboyBrian
Copy link
Collaborator

LGTM

@zhiyuan8
Copy link
Contributor

Fix merge conflicts and ready to merge

@qmeng222
Copy link
Collaborator Author

git checkout main
git pull origin main
git checkout qingying-logits
git merge main
Accept current change:
20240910-211203
20240910-211210

@zhiyuan8
Copy link
Contributor

@JoyboyBrian plz check merge conflicts, we just updated llama.cpp

JoyboyBrian
JoyboyBrian reviewed Sep 11, 2024
View reviewed changes
Copy link
Collaborator

@JoyboyBrian JoyboyBrian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

conflicts resolved

@zhiyuan8 zhiyuan8 merged commit 064fbdd into main Sep 12, 2024
2 checks passed
@JoyboyBrian JoyboyBrian deleted the qingying-logits branch October 17, 2024 22:04
@JoyboyBrian JoyboyBrian restored the qingying-logits branch October 17, 2024 22:04
@JoyboyBrian JoyboyBrian deleted the qingying-logits branch October 24, 2024 20:11
@JoyboyBrian JoyboyBrian restored the qingying-logits branch October 24, 2024 20:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JoyboyBrian JoyboyBrian JoyboyBrian left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@qmeng222 @JoyboyBrian @zhiyuan8