Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Messages API is missing a whitespace after data: to be compliant #2019

Closed
2 of 4 tasks
philschmid opened this issue Jun 5, 2024 · 1 comment
Closed
2 of 4 tasks

Comments

@philschmid
Copy link
Member

System Info

ghcr.io/huggingface/text-generation-inference:2.0.4

Information

  • Docker
  • The CLI directly

Tasks

  • An officially supported command
  • My own modifications

Reproduction

  1. Start container
  2. run curl request with streaming

Expected behavior

TGI

data:{"id":"","object":"text_completion","created":1717586509,"model":"meta-llama/Meta-Llama-3-8B-Instruct","system_fingerprint":"2.0.4-sha-f426a33","choices":[{"index":0,"delta":{"role":"assistant","content":"Here"},"logprobs":null,"finish_reason":null}]}

vLLM

data: {"id":"cmpl-a33baad7fc1b46518194d433bd046551","object":"chat.completion.chunk","created":1717586822,"model":"meta-llama/Meta-Llama-3-8B-Instruct","choices":[{"index":0,"delta":{"content":""},"finish_reason":"stop"}],"usage":{"prompt_tokens":23,"total_tokens":68,"completion_tokens":45}}

OpenAI

data: {"id":"chatcmpl-9Wj3xCu6xZ2GlmuQ3hKQuAeeNki4C","object":"chat.completion.chunk","created":1717586745,"model":"gpt-4o-2024-05-13","system_fingerprint":"fp_319be4768e","choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}]}
@drbh
Copy link
Collaborator

drbh commented Jun 5, 2024

Hey @philschmid this should be resolved in the main

The issue was originally related to axum. In versions before 0.7.0 there was no space added after the data: field. This PR fixes it and should be available tokio-rs/axum#2149

As of last week TGI bumped axum to 0.7.0 #1959 so we should see the space in out responses now

@drbh drbh closed this as completed Jun 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants