Skip to content

Update tgi-messages-api.md #1807

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 8, 2024
Merged

Update tgi-messages-api.md #1807

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions tgi-messages-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,7 +121,7 @@ for message in chat_completion:
Behind the scenes, TGI’s Messages API automatically converts the list of messages into the model’s required instruction format using its [chat template](https://huggingface.co/docs/transformers/chat_templating).


> ##### 💡 Certain OpenAI features, like function calling, are not compatible with TGI. Currently, the Messages API supports the following chat completion parameters: `stream`, `max_new_tokens`, `frequency_penalty`, `logprobs`, `seed`, `temperature`, and `top_p`.
> ##### 💡 Certain OpenAI features, like function calling, are not compatible with TGI. Currently, the Messages API supports the following chat completion parameters: `stream`, `max_tokens`, `frequency_penalty`, `logprobs`, `seed`, `temperature`, and `top_p`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure?

https://github.com/huggingface/text-generation-inference/blob/39af000cb9bfa12628c21c6e65d3ce8930250171/router/src/server.rs#L606

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

coming from here https://huggingface.co/docs/text-generation-inference/messages_api

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also our snippets use max_tokens

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see: https://github.com/huggingface/text-generation-inference/blob/39af000cb9bfa12628c21c6e65d3ce8930250171/router/src/lib.rs#L359

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/huggingface/text-generation-inference/blob/39af000cb9bfa12628c21c6e65d3ce8930250171/router/src/server.rs#L571

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its mapped

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a still an indentation issue in line 113 above, can you add a fix to this pr? Otherwise we can open another one.


### With the JavaScript client

Expand Down Expand Up @@ -299,4 +299,4 @@ endpoint.delete()

The new Messages API in Text Generation Inference provides a smooth transition path from OpenAI models to open LLMs. We can’t wait to see what use cases you will power with open LLMs running on TGI!

_See [this notebook](https://github.com/andrewrreed/hf-notebooks/blob/main/tgi-messages-api-demo.ipynb) for a runnable version of the code outlined in the post._
_See [this notebook](https://github.com/andrewrreed/hf-notebooks/blob/main/tgi-messages-api-demo.ipynb) for a runnable version of the code outlined in the post._