Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Return only new text from text generation #302

Merged
merged 1 commit into from
Dec 12, 2023

Conversation

jonatanklosko
Copy link
Member

Closes #247.

This changes text generation serving to only return the new text (without the prompt). This is consistent with streaming. Also, encoder-decoder models like BART already don't return the input text, since it is used as a "context" rather than a "prompt" to complete.

This is a small breaking change, but the next release is going to be 0.5.0 and I think it's fine.

Initially I thought about adding an option like :return_full_text, but to make it handle leading space in a generic way, we would need to make another tokenizer pass on the input, then prefix replace (that's what hf/transformers do). I don't think this is necessarily worth it, because the end user knows what model they work with, so they can easily concatenate the prompt, either adding a space or not. We can revisit the option if there is an actual use case, but it's usually the new text that users care about.

@jonatanklosko jonatanklosko merged commit 892b31a into main Dec 12, 2023
2 checks passed
@jonatanklosko jonatanklosko deleted the jk-generation-new-text branch December 12, 2023 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Option to exclude prompt from generated text
2 participants