Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dynamically intterupt token generation #599

Open
Bartvelp opened this issue Aug 10, 2023 · 0 comments · May be fixed by #733
Open

Dynamically intterupt token generation #599

Bartvelp opened this issue Aug 10, 2023 · 0 comments · May be fixed by #733
Labels
enhancement New feature or request

Comments

@Bartvelp
Copy link

Is your feature request related to a problem? Please describe.
During the generation of tokens I would like to stop when I encounter some condition that changes during the runtime, using Stream=True.
E.g. I would like to stop generation after 5 lines of generation.

Describe the solution you'd like
I would like a method on llm called stop(), or interrupt(), that forces the model to stop after the next token is generated, similar to CTRL+C in the regular llama.cpp

Describe alternatives you've considered
I have considered adding a newline as stop token, but I think this is not performant. Another way I can think of is changing the stop list after passing it the generation method, but that feels hacky.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants