[Feature]: Initial LLM token

[CHesketh76]

🚀 The feature, motivation and pitch

Not sure that this has been implemented but could add in initial tokens (i.e. text) to the beginning of the generation process be possible. So basically having the first few tokens be "Sure Thing!", for example, then the model continues generating tokens from that point on.

Alternatives

No response

Additional context

This is in effort to have more control of the model output for returning certain formats and to reduce randomness in the responses.

[simon-mo]

I think I understand this use case but just to confirm is there a reason that this cannot be part of the prompt?

[CHesketh76]

There is a difference between instructing the model with a prompt like Start all your responses with "Sure Thing!" and the model having Sure Thing! as the initial tokens that follow the begin-response token. The former is an instruction given to the model each time, while the latter means the model is pre-configured to always start responses with 'Sure Thing!'

Here is an example for my use case:
You can also think of it this way Start all your responses with "[" and create a json with the following information that I provide . I am telling the prompt to start as a Json but it is not a guarantee and all too often it will start off saying Here is your Json: [{. But if you have the initial token following the begin response token as [ then it will basically guarantee returning a Json.

[Etelis]

Why not just using the system prompt for this one?

Also, where have you seen this before? I have never encountered such feature

[CHesketh76]

@Etelis What I described was for the system prompt.

[twaka]

I think anthropic's api has similar feature.
https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prefill-claudes-response
I'm not familiar but may it achievable with outlines?