What is [Prompt len] and [Tokens to Generate]?

[fishiu]

Sorry I am not quite familiar with inference: in fine-tune/training, I simply use the concept of max_seq_length. Are [Prompt len] and [Tokens to Generate] the same as max_seq_length? How could they be different?

[RahulSChand]

You are right that in fine-tuning/training there is no concept of "prompt Len" & "tokens to generate". Only max_seq_length is required. When you use the GitHub site, the max_seq_length = prompt Len + tokens to Generate.

The prompt Len & context Len concept is only for inference time. For example, if you have a question which is made of 100 words(tokens) and you want to generate an answer of 500 tokens. Here the first 100 tokens are processed at once while the next 500 tokens are processed token by token. Therefore a distinction is needed. The first 100 words are your "prompt Len" & the next 500 words are your "tokens to generate"

[fishiu]

Thanks for your reply! So if I want to get the memory result of fine-tuning, I should set "tokens to generate" to 0 right? However that is forbidden (warning that it have to be positive)

[RahulSChand]

You can set it as 1. The results will be almost exactly the same (maybe a 1-2 MB difference at most)