Increase max tokens from 200 to 1000 and truncation length to 4096

[CRCODE22]

I tried to do this on my own but it is not working yet here is the modifications I made.

I am using a 70B model with 4096 context so I need those values to go from max tokens to 1000 and truncation length to 4096 because this language model is so good at writing detailed prompts it exceeds those default low values. and I am getting warnings related to that the token is too long so how can you pass that to text generation web ui to use max tokens of 1000 and truncation length of 4096? Here is what I tried:

This is my version of IF_promptMKR_preset.yaml

max_new_tokens: 1000
temperature: 1.21
top_p: 0.91
top_k: 35
typical_p: 1
epsilon_cutoff: 0
eta_cutoff: 0
tfs: 1
top_a: 0
repetition_penalty: 1.15
repetition_penalty_range: 0
encoder_repetition_penalty: 1
no_repeat_ngram_size: 0
min_length: 0
seed: -1
do_sample: true
mirostat_mode: 0
mirostat_tau: 5
mirostat_eta: 0.1
penalty_alpha: 0
num_beams: 1
length_penalty: 1.31
early_stopping: false
truncation_length: 4096

Changes to if_prompt_mkr.py

    data = {
        'user_input': prompt,
        'history': {'internal': [], 'visible': []},
        'mode': "chat",
        'your_name': "You",
        'character': character,
        'instruction_template': instruction_template,
        'preset': preset,
        'regenerate': False,
        '_continue': False,
        'stop_at_newline': False,
        'chat_prompt_size': 4096,
        'max_new_tokens': 1000,
        'chat_generation_attempts': 1,
        'chat-instruct_command': 'Act like a prompt creator, brake keywords by comas, provide high quality, non-verboose, coherent, brief, concise, and not superfluous prompts, Only write the visuals elements of the picture, Never write art commentaries or intentions. Construct the prompt with the componet format, Always include all the keywords from the request verbatim as the main subject of the response: "".\n\n',
        'seed': -1,
        'add_bos_token': True,
        'custom_stopping_strings': [stopping,],
        'truncation_length': 4096,
        'ban_eos_token': False,
    }

[if-ai]

Right I purposely made it short, because the SD 1.5 models did not processed most of the words, I even had those features exposed in the settings but I decided to hide them because it felt too complicated. The prompt could be huge but SD will only support about 200 tokens decently plus very few people can run Llama 70b at the moment.

…

On Wed, Aug 16, 2023, 8:08 PM CRCODE22 ***@***.***> wrote: I tried to do this on my own but it is not working yet here is the modifications I made. I am using a 70B model with 4096 context so I need those values to go from max tokens to 1000 and truncation length to 4096 because this language model is so good at writing detailed prompts it exceeds those default values. This is my version of IF_promptMKR_preset.yaml max_new_tokens: 1000 temperature: 1.21 top_p: 0.91 top_k: 35 typical_p: 1 epsilon_cutoff: 0 eta_cutoff: 0 tfs: 1 top_a: 0 repetition_penalty: 1.15 repetition_penalty_range: 0 encoder_repetition_penalty: 1 no_repeat_ngram_size: 0 min_length: 0 seed: -1 do_sample: true mirostat_mode: 0 mirostat_tau: 5 mirostat_eta: 0.1 penalty_alpha: 0 num_beams: 1 length_penalty: 1.31 early_stopping: false truncation_length: 4096 Changes to if_prompt_mkr.py data = { 'user_input': prompt, 'history': {'internal': [], 'visible': []}, 'mode': "chat", 'your_name': "You", 'character': character, 'instruction_template': instruction_template, 'preset': preset, 'regenerate': False, '_continue': False, 'stop_at_newline': False, 'chat_prompt_size': 4096, 'max_new_tokens': 1000, 'chat_generation_attempts': 1, 'chat-instruct_command': 'Act like a prompt creator, brake keywords by comas, provide high quality, non-verboose, coherent, brief, concise, and not superfluous prompts, Only write the visuals elements of the picture, Never write art commentaries or intentions. Construct the prompt with the componet format, Always include all the keywords from the request verbatim as the main subject of the response: "".\n\n', 'seed': -1, 'add_bos_token': True, 'custom_stopping_strings': [stopping,], 'truncation_length': 4096, 'ban_eos_token': False, } — Reply to this email directly, view it on GitHub <#7>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFBUFQV6YAF2JSBDGHQVXNTXVULDPANCNFSM6AAAAAA3S7ECNI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[CRCODE22]

The tokens at 200 is ok but what should be increased is the truncation length to 4096 and context size to 4096 because there is Llama 2 models and other models now and that support 4096 and 8192 and now even 16K context and truncation length. Can you make that an option?

[if-ai]

I can but it doesn't change anything because it doesn't keep history but I will add a way to change it. I don't remember if just changing the yaml preset file IF_prompt_MKR_preset.yaml can work but yes I will add it. This weekend