-
-
Notifications
You must be signed in to change notification settings - Fork 10
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Increase max tokens from 200 to 1000 and truncation length to 4096 #7
Comments
Right I purposely made it short, because the SD 1.5 models did not
processed most of the words, I even had those features exposed in the
settings but I decided to hide them because it felt too complicated.
The prompt could be huge but SD will only support about 200 tokens decently
plus very few people can run Llama 70b at the moment.
…On Wed, Aug 16, 2023, 8:08 PM CRCODE22 ***@***.***> wrote:
I tried to do this on my own but it is not working yet here is the
modifications I made.
I am using a 70B model with 4096 context so I need those values to go from
max tokens to 1000 and truncation length to 4096 because this language
model is so good at writing detailed prompts it exceeds those default
values.
This is my version of IF_promptMKR_preset.yaml
max_new_tokens: 1000
temperature: 1.21
top_p: 0.91
top_k: 35
typical_p: 1
epsilon_cutoff: 0
eta_cutoff: 0
tfs: 1
top_a: 0
repetition_penalty: 1.15
repetition_penalty_range: 0
encoder_repetition_penalty: 1
no_repeat_ngram_size: 0
min_length: 0
seed: -1
do_sample: true
mirostat_mode: 0
mirostat_tau: 5
mirostat_eta: 0.1
penalty_alpha: 0
num_beams: 1
length_penalty: 1.31
early_stopping: false
truncation_length: 4096
Changes to if_prompt_mkr.py
data = {
'user_input': prompt,
'history': {'internal': [], 'visible': []},
'mode': "chat",
'your_name': "You",
'character': character,
'instruction_template': instruction_template,
'preset': preset,
'regenerate': False,
'_continue': False,
'stop_at_newline': False,
'chat_prompt_size': 4096,
'max_new_tokens': 1000,
'chat_generation_attempts': 1,
'chat-instruct_command': 'Act like a prompt creator, brake keywords by comas, provide high quality, non-verboose, coherent, brief, concise, and not superfluous prompts, Only write the visuals elements of the picture, Never write art commentaries or intentions. Construct the prompt with the componet format, Always include all the keywords from the request verbatim as the main subject of the response: "".\n\n',
'seed': -1,
'add_bos_token': True,
'custom_stopping_strings': [stopping,],
'truncation_length': 4096,
'ban_eos_token': False,
}
—
Reply to this email directly, view it on GitHub
<#7>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFBUFQV6YAF2JSBDGHQVXNTXVULDPANCNFSM6AAAAAA3S7ECNI>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
The tokens at 200 is ok but what should be increased is the truncation length to 4096 and context size to 4096 because there is Llama 2 models and other models now and that support 4096 and 8192 and now even 16K context and truncation length. Can you make that an option? |
I can but it doesn't change anything because it doesn't keep history but I will add a way to change it. I don't remember if just changing the yaml preset file IF_prompt_MKR_preset.yaml can work but yes I will add it. This weekend |
I tried to do this on my own but it is not working yet here is the modifications I made.
I am using a 70B model with 4096 context so I need those values to go from max tokens to 1000 and truncation length to 4096 because this language model is so good at writing detailed prompts it exceeds those default low values. and I am getting warnings related to that the token is too long so how can you pass that to text generation web ui to use max tokens of 1000 and truncation length of 4096? Here is what I tried:
This is my version of IF_promptMKR_preset.yaml
max_new_tokens: 1000
temperature: 1.21
top_p: 0.91
top_k: 35
typical_p: 1
epsilon_cutoff: 0
eta_cutoff: 0
tfs: 1
top_a: 0
repetition_penalty: 1.15
repetition_penalty_range: 0
encoder_repetition_penalty: 1
no_repeat_ngram_size: 0
min_length: 0
seed: -1
do_sample: true
mirostat_mode: 0
mirostat_tau: 5
mirostat_eta: 0.1
penalty_alpha: 0
num_beams: 1
length_penalty: 1.31
early_stopping: false
truncation_length: 4096
Changes to if_prompt_mkr.py
The text was updated successfully, but these errors were encountered: