How to prompt Falcon correctly for each model type

[cmp-nct]

Update

With the new release today this is mostly educational now.
Use the new -enc parameter and it will automatically encapsulate your prompt with the right syntax.
Also works in instruct mode -ins
And with a system prompt -sys

General:
Important params for ggllm.cpp:

• The "-e" parameter allows you to add newlines into your prompt by writing "\n".
• "--no-penalize-nl" can be used if you would like the answer to more likely include new lines.
• "-b 512" is used if you have plenty of VRAM and want to use a large prompt.

All Wizard models
Who is smarter, Newton or Einstein?\n### Response:

Who is smarter, Newton or Einstein?
### Response:Both Sir Isaac Newton and Albert Einstein were incredibly intelligent individuals who made significant contributions to science and mathematics. However, it is difficult to definitively declare one of them as "smarter" than the other, as intelligence can be subjective and multifaceted. Both men had different areas of expertise and made groundbreaking discoveries in their respective fields. Ultimately, it comes down to personal preference and opinion as to who is considered smarter.<|endoftext|>

Original Falcon Instruct
User: Who is smarter, Newton or Einstein?\nAssistant:

User: Who is smarter, Newton or Einstein?
Assistant: It is impossible to compare the intelligence of two people as every individual has unique abilities and areas of expertise. However, both Newton and Einstein made significant contributions to physics and are widely regarded as geniuses in their respective fields.<|endoftext|> [end of text]

OpenAssist
<|prompter|>Who is smarter, Newton or Einstein?<|endoftext|><|assistant|>

<|prompter|>Who is likely smarter and why, Newton or Einstein?<|endoftext|><|assistant|>Newton was undoubtedly a genius, but in terms of sheer intelligence, Einstein may have had the edge. Einstein's theories of relativity revolutionized our understanding of space and time, while Newton made groundbreaking contributions to calculus and the laws of motion. Both were brilliant scientists who left an indelible mark on human knowledge, but it is difficult to compare their intelligence directly as they lived in different eras and had unique areas of expertise.<|/endoftext|><|endoftext|>

Falcon special tokens - guiding the foundation models
Falcon was trained with data from forums like Quora and science magazines, TII used special tokens to replace some headers which you can leverage.
This can be used in fine tuned as well as in the foundation models to guide the response, it can also trigger weird behavior.
This will cause Falcon to write a scientific abstract:
>>DOMAIN<<\nEvolutionary Biology\n>>TITLE<<\nAbout the origin of falcons\n>>ABSTRACT<<

>>DOMAIN<<
Evolutionary Biology
>>TITLE<<
About the origin of falcons
>>ABSTRACT<<

The monophyly of the falcon family Accipitridae is strongly supported by morphological and behavioral data. However, recent molecular data revealed a paraphyletic structure for the group. Based on the mitochondrial 16S rRNA, the pygmy falcon, P. nanus, is embedded within the kestrel clade, which was previously considered monophyletic. We tested the monophyly of the falcon family by using two datasets obtained from nuclear introns and the mitochondrial control region. Our analyses provide no evidence for the monophyly of the falcon family. P. nanus is monophyletic with the kestrel clade, and the rest of the pygmy falcon clade is paraphyletic to the rest of the falcon family. Our results are in agreement with other analyses based on morphological and behavioral data. The monophyly of the falcon family is refuted.

This will cause it to write a question/answer/comment forum-like sequence:
>>QUESTION<<\nWhat is the answer to everything?\n>>ANSWER<<

>>QUESTION<<
What is the answer to everything?
>>ANSWER<< 42, of course.
>>COMMENT<< @mikey, i will be the witness of this joke. Just give me the time to get a drink. :)>>COMMENT<< Hey, wait!  24 is a better answer.  There are 24 hours in a day.  24 is closer to pi

[maddes8cht]

Okay -
I'm not exactly sure what to write in the prompt
You said:
All Wizard models

Who is smarter, Newton or Einstein?\n### Response:

do i actually write the whole line in the prompt, like

-p "Who is smarter, Newton or Einstein?\n### Response:"

Same with all the other lines:

In OpenAssist, do i have to write

-p "<|prompter|>Who is smarter, Newton or Einstein?<|endoftext|><|assistant|>"

either on the prompt or a prompt file?
And in interactive mode?
Do i need this

<|prompter|>
<|endoftext|>
<|assistant|>

tagging to get good results?

Actually, yes, i was wondering sometimes about the way Falccon responds.

Then, now we have

DOMAIN<<
TITLE<<
ABSTRACT<<
and
QUESTION<<
ANSWER<<
COMMENT<<
as tags with a meaning,
Are there more of this?

[cmp-nct]

1. You really write it like this, yes.
   -e -p "<|prompter|>Who is smarter, Newton or Einstein?<|endoftext|><|assistant|>"
   The -e means that "\n" is internally transformed into a new line (return), you don't need it for openassistant but often you want lines. It does not hurt if you add it in any case.

2. The special tokens really need to be written exactly as they are meant including the case of each letter and no spaces.

3. You can always control if the special tokens were correct by writing --verbose-prompt. When you see the special word processed as one number it was correctly integrated. If you'd see it split into multiple numbers/tokens you know something is wrong. With the latest ggllm.cpp those problems are mostly solved but not in all cases (you can report them if an important special word doesn't work and is correctly written)

4. These special tokens are fine tuning vocabulary. This means the model was trained to behave differently when they see it. That's why it's important to use the correct syntax for each model. You could use OpenAssistant prompts for Wizard but Wizard will not understand them and quality/response is significantly reduced in that case.

5. when you look at the original HF model page, you'll find a file called tokenizer.json (large file, you can open it in any editor)
   One of the first sections of that file is "special tokens" or "added vocabulary" which lists every token that has been added beyond normal vocabulary.
   The original TII models all have these:

     {">>TITLE<<", 0},
     {">>ABSTRACT<<", 1},
     {">>INTRODUCTION<<", 2},
     {">>SUMMARY<<", 3},
     {">>COMMENT<<", 4},
     {">>ANSWER<<", 5},
     {">>QUESTION<<", 6},
     {">>DOMAIN<<", 7},
     {">>PREFIX<<", 8},
     {">>SUFFIX<<", 9},
     {">>MIDDLE<<", 10},
     {"<|endoftext|>", 11},

[cmp-nct]

With the new release today this is mostly educational now.
Use the new -enc parameter and it will automatically encapsulate your prompt with the right syntax.
Also works in instruct mode -ins
And with a system prompt -sys

[Sledge-]

Does anyone know how to apply these modes when the model is deployed via a SageMaker endpoint?

[cmp-nct]

That's all just when using ggllm (this project), no matter how you deploy it. When using ggllm it will automatically detect the model in most cases when using "-enc" and you won't have to care about syntax.
When not using ggllm you'll need to manually write the syntax without any changes to what the model expects, any tiny change in the syntax will reduce quality significantly