Use Better Models, Better Prompts in Docs

[lapp0]

Some of the results in the docs are difficult to reproduce. Prompt engineering and better model recommendations are applied here to alleviate that.

fixes #586

• Update docs: never recommend mistralai/Mistral-7B-v0.1, always prefer WizardLM/WizardMath-7B-V1.1 for math above kindergarten level and mistralai/Mistral-7B-Instruct-v0.2 for everything else.
  • Smoke tested all README.md and quickstart.md examples. Replaced model with mistralai/Mistral-7B-Instruct-v0.2, generation quality was equal or better in all cases.
  • fixes Error during constrained JSON generation. #612 (see Error during constrained JSON generation. #612 (comment))
• don't force device="cuda" in any outlines.models.transformers
• fix arithmetic grammar to not have recursion issue (fixes Error when trying to run the grammar example with Mistral #580)

Notes:

https://github.com/outlines-dev/outlines#multiple-choices

• update prompt
• use mistralai/Mistral-7B-Instruct-v0.2

Before (first token probs):

Top Raw Tokens: 'The': 0.560, '\n': 0.212, 'I': 0.106, 'Great': 0.017, 'From': 0.017, 'We': 0.013, 'F': 0.011, 'They': 0.006, EOS: 0.006
Top Guided Tokens: 'P': 0.456, 'N': 0.330, 'Pos': 0.173, 'Ne': 0.018, 'Neg': 0.016, 'Po': 0.007, 'P': 0.000, 'N': 0.000, EOS: -0.000

After (first token probs):

Top Raw Tokens: 'Pos': 0.858, 'POS': 0.060, '\n': 0.031, 'pos': 0.022, '+': 0.007, '**': 0.007, 'The': 0.004, 'Pos': 0.002, EOS: 0.002
Top Guided Tokens: 'Pos': 1.000, 'P': 0.000, 'Po': 0.000, 'Neg': 0.000, 'Ne': 0.000, 'N': 0.000, 'P': 0.000, 'N': 0.000, EOS: -0.000

https://github.com/outlines-dev/outlines?tab=readme-ov-file#type-constraint

• use `meta-math/MetaMath-Llemma-7B

Now generates 1 + 2 = 3 and sqrt(2) = 1.41421356.

https://github.com/outlines-dev/outlines?tab=readme-ov-file#efficient-json-generation-following-a-pydantic-model

• use mistralai/Mistral-7B-Instruct-v0.2

Before:

generator("Give me a character description", rng=rng)
Character(name='Anderson', age=28, armor=<Armor.chainmail: 'chainmail'>, weapon=<Weapon.sword: 'sword'>, strength=8)

generator("Give me an interesting character description", rng=rng)
Character(name='B__k_ __lf', age=15, armor=<Armor.plate: 'plate'>, weapon=<Weapon.sword: 'sword'>, strength=55)

name=B__k_ __lf?

After:

generator("Give me a character description", rng=rng)
Character(name='Alex Chen', age=28, armor=<Armor.leather: 'leather'>, weapon=<Weapon.bow: 'bow'>, strength=65)

generator("Give me an interesting character description", rng=rng)
Character(name='Vivian Thr', age=44, armor=<Armor.plate: 'plate'>, weapon=<Weapon.crossbow: 'crossbow'>, strength=125)

https://github.com/outlines-dev/outlines?tab=readme-ov-file#using-context-free-grammars-to-guide-generation

• changed prompt
• Use WizardLM/WizardMath-7B-V1.1

Before:

• Error when trying to run the grammar example with Mistral #580 (comment)

After:

sequence = generator("Alice had 4 apples and Bob ate 2. Write an expression for Alice's apples:")

print(sequence)
# (8-2)

https://github.com/outlines-dev/outlines?tab=readme-ov-file#open-functions

• Use WizardLM/WizardMath-7B-V1.1
• Updated prompt, explicitly say object type

Before:

Generates whitespace, never ending

After:

print(add(**result))
# 3

https://github.com/lapp0/outlines/blob/cfg-examples/docs/quickstart.md#regex-guided-generation

• new prompt
• use mistralai/Mistral-7B-Instruct-v0.2

Before:

0.0.0.0

After:

result = generator("What is the IP address of localhost?\nIP: ")
print(result)
# 127.0.0.100

https://github.com/lapp0/outlines/blob/cfg-examples/docs/quickstart.md#deploy-using-vllm-and-fastapi

This always results in low quality generations, debugging it can be a separate issue. It hardly ever says just "paris" regardless of the model. Pro

curl http://127.0.0.1:8000/generate \
    -d '{
        "prompt": "<s>Capital of France:</s><s> ",
        "schema": {"type": "string"}

        }'

Changing to

curl http://127.0.0.1:8000/generate \
    -d '{
        "prompt": "Question: What is a language model? Answer:",
        "schema": {"type": "string"}
        }'

[rlouf]

Awesome! It looks like you included commits from #616. Can we keep the two PRs separate?

[lapp0]

My mistake. Fixed!

[rlouf]

Thank you so much, contributions to the documentation are very appreciated!