Generating str/int/float/bool values with GPT2 using generate.json leads to absurd results

[gaspard-dv]

Describe the issue as clearly as possible:

I experimented a bit with outlines and I wonder what would be the recommended way to generate bools/numbers/ints.

• When I use generate.format(model, <type>, allow_empty_tokens=False) the resulting response is often just an empty string, which can't be converted to the desired <type>.
• When I use generate.json(model, json.dumps({"type": "<type_in_json_schema_style>"}) the model tends to "block" or to produce stupid answers.

Regardless of my question, I believe that this second bullet point is an issue, so I'm adding a reproduction snippet below.

NB: I'm not allowed to add the "correctness" label.

Steps/code to reproduce the bug:

from json import dumps

from outlines import models
from outlines.text.generate import continuation
from outlines.text.generate import json
import torch

model = models.transformers("gpt2")
prompt = "a b c d e f"

rng = torch.Generator()

rng.manual_seed(789001)
print(continuation(model, max_tokens=30)(prompt, rng=rng))

rng.manual_seed(789001)
print(json(model, dumps({"type": "string"}))(prompt, rng=rng))

Expected result:

EXPECTED:
g h i j k (...)
g h i j k (...)

ACTUAL:
g h i j k (...)
for all readable words H. Servo Oamber Trident Warrior Drug of others (...)

Error message:

No response

Outlines/Python version information:

Outlines 0.0.13
Python 3.10.12

Context for the issue:

No response

[rlouf]

Will take a look!

[brandonwillard]

Just as described in #460 (comment), this should be converted to a discussion.

[brandonwillard]

It does look like this one has the potential to involve a real bug, though, so we might end up changing this back...