Length constraint causes infinite looping of generation

[piercefreeman]

Describe the issue as clearly as possible:

I saw some generation irregularities similar to #450, so benchmarked it a bit further. It seems like there are certain classes of constraints that cause an infinite loop of the state machine, where the model keeps generating "valid" tokens but they don't contribute to the overall string content.

This script uses a pydantic schema that compiles to a valid regex and state machine, which has length constraints in place: \{[\n ]*"poem"[\n ]*:[\n ]*"(?:[^"\\\x00-\x1f\x7f-\x9f]|\\.){150,200}"[\n ]*\}. Each line of the sequence_generator is logged separately in the format:

{loop} {value.token_ids.shape[1]} {generated text}

Within the loop, runs 0 and 1 complete successfully. On run 2 it stalls with the same maximum length output (Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h).

Even as this generated text stays the same, the value.token_ids keep growing. The tokens evaluate to 28705 (▁) and 13 (<0x0A>), over and over again:

86 153 tensor(28705, device='cuda:0') "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
87 154 tensor(13, device='cuda:0') "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
88 155 tensor(28705, device='cuda:0') "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
89 156 tensor(13, device='cuda:0') "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"

I've included a minimum reproducible case below.

Steps/code to reproduce the bug:

from pydantic import BaseModel, constr
from outlines import models
import json as pyjson
import re
import torch
from unittest.mock import patch
from transformers import AutoTokenizer

prompt = """<|system|>
You are a friendly AI assistant.
You're specialized in mathematics and open source Github repositories.
Your answers must be concise and factual.</s>
<|user|>
Write a very long poem</s>
<|assistant|>
"""

class OutputModel(BaseModel):
    poem: constr(min_length=150, max_length=200)

model = models.transformers("mistralai/Mistral-7B-Instruct-v0.2", device="cuda")
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.2")

from outlines.generate.generator import sequence_generator as original_sequence_generator

def mock_sequence_generator(*args, **kwargs):
    iterator = original_sequence_generator(*args, **kwargs)
    prompt_length = -1
    for i, value in enumerate(iterator):
        text_value = tokenizer.decode(value.token_ids[0])

        if i == 0:
            prompt_length = len(text_value)
            print(f"INITIAL PROMPT: ``` {text_value} ```")
        else:
            print(i, value.token_ids.shape[1], text_value[prompt_length:].strip())
        yield value

rng = torch.Generator(device="cuda")
rng.manual_seed(789001)

with patch("outlines.generate.api.sequence_generator") as seq_gen:
    from outlines.fsm.json_schema import build_regex_from_schema, get_schema_from_signature
    from outlines.generate.api import SequenceGenerator
    from outlines.models import OpenAI
    from outlines.samplers import Sampler, multinomial
    from outlines.generate.regex import regex
    from outlines.samplers import Sampler, multinomial

    seq_gen.side_effect = mock_sequence_generator

    for i in range(20):
        print(f"RUNNING {i}")
        # https://github.com/outlines-dev/outlines/blob/main/outlines/generate/json.py
        schema_object = OutputModel
        whitespace_pattern = None
        sampler = multinomial()
        schema = pyjson.dumps(schema_object.model_json_schema())
        regex_str = build_regex_from_schema(schema, whitespace_pattern)
        print("regex", regex_str)
    
        generator = regex(model, regex_str, sampler)
        generator.format_sequence = lambda x: schema_object.parse_raw(x)
    
        output_obj = generator(prompt, rng=rng)
        print("Final", output_obj)
        print("----------")

Expected result:

Generates 20 poems of length 150-200 characters.

Error message:

RUNNING 0
regex \{[\n ]*"poem"[\n ]*:[\n ]*"(?:[^"\\\x00-\x1f\x7f-\x9f]|\\.){150,200}"[\n ]*\}
INITIAL PROMPT
<s> <|system|>
You are a friendly AI assistant.
You're specialized in mathematics and open source Github repositories.
Your answers must be concise and factual.</s> 
<|user|>
Write a very long poem</s> 
<|assistant|>
{
------
1 68 
2 69 
3 70 "
4 71 "po
5 72 "poem
6 73 "poem":
7 74 "poem": "
8 75 "poem": "Once
9 76 "poem": "Once upon
10 77 "poem": "Once upon a
11 78 "poem": "Once upon a mathematical
12 79 "poem": "Once upon a mathematical means
13 80 "poem": "Once upon a mathematical means,\
14 81 "poem": "Once upon a mathematical means,\n
15 82 "poem": "Once upon a mathematical means,\nIn
16 83 "poem": "Once upon a mathematical means,\nIn a
17 84 "poem": "Once upon a mathematical means,\nIn a land
18 85 "poem": "Once upon a mathematical means,\nIn a land where
19 86 "poem": "Once upon a mathematical means,\nIn a land where complex
20 87 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts
21 88 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign
22 89 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\
23 90 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\n
24 91 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere
25 92 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere inte
26 93 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers
27 94 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance
28 95 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and
29 96 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations
30 97 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing
31 98 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\
32 99 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\n
33 100 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd
34 101 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the
35 102 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Py
36 103 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pyth
37 104 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythag
38 105 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagore
39 106 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean
40 107 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem
41 108 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes
42 109 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its
43 110 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing
44 111 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\
45 112 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n
46 113 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\
47 114 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\n
48 115 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nTh
49 116 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough
50 117 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the
51 118 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden
52 119 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden ratio
53 120 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden ratio'
54 121 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden ratio's
55 122 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden ratio's gentle
56 123 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden ratio's gentle grace
57 124 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden ratio's gentle grace,\
58 125 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden ratio's gentle grace,\n
59 126 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden ratio's gentle grace,\n"
60 127 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden ratio's gentle grace,\n"
61 128 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden ratio's gentle grace,\n"
62 129 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden ratio's gentle grace,\n"
63 130 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden ratio's gentle grace,\n"
64 131 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden ratio's gentle grace,\n"
65 132 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden ratio's gentle grace,\n" 
                       }
66 133 "poem": "Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden ratio's gentle grace,\n" 
                       }</s>
Final poem="Once upon a mathematical means,\nIn a land where complex thoughts reign,\nWhere integers dance and equations sing,\nAnd the Pythagorean theorem takes its swing,\n\nThrough the golden ratio's gentle grace,\n"
----------
RUNNING 1
regex \{[\n ]*"poem"[\n ]*:[\n ]*"(?:[^"\\\x00-\x1f\x7f-\x9f]|\\.){150,200}"[\n ]*\}
INITIAL PROMPT
<s> <|system|>
You are a friendly AI assistant.
You're specialized in mathematics and open source Github repositories.
Your answers must be concise and factual.</s> 
<|user|>
Write a very long poem</s> 
<|assistant|>
{
------
1 68 
2 69 
3 70 "
4 71 "po
5 72 "poem
6 73 "poem":
7 74 "poem": "
8 75 "poem": "Once
9 76 "poem": "Once upon
10 77 "poem": "Once upon a
11 78 "poem": "Once upon a cos
12 79 "poem": "Once upon a cosmic
13 80 "poem": "Once upon a cosmic qu
14 81 "poem": "Once upon a cosmic quilt
15 82 "poem": "Once upon a cosmic quilt,
16 83 "poem": "Once upon a cosmic quilt, where
17 84 "poem": "Once upon a cosmic quilt, where stars
18 85 "poem": "Once upon a cosmic quilt, where stars tw
19 86 "poem": "Once upon a cosmic quilt, where stars twinkle
20 87 "poem": "Once upon a cosmic quilt, where stars twinkle bright
21 88 "poem": "Once upon a cosmic quilt, where stars twinkle bright,
22 89 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\
23 90 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\n
24 91 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA
25 92 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe
26 93 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hum
27 94 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums
28 95 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in
29 96 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm
30 97 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm,
31 98 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a
32 99 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand
33 100 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cos
34 101 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic
35 102 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night
36 103 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\
37 104 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\n
38 105 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery
39 106 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery spe
40 107 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck
41 108 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of
42 109 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of st
43 110 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stard
44 111 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust
45 112 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust d
46 113 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances
47 114 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in
48 115 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in unt
49 116 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold
50 117 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight
51 118 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\
52 119 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\n
53 120 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nS
54 121 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSway
55 122 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying
56 123 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to
57 124 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the
58 125 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the viol
59 126 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the violin
60 127 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the violin'
61 128 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the violin's
62 129 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the violin's son
63 130 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the violin's sonata
64 131 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the violin's sonata in
65 132 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the violin's sonata in the
66 133 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the violin's sonata in the infinite
67 134 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the violin's sonata in the infinite"
68 135 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the violin's sonata in the infinite"
69 136 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the violin's sonata in the infinite"
70 137 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the violin's sonata in the infinite"
71 138 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the violin's sonata in the infinite"

  }
72 139 "poem": "Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the violin's sonata in the infinite"

  }</s>
Final poem="Once upon a cosmic quilt, where stars twinkle bright,,\nA universe hums in rhythm, a grand cosmic night.\nEvery speck of stardust dances in untold delight,\nSwaying to the violin's sonata in the infinite"
----------
RUNNING 2
regex \{[\n ]*"poem"[\n ]*:[\n ]*"(?:[^"\\\x00-\x1f\x7f-\x9f]|\\.){150,200}"[\n ]*\}
INITIAL PROMPT
<s> <|system|>
You are a friendly AI assistant.
You're specialized in mathematics and open source Github repositories.
Your answers must be concise and factual.</s> 
<|user|>
Write a very long poem</s> 
<|assistant|>
{
------
1 68 
2 69 
3 70 "
4 71 "po
5 72 "poem
6 73 "poem":
7 74 "poem": "
8 75 "poem": "Once
9 76 "poem": "Once upon
10 77 "poem": "Once upon a
11 78 "poem": "Once upon a cos
12 79 "poem": "Once upon a cosmic
13 80 "poem": "Once upon a cosmic dance
14 81 "poem": "Once upon a cosmic dance,\
15 82 "poem": "Once upon a cosmic dance,\n
16 83 "poem": "Once upon a cosmic dance,\nIn
17 84 "poem": "Once upon a cosmic dance,\nIn the
18 85 "poem": "Once upon a cosmic dance,\nIn the vast
19 86 "poem": "Once upon a cosmic dance,\nIn the vast exp
20 87 "poem": "Once upon a cosmic dance,\nIn the vast expanse
21 88 "poem": "Once upon a cosmic dance,\nIn the vast expanse of
22 89 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numer
23 90 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric
24 91 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance
25 92 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\
26 93 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\n
27 94 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere
28 95 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere dig
29 96 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits
30 97 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits inter
31 98 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertw
32 99 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined
33 100 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in
34 101 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand
35 102 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design
36 103 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\
37 104 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\n
38 105 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd
39 106 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd in
40 107 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity
41 108 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was
42 109 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but
43 110 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a
44 111 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear
45 112 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line
46 113 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\
47 114 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n
48 115 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\
49 116 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\n
50 117 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNum
51 118 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers
52 119 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered
53 120 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet
54 121 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets
55 122 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to
56 123 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the
57 124 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars
58 125 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\
59 126 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\n
60 127 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nY
61 128 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYield
62 129 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded
63 130 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h
64 131 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
65 132 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
66 133 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
67 134 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
68 135 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
69 136 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
70 137 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
71 138 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
72 139 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
73 140 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
74 141 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
75 142 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
76 143 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
77 144 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
78 145 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
79 146 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
80 147 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
81 148 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
82 149 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
83 150 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
84 151 "poem": "Once upon a cosmic dance,\nIn the vast expanse of numeric romance,\nWhere digits intertwined in grand design,\nAnd infinity was but a linear line,\n\nNumbers whispered sweet secrets to the stars,\nYielded h"
[2000 more lines of the same value, with growing token lengths]

Outlines/Python version information:

Version information:

outlines==0.0.32
transformers==4.37.2
torch==2.1.1

Running on a H100 GPU.

Context for the issue:

No response

[rlouf]

This is not the first time this shows up, but the first time we have a report detailed enough and that we can work with. Thank you for putting the time, I will take a look!

[lapp0]

Hi @piercefreeman

Thanks for your great debug details. This is an issue that we've run into a lot. It's not that the state machine has an infinite loop, it's that the language models suffer from a "Repetition Problem".

I ran your reproduction script with no error after changing

whitespace_pattern = None

to

whitespace_pattern=r"[ ]?".

@remic33 since this keeps coming up, we should consider setting the whitespace_pattern default to [\t\r\n ]{0,8} and allowing users to change it to None if they want to remove this reasonable constraint. I'll open a PR if you agree.

[rlouf]

Should be solved by #916. Feel free to re-open if this problem persists.