Pydantic example from README crashes

[crclark]

Describe the issue as clearly as possible:

---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
Cell In[1], line 40
     37 rng = torch.Generator(device="cuda")
     38 rng.manual_seed(789001)
---> 40 character = generator("Give me a character description", rng=rng)
     42 print(repr(character))
     43 # Character(name='Anderson', age=28, armor=<Armor.chainmail: 'chainmail'>, weapon=<Weapon.sword: 'sword'>, strength=8)

File ~/<redacted>/lib/python3.10/site-packages/outlines/generate/api.py:207, in SequenceGenerator.__call__(self, prompts, max_tokens, stop_at, rng)
    205 while True:
    206     try:
--> 207         last_state = next(states)
    208         if max_tokens or stop_sequences:
    209             token_ids = last_state.token_ids

File ~/<redacted>/lib/python3.10/site-packages/outlines/generate/generator.py:81, in sequence_generator(model, sampler, fsms, token_ids, sequence_weights, attention_masks, fsm_states, rng)
     76     raise ContextLengthExceededError(
     77         "The input length exceeds the context length of the model."
     78     )
     80 allowed_tokens = get_allowed_tokens(fsms, fsm_states)
---> 81 biased_logits = bias_logits(logits, allowed_tokens)
     82 next_token_ids, ancestors, sequence_weights = sampler(
     83     biased_logits, sequence_weights, rng
     84 )
...
    304 for i, ids in enumerate(allowed_token_ids):
--> 305     biased_logits[i, ids] = logits[i, ids]
    306 return biased_logits

IndexError: index 54732 is out of bounds for dimension 0 with size 32000

Steps/code to reproduce the bug:

Copy this example from the readme into a notebook.

Expected result:

no crash

Error message:

see above

Outlines/Python version information:

Python 3.10.14

Library versions (let me know if you need more, there are many others that are non-ML that just add noise)

outlines                  0.0.40
torch                     2.3.0+cu118
transformers              4.40.1

Context for the issue:

No response

[lapp0]

Unfortunately I cannot reproduce. The only difference is I'm running torch==2.3.0 with torch.version.cuda == 12.1

My output is

Character(name='Alex Chen', age=28, armor=<Armor.leather: 'leather'>, weapon=<Weapon.bow: 'bow'>, strength=65)
Character(name='Boo', age=27, armor=<Armor.chainmail: 'chainmail'>, weapon=<Weapon.bow: 'bow'>, strength=70)

Could you please try the following:

• 1. delete the model folder (it's somewhere in ~/.cache/huggingface/hub) so it's re-downloaded
• 2. delete your outlines cache via import outlines.caching; outlines.caching.clear_cache()
• 3. pip install --upgrade outlines

If that doesn't work, I'd like to know your full pip freeze, nvcc --version, nvidia-smi, and the model of your device please.

[rlouf]

Closing for now, as it is not reproducible. Feel free to re-open if the error persists.