How adding a guidance affects model performance ?

[Vi-cs]

Hi Rémy,

Thanks for your library, it's a pleasure to go through the code to understand how it works.

I try to use it to guide mymodel to produce YAML. With "regex" it is pretty straightforward.
However, I don't fully understand a particular behavior:

Context :

• my model is already finetuned to produce a YAML, but sometimes outputs a string that is not strictly a YAML.
• that's why I want to help the model with guidance.

Now :

• without guidance, the model produces a YAMLlike :
  'data_type: TRANSACTION\namount: 6543.45\nand_so_on: ...'
  -with guidance, the model produces a correct YAML but sometimes infers a wrong number, like :
  'data_type: TRANSACTION\namount: 26543.45\nand_so_on: ...'

any idea why this happens?

To my understanding, the final selection of the next token is performed by the vectorized_random_choice function.
Can you explain why adding some randomness here? Why not "simply" return the token with the highest probability?

Edit :
I did not fully understand vectorized_random_choice, but replacing the current implementation with a torch.argmax returns the same token. And I still have the same issue

Mille mercis !

[Vi-cs]

closed