How does it handle unexpected answers from LLMs under the hood?

[gornostal]

Hi
I'd appreciate if someone would explain on the high level the answer to the question above! 🙏

I'm guessing that if outlines detects that output type/schema/etc. doesn't match the desired schema (due to model hallucination or whatever), it retries the LLM query several times until it gets what it expects. Otherwise it'll fail after the number attempts. Is that so?

[brandonwillard]

We explain the approach in our paper, and it does not involve retries.

Simply put, we construct regular expressions for the JSON specs, convert those to finite state machines, and use an index constructed from the states of the finite state machines to get the LLM tokens that could come next. The indexing is akin to standard string search algorithms (e.g. Aho-Corasick) and regex-based indexing approaches.

The approach was designed to be adaptable to existing regular expressions parsing/matching approaches and software, but we're currently using a simple DFA library called interegular (based on greenery). We plan to extract the necessary state information from existing regex libraries so that we can leverage their performance and advanced non-regular capabilities as well.

[gornostal]

I've been reading the paper and exploring the code. It's fascinating how genius the idea is!

I wonder, what's stopping from adding a regex based completions for OpenAI models into Outlines so it can be used like this?

model = models.text_completion.openai("text-davinci-002")
prompt = "Is 1+1=2? "
guided = generate.regex(model, r"\s*([Yy]es|[Nn]o|[Nn]ever|[Aa]lways)", max_tokens=30)(
    prompt
)

Is it the limitations of the OpenAI API?

[rlouf]

Yes it is a limitation of OpenAI, we can only mask up to 300 tokens at every step.

Now, since we know in advance how many tokens need to be masked at each step there may be situations in which we could do it. I doubt it's going to be many cases though, GPT4's vocab is 100k long.

[gornostal]

Awesome! Thanks for the prompt reply!

…

On Sat, Aug 19, 2023, 20:42 Brandon T. Willard ***@***.***> wrote: We explain the approach in our paper <https://arxiv.org/abs/2307.09702>. — Reply to this email directly, view it on GitHub <#250 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAJFS32KOQM26VOPML5SFNDXWD3ILANCNFSM6AAAAAA3WWCNMY> . You are receiving this because you authored the thread.Message ID: ***@***.*** com>