Suggestion / Question for Refactoring to Enable Multiple Generation

[lapp0]

Continuation from #416 (comment)

Efficient Caching

In the existing outlines codebase, the RegexLogitsProcessor is called with seq_id which corresponds to a distinct FSM state

outlines/outlines/serve/vllm.py

Lines 68 to 70 in 0355ab4

	self.fsm_state[seq_id] = self.fsm.next_state(
	self.fsm_state[seq_id], last_token
	)

_apply_logits_processors is patched in outlines to pass seq_id.

This works, but if we used tuple(input_ids) as a distinct key, we would be able to prevent repeated work for a given (fsm, input_ids)

Additionally, we would no longer need to patch _apply_logits_processors which would prevent problems if vLLMs implementation of the function changed.

Further, caching FSM state by input_ids may allow us to efficiently manage the FSM internally within the logits processor as described in the next section.

Multiple Sampling By Eliminating Outlines Generation Logic

I'm not fluent in outlines codebase, so if you could point out what I'm missing I'd appreciate it.

My understanding is that outlines allows vLLMs AsyncEngine to handle all KV management and simply provides a logits processor

outlines/outlines/serve/serve.py

Line 80 in 0355ab4

results_generator = engine.generate(prompt, sampling_params, request_id) # type: ignore

This is a clean paradigm and it makes sense to me that outlines would leave KV cache management to transformers as well, and simply provide an efficient logits processor.

Regarding outlines/outlines/generate/api.py, why wouldn't we be able to simplify SequenceGenerator so it allows transformers to do the bulk of the work in managing generation? It appears that we would be able to provide transformers with a logits processor which retrieves the cached FSM state as fn(generation_token_ids) rather than tracking and manipulating FSMState within SequenceGenerator.

[rlouf]

This works, but if we used tuple(input_ids) as a distinct key, we would be able to prevent repeated work for a given (fsm, input_ids)

This is a good idea and would simplify the code substantially. Do you want to open a PR?

[lapp0]

I should be able to do this sometime soon.

[rlouf]

This is a clean paradigm and it makes sense to me that outlines would leave KV cache management to transformers as well, and simply provide an efficient logits processor.

We have features on the roadmap that require us to do that, like sampling algorithms not available in any library or better cache management.

[lapp0]

Could you link me the roadmap? I can't find anything related to a roadmap in search. (Or ping me when you write one up if you have yet to).

[lapp0]

Also I'm curious about the design philosophy. Considering KV Cache management is an integral function of inference libraries, where does the responsibilities of the inference library (transformers, vLLM) end and outlines begin as outlines is developed further?

[rlouf]

I haven't had the time to write a roadmap, but the design philosophy is simple:

1. Guided generation
2. Better sampling method (coming soon)

Anything that is not implemented at the level of inference libraries we will implement in Outlines for now. For instance, for some sampling methods we will need to improve the way cache is currently managed. Now, if some of that can be implemented upstream later all the better!

The original goal was to only support transformers and openai (for comparison reasons), but the method has become popular enough for people to request other integrations which we happily support.

[rlouf]

We now have a roadmap.

[lapp0]

Extremely exciting stuff! With correct / efficient grammar-constrained generation, token healing, function calling, and advanced sampling methods Outlines will be a standard in the LLM space.

Thanks for sharing.