[Misc]: How to use guided decoding and regex as well?

[debraj135]

Anything you want to discuss about vllm.

Here is my guided_json

guided_json = {"type": "object", "properties": { "age": {"type": "string", "pattern": r"^Fifty"}, "height": {"type": "string", "pattern": r"^five"}}}

I want to enforce the initial string in each of the keys, is this supported? And how can I go about doing this?

[AswanthManoj]

I also have the same question on how to use do contrained decoding with regex string as the input or maybe a pydantic object schema json

[clj55]

I have the same question too!

I tried doing this:
from outlines import models, generate
model = models.vllm(model="TheBloke/Mistral-7B-OpenOrca-AWQ", device=device, quantization="awq", dtype="half", tensor_parallel_size=2)
sampling_params = SamplingParams(temperature=0.0, top_p=0.95, max_tokens=300)
generator = generate.json(model, schema)
answer = generator(prompt, sampling_params = sampling_params)

But got the error "AttributeError: module 'outlines.models' has no attribute 'vllm'"

[debraj135]

Does anyone have an answer to this?

[clj55]

When I give the JSON output in the example and add a line like "Only use the fields given in the example" in the prompt, the output is constrained to only the keys given in the example. I didn't the outlines library at all. Hope this helps!