You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Running image ghcr.io/predibase/lorax:0.9.0 in kubernetes.
Information
Docker
The CLI directly
Tasks
An officially supported command
My own modifications
Reproduction
Using the /generate endpoint.
When reading the swagger at: REST API, the property schema is defined as "schema": "{\"type\": \"string\", \"title\": \"response\"}",
This makes me believe that this parameter would be responsible for enforcing structured output, as described in Structured Output. However, when setting it to any schema (using pydantics .model_json_schema() or by passing a json schema directly) the schema is not enforced on the generated text. This is the only parameter in the swagger for this endpoints that I would relate to structured output.
However, if you pass the parameter "response_format", structured output is enabled. An example would be
This is not present in the swagger for /generate or any other documentation that I could find for the endpoint. I only tested it because of the example in Structured Output, where a Python Client is used.
Expected behavior
I expected the documentation to more clear on how to enable structured output in the /generate endpoint.
The text was updated successfully, but these errors were encountered:
System Info
Running image ghcr.io/predibase/lorax:0.9.0 in kubernetes.
Information
Tasks
Reproduction
Using the
/generate
endpoint.When reading the swagger at: REST API, the property
schema
is defined as"schema": "{\"type\": \"string\", \"title\": \"response\"}",
This makes me believe that this parameter would be responsible for enforcing structured output, as described in Structured Output. However, when setting it to any schema (using pydantics
.model_json_schema()
or by passing a json schema directly) the schema is not enforced on the generated text. This is the only parameter in the swagger for this endpoints that I would relate to structured output.However, if you pass the parameter
"response_format"
, structured output is enabled. An example would beThis is not present in the swagger for
/generate
or any other documentation that I could find for the endpoint. I only tested it because of the example in Structured Output, where a Python Client is used.Expected behavior
I expected the documentation to more clear on how to enable structured output in the
/generate
endpoint.The text was updated successfully, but these errors were encountered: