Feat/json schemas for llm output

[srijanpatel]

This pull request includes several changes to enhance the handling of JSON schemas and response formats in the LLM-related modules. The most important changes include adding support for JSON schema validation, improving error handling, and refactoring some utility functions.

Enhancements to JSON schema handling:

• backend/app/nodes/llm/_providers.py: Added response_format to OllamaOptions to specify the format of the response.
• backend/app/nodes/llm/_utils.py: Added parameters output_json_schema and output_schema to generate_text and implemented logic to handle these parameters, including converting simple schemas to JSON schemas. [1] [2]
• backend/app/nodes/llm/generative/best_of_n.py: Updated setup_subworkflow to handle output_json_schema and convert it to a simple schema if needed. [1] [2]
• backend/app/nodes/llm/single_llm_call.py: Added output_json_schema to SingleLLMCallNodeConfig and updated the run method to use this schema for response validation. [1] [2]

Error handling improvements:

• backend/app/execution/workflow_executor.py: Added a try-except block to handle ValidationError when processing precomputed outputs, logging a warning message if validation fails.

Refactoring and utility functions:

• backend/app/nodes/llm/_utils.py: Refactored setup_azure_configuration function and moved json_schema_to_model import to a more appropriate location. [1] [2]
• backend/app/utils/pydantic_utils.py: Added utility function convert_output_schema_to_json_schema to convert simple output schemas to JSON schemas.

[srijanpatel]

@ellipsis-dev review this

[ellipsis-dev]

👍 Looks good to me! Reviewed everything up to 0818003 in 2 minutes and 35 seconds

More details

• Looked at 796 lines of code in 11 files
• Skipped 2 files when reviewing.
• Skipped posting 7 drafted comments based on config settings.

1. backend/app/execution/workflow_executor.py:228

• Draft comment:
  Consider using a logging framework instead of print for better control over log levels and outputs.
• Reason this comment was not posted:
  Confidence changes required: 50%
  The code in workflow_executor.py uses print statements for logging, which is not a best practice for production code. It should use a logging framework instead.

2. backend/app/nodes/llm/_providers.py:39

• Draft comment:
  Ensure that all references to _setup_azure_configuration are updated to setup_azure_configuration to avoid runtime errors.
• Reason this comment was not posted:
  Comment did not seem useful.

3. backend/app/nodes/llm/_utils.py:482

• Draft comment:
  Consider refactoring the logic for handling output_json_schema and output_schema to make it clearer and more robust. Ensure that the default schema is applied correctly when both are None.
• Reason this comment was not posted:
  Confidence changes required: 50%
  The generate_text function in _utils.py has a potential issue with the output_json_schema and output_schema handling. If both are None, it defaults to a simple schema, but this logic could be clearer and more robust.

4. backend/app/nodes/llm/generative/best_of_n.py:126

• Draft comment:
  Ensure that json_schema_to_simple_schema correctly handles all necessary cases when converting output_json_schema to a simple schema.
• Reason this comment was not posted:
  Confidence changes required: 50%
  In best_of_n.py, the output_json_schema is being converted to a simple schema using json_schema_to_simple_schema. This conversion should be verified to ensure it handles all necessary cases.

5. backend/app/nodes/llm/single_llm_call.py:72

• Draft comment:
  Ensure that json_schema_to_model correctly handles all necessary cases when converting output_json_schema to a Pydantic model.
• Reason this comment was not posted:
  Confidence changes required: 50%
  In single_llm_call.py, the output_json_schema is being converted to a Pydantic model using json_schema_to_model. This conversion should be verified to ensure it handles all necessary cases.

6. backend/app/utils/pydantic_utils.py:129

• Draft comment:
  Consider logging the ValueError in json_schema_to_pydantic_type for better debugging and traceability.
• Reason this comment was not posted:
  Confidence changes required: 30%
  In pydantic_utils.py, the function json_schema_to_pydantic_type raises a ValueError for unsupported types. It might be beneficial to log this error for better debugging.

7. frontend/src/components/nodes/nodeSidebar/NodeSidebar.tsx:89

• Draft comment:
  Ensure that the JSON cleaning process in extractSchemaFromJsonSchema is robust and handles edge cases effectively.
• Reason this comment was not posted:
  Confidence changes required: 40%
  In NodeSidebar.tsx, the extractSchemaFromJsonSchema function attempts to clean and parse JSON. Ensure that this cleaning process is robust and handles edge cases.

Workflow ID: wflow_KgEOymDNSk4egYDZ

---

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.