Generate Test Set: extract response attr if app returns object with that attr

[joshreini1]

Enables generate_test_set for Llama-Index apps that return response object rather than str

---

	🚀 This PR description was created by Ellipsis for commit f008169.

Summary:

This PR modifies the GenerateTestSet class in trulens_eval/trulens_eval/generate_test_set.py to handle Llama-Index apps that return a response object, extracting the response attribute if it exists.

Key points:

• Modified _generate_themes, _format_themes, _generate_test_prompts, and _format_test_prompts methods in GenerateTestSet class in trulens_eval/trulens_eval/generate_test_set.py.
• Added code to extract response attribute from the object returned by app_callable.
• Enables generate_test_set to work with Llama-Index apps returning response objects.

---

Generated with ❤️ by ellipsis.dev

[ellipsis-dev]

👍 Looks good to me!

• Reviewed the entire pull request up to b22ca94
• Looked at 64 lines of code in 1 files
• Took 2 minutes and 37 seconds to review

More info

• Skipped 0 files when reviewing.
• Skipped posting 4 additional comments because they didn't meet confidence threshold of 50%.

1. trulens_eval/trulens_eval/generate_test_set.py:52:

• Assessed confidence : 50%
• Comment:
  Ensure that the 'response' attribute of the object returned by the app_callable is always a string, as the rest of the code assumes this. If it's not guaranteed to be a string, this could lead to errors later on.
• Reasoning:
  The PR author has added checks to see if the response from the app_callable has a 'response' attribute and if it does, it uses that for further processing. This is done to handle cases where the app_callable returns an object with a 'response' attribute instead of a string. This seems like a reasonable change, but it's important to ensure that the 'response' attribute is always a string, as the rest of the code assumes this. If it's not guaranteed to be a string, this could lead to errors later on.

2. trulens_eval/trulens_eval/generate_test_set.py:110:

• Assessed confidence : 50%
• Comment:
  Ensure that the 'response' attribute of the object returned by the app_callable is always a string, as the rest of the code assumes this. If it's not guaranteed to be a string, this could lead to errors later on.
• Reasoning:
  The PR author has added checks to see if the response from the app_callable has a 'response' attribute and if it does, it uses that for further processing. This is done to handle cases where the app_callable returns an object with a 'response' attribute instead of a string. This seems like a reasonable change, but it's important to ensure that the 'response' attribute is always a string, as the rest of the code assumes this. If it's not guaranteed to be a string, this could lead to errors later on.

3. trulens_eval/trulens_eval/generate_test_set.py:130:

• Assessed confidence : 50%
• Comment:
  Ensure that the 'response' attribute of the object returned by the app_callable is always a string, as the rest of the code assumes this. If it's not guaranteed to be a string, this could lead to errors later on.
• Reasoning:
  The PR author has added checks to see if the response from the app_callable has a 'response' attribute and if it does, it uses that for further processing. This is done to handle cases where the app_callable returns an object with a 'response' attribute instead of a string. This seems like a reasonable change, but it's important to ensure that the 'response' attribute is always a string, as the rest of the code assumes this. If it's not guaranteed to be a string, this could lead to errors later on.

4. trulens_eval/trulens_eval/generate_test_set.py:75:

• Assessed confidence : 66%
• Grade: 40%
• Comment:
  Consider logging the original error message in addition to the custom error message. This will provide more context in case of a SyntaxError and help with debugging.

except SyntaxError as e:
    logger.error(f'Error while parsing themes string: {e}')
    raise ValueError(f'Failed to parse themes string: {test_categories}') from e

• Reasoning:
  The PR author has added a try-except block around the literal_eval call to handle cases where the string cannot be parsed as a Python literal. This is a good practice as it improves the robustness of the code. However, it would be better to also log the original error message to help with debugging in case of a SyntaxError.

Workflow ID: wflow_S8bRqr9eLHQNkGDk

---

Not what you expected? You can customize the content of the reviews using rules. Learn more here.

[ellipsis-dev]

No problems found on commit f008169.

---

Not what you expected? You can customize the content of the reviews using rules. Learn more here.