Context filtering guardrails #1192
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
dosubot
bot
added
the
size:XS
dosubot
bot
added
size:L
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
![ellipsis-dev[bot]](https://avatars.githubusercontent.com/in/64358?s=60&v=4){kind=small}
There was a problem hiding this comment.
❌ Changes requested. Reviewed everything up to dca981b in 1 minute and 10 seconds
More details
- Looked at
31lines of code in1files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_bIRnghBTPl1blAef
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
There was a problem hiding this comment.
❌ Changes requested. Incremental review on 3ec0c72 in 1 minute and 52 seconds
More details
- Looked at
13lines of code in1files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_A0LvWTd6yrQzYdbX
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on 9090135 in 1 minute and 6 seconds
More details
- Looked at
13lines of code in1files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. trulens_eval/trulens_eval/tru_llama.py:83
- Draft comment:
Ensure that all references toWithFeedbackFilterNodesthroughout the codebase are updated to reflect the new import pathtrulens_eval.guardrails.llamato avoid import errors. - Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 50%.
Workflow ID: wflow_w6b5Npw3qgwPOZ0U
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
|
Skipped PR review on 777240f because no changed files had a supported extension. If you think this was in error, please contact us and we'll fix it right away. Generated with ❤️ by ellipsis.dev |
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on 8b6abf3 in 33 seconds
More details
- Looked at
21lines of code in1files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. trulens_eval/trulens_eval/utils/llama.py:9
- Draft comment:
The import forListwas removed. Please ensure thatListis not used elsewhere in this file or in any related files where this module's objects are utilized. IfListis indeed not used, then this change is appropriate. - Reason this comment was not posted:
Confidence changes required:50%
The import statement for 'List' was removed, but it's not clear if 'List' was actually unused. The code snippet provided does not show any usage of 'List', but it's possible that other parts of the file or related files might still require it. I need to check if 'List' is used elsewhere in the file or if its removal is justified.
Workflow ID: wflow_M3l4SiH2YvcTpctr
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
❌ Changes requested. Incremental review on 1110aea in 1 minute and 25 seconds
More details
- Looked at
98lines of code in1files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_5TAnJJWcvNzS8EH1
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on d525b35 in 3 minutes and 22 seconds
More details
- Looked at
79lines of code in2files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_wkgvRbTyfTiZBgyK
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on b6eadac in 1 minute and 42 seconds
More details
- Looked at
31lines of code in1files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. mkdocs.yml:29
- Draft comment:
The PR description mentions significant updates related to context filtering and display utilities, but the diff provided only shows changes to the mkdocs.yml file, which is unrelated to the described features. Please ensure that all relevant changes are included in the PR or update the PR description to accurately reflect the changes made. - Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 50%.
Workflow ID: wflow_vyvQTpHWr7XzZlGN
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
👍 Looks good to me! Incremental review on 7d77dc2 in 1 minute and 47 seconds
More details
- Looked at
13lines of code in1files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. docs/trulens_eval/guardrails/index.md:23
- Draft comment:
The PR description suggests updates related to context filtering guardrails, but the diff only shows a minor formatting change. Please ensure that all intended updates are included in the PR. - Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 50%.
Workflow ID: wflow_hZ84EDCrX9GRxGKZ
You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
❌ Changes requested. Incremental review on 310d0b1 in 1 minute and 2 seconds
More details
- Looked at
14lines of code in1files - Skipped
0files when reviewing. - Skipped posting
0drafted comments based on config settings.
Workflow ID: wflow_YhOhkgJlSkSWzP5t
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
There was a problem hiding this comment.
❌ Changes requested. Incremental review on 9036798 in 1 minute and 52 seconds
More details
- Looked at
133lines of code in6files - Skipped
0files when reviewing. - Skipped posting
1drafted comments based on config settings.
1. trulens_eval/trulens_eval/guardrails/langchain.py:83
- Draft comment:
The lambda function inside the ThreadPoolExecutor is capturing the loop variabledoc. This can lead to all lambda functions capturing the same reference, which is the last item in the loop. To fix this, you should use a default argument in the lambda to capture the current value ofdocfor each iteration.
lambda doc=doc: self.feedback(query, doc.page_content)
- Reason this comment was not posted:
Confidence of 0% on close inspection, compared to threshold of 50%.
Workflow ID: wflow_RqCHOpNAR2B4hB4A
Want Ellipsis to fix these issues? Tag @ellipsis-dev in a comment. You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.
sfc-gh-pmardziel
requested changes
Jun 20, 2024
sfc-gh-chu
reviewed
Jun 21, 2024
sfc-gh-chu
approved these changes
Jun 21, 2024
1 task
sfc-gh-pmardziel
approved these changes
Jun 21, 2024
sfc-gh-jreini
added a commit
that referenced
this pull request
Jun 21, 2024
* fix select_context to call return individual items, not collected list * display utilities, context filtering, oh my * Update trulens_eval/trulens_eval/tru_chain.py Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * undo bad ellipsis change * change moved import * update llama quickstart * get llama working * llama updates * add docstring examples * guardrails docs page * allow tabbed content setting in mkdocs yml * formatting * api ref links * api ref files for guardrails * api ref to guardrails in mkdocs * update in trubot * fix typo * add missing init * api ref updates * docstring updates * docstring updates * base context filter for custom apps * guardrails in quickstart * make display util more efficient * switch openai model to std * drop unneccessary import * docs for base guardrails * remove unneccessary name * drop unneccessary args * using sentence * fix select_context * langchain quickstart updates * fix link * langchain feedback type error handling * llama feedback type error handling * base feedback type error handling * pass context as default argument to lambda function in base Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * revert ellipsis change * fix test failure * move docstrings, simplify * accept directionality from feedback * set wraper.__dict__ too * filter on demand * format, remove unused imports * format * format * assert llama-index installed * assert langchain installed * test llam aguardrails optional mod * format * copy signature instead of dict * add note --------- Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
sfc-gh-jreini
added a commit
that referenced
this pull request
Jun 21, 2024
* fix select_context to call return individual items, not collected list * display utilities, context filtering, oh my * Update trulens_eval/trulens_eval/tru_chain.py * undo bad ellipsis change * change moved import * update llama quickstart * get llama working * llama updates * add docstring examples * guardrails docs page * allow tabbed content setting in mkdocs yml * formatting * api ref links * api ref files for guardrails * api ref to guardrails in mkdocs * update in trubot * fix typo * add missing init * api ref updates * docstring updates * docstring updates * base context filter for custom apps * guardrails in quickstart * make display util more efficient * switch openai model to std * drop unneccessary import * docs for base guardrails * remove unneccessary name * drop unneccessary args * using sentence * fix select_context * langchain quickstart updates * fix link * langchain feedback type error handling * llama feedback type error handling * base feedback type error handling * pass context as default argument to lambda function in base * revert ellipsis change * fix test failure * move docstrings, simplify * accept directionality from feedback * set wraper.__dict__ too * filter on demand * format, remove unused imports * format * format * assert llama-index installed * assert langchain installed * test llam aguardrails optional mod * format * copy signature instead of dict * add note --------- Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Merged
sfc-gh-dhuang
pushed a commit
that referenced
this pull request
Jun 28, 2024
* fix select_context to call return individual items, not collected list * display utilities, context filtering, oh my * Update trulens_eval/trulens_eval/tru_chain.py Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * undo bad ellipsis change * change moved import * update llama quickstart * get llama working * llama updates * add docstring examples * guardrails docs page * allow tabbed content setting in mkdocs yml * formatting * api ref links * api ref files for guardrails * api ref to guardrails in mkdocs * update in trubot * fix typo * add missing init * api ref updates * docstring updates * docstring updates * base context filter for custom apps * guardrails in quickstart * make display util more efficient * switch openai model to std * drop unneccessary import * docs for base guardrails * remove unneccessary name * drop unneccessary args * using sentence * fix select_context * langchain quickstart updates * fix link * langchain feedback type error handling * llama feedback type error handling * base feedback type error handling * pass context as default argument to lambda function in base Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * revert ellipsis change * fix test failure * move docstrings, simplify * accept directionality from feedback * set wraper.__dict__ too * filter on demand * format, remove unused imports * format * format * assert llama-index installed * assert langchain installed * test llam aguardrails optional mod * format * copy signature instead of dict * add note --------- Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
sfc-gh-dhuang
pushed a commit
that referenced
this pull request
Jul 1, 2024
* fix select_context to call return individual items, not collected list * display utilities, context filtering, oh my * Update trulens_eval/trulens_eval/tru_chain.py Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * undo bad ellipsis change * change moved import * update llama quickstart * get llama working * llama updates * add docstring examples * guardrails docs page * allow tabbed content setting in mkdocs yml * formatting * api ref links * api ref files for guardrails * api ref to guardrails in mkdocs * update in trubot * fix typo * add missing init * api ref updates * docstring updates * docstring updates * base context filter for custom apps * guardrails in quickstart * make display util more efficient * switch openai model to std * drop unneccessary import * docs for base guardrails * remove unneccessary name * drop unneccessary args * using sentence * fix select_context * langchain quickstart updates * fix link * langchain feedback type error handling * llama feedback type error handling * base feedback type error handling * pass context as default argument to lambda function in base Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * revert ellipsis change * fix test failure * move docstrings, simplify * accept directionality from feedback * set wraper.__dict__ too * filter on demand * format, remove unused imports * format * format * assert llama-index installed * assert langchain installed * test llam aguardrails optional mod * format * copy signature instead of dict * add note --------- Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
sfc-gh-chu
pushed a commit
that referenced
this pull request
Sep 25, 2024
* fix select_context to call return individual items, not collected list * display utilities, context filtering, oh my * Update trulens_eval/trulens_eval/tru_chain.py Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * undo bad ellipsis change * change moved import * update llama quickstart * get llama working * llama updates * add docstring examples * guardrails docs page * allow tabbed content setting in mkdocs yml * formatting * api ref links * api ref files for guardrails * api ref to guardrails in mkdocs * update in trubot * fix typo * add missing init * api ref updates * docstring updates * docstring updates * base context filter for custom apps * guardrails in quickstart * make display util more efficient * switch openai model to std * drop unneccessary import * docs for base guardrails * remove unneccessary name * drop unneccessary args * using sentence * fix select_context * langchain quickstart updates * fix link * langchain feedback type error handling * llama feedback type error handling * base feedback type error handling * pass context as default argument to lambda function in base Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * revert ellipsis change * fix test failure * move docstrings, simplify * accept directionality from feedback * set wraper.__dict__ too * filter on demand * format, remove unused imports * format * format * assert llama-index installed * assert langchain installed * test llam aguardrails optional mod * format * copy signature instead of dict * add note --------- Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Items to add to release announcement:
select_contextfor Langchain2024-06-13.12-33-13.mp4
Summary:
Enhancements in context filtering and display utilities, with significant improvements in the
select_contextmethod across different classes, added checks for method availability in retrievers, and updated import paths and documentation for better organization.Key points:
select_contextfor Langchainselect_contextmethod across different classestrulens_eval/trulens_eval/utils/display.pyGenerated with ❤️ by ellipsis.dev