Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Dynamically exclude tokens from RAG output #92

Open
mawandm opened this issue May 24, 2024 · 0 comments
Open

[Feature] Dynamically exclude tokens from RAG output #92

mawandm opened this issue May 24, 2024 · 0 comments
Labels
API Backend API enhancement New feature or request rag Rag Engine

Comments

@mawandm
Copy link
Contributor

mawandm commented May 24, 2024

Description
We want the ability to exclude tokens based on an exclusion list combined with RBAC.

Detail
When handling some kind of information, some tokens need to be excluded. Two scenarios where this is necessary include;

  1. Handling sensitive PII information. In this scenario it may be permitted for one user to view the sensitive information such as a customer service representative while another user may not be permitted to view the same sensitive information. In order to achieve this, an exclusion list must be created and associated to the roles of the respective user.
  2. Filtering offensive language. This scenario can be useful when we want to exclude some perceived offensive tokens for example when there age restrictions.

Acceptance Criteria
This is an exploratory task so this list may be changed.

  1. Add an exclusion list data source. This can include;
    1. SQL database
    2. REST API
    3. File upload
  2. An exclusion list can be associated to a role.
  3. Exclusion rules are applied to user dynamically.
@mawandm mawandm added enhancement New feature or request rag Rag Engine API Backend API labels May 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
API Backend API enhancement New feature or request rag Rag Engine
Projects
None yet
Development

No branches or pull requests

1 participant