[Request] 8.15 Security Assistant Custom Knowledge Bases #5337
Assignees
Labels
Effort: X-Large
Issues that require several days or weeks substantial planning, research, writing, and testing
Team: GenAI
Generative AI team
Comments
benironside
added
Team: GenAI
8 tasks
spong
added a commit
to elastic/kibana
that referenced
this issue
Jun 14, 2024
## Summary Migrates our existing RAG pipeline to use LangGraph, and adds tools for Knowledge Base retrieval/storage. When the `assistantKnowledgeBaseByDefault` FF is enabled, a new branch, `callAssistantGraph()`, is taken in `postActionsConnectorExecuteRoute` that exercises the LangGraph implementation. This is a drop-in replacement for the existing `callAgentExecutor()` in effort to keep adoption as clean and easy as possible. The new control flow is as follows: `postActionsConnectorExecuteRoute` -> `callAssistantGraph()` -> `getDefaultAssistantGraph()` -> `isStreamingEnabled ? streamGraph() : invokeGraph()` Graph creation is isolated to `getDefaultAssistantGraph()`, and execution (streaming or not) has been extracted to `streamGraph()` and `invokeGraph()` respectively. Note: Streaming currently only works with `ChatOpenAI` models, but `SimpleChatModelStreaming` was de-risked and just need to discuss potential solutions with @stephmilovic. See [comment here](https://github.com/elastic/kibana/pull/184554/files#diff-ad87c5621b231a40810419fc1e56f28aeb4f8328e125e465dfe95ae0e1c305b8R97-R98). #### DefaultAssistantGraph To start with a predictable and piecemeal migration, our existing `agentExecutor` pipeline has been recreated in LangGraph. It consists of a single agent node, either `OpenAIFunctionsAgent`, or `StructuredChatAgent` (depending on the backing LLM), a tool executing node, and a conditional edge that routes between the two nodes until there are no more function calls chosen by the agent. This varies from our initial implementation in that multiple tool calls are now supported, so a user could ask about their alerts AND retrieve additional knowledge base information in the same response. > [!NOTE] > While `chat_history` has been plumbed into the graph, after discussing with @YulNaumenko we decided to wait to plumb the rest of persistence into the graph until #184485 is merged. I had already plumbed through the `chatTitleGeneration` node ([here](https://github.com/elastic/kibana/pull/184554/files#diff-26038489e9a3f1a14c5ea2ac2954671973d833349ef3ffaddcf9b29ce9e2b96eR33)), and so will just need to include initial conversation creation and append/update operations. #### Knowledge History & KB Tools Knowledge History is now always added in the initial prompt for any KB documents marked as `required`, and two new tools were added for creating and recalling KB entries from within the conversation, `KnowledgeBaseWriteTool` and `KnowledgeBaseRetrievalTool` respectively. All three methods of storing and retrieving KB content use the `kbDataClient` for access, and scopes all requests to the authenticatedUser that made the initial request. Additional Notes: * LangChain dependencies have been updated, and a new dependency on `LangGraph` has been added. ### Checklist Delete any items that are not applicable to this PR. - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials * Feature currently behind a FF, documentation to be added once feature is complete. Tracked in elastic/security-docs#5337. - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios * Test coverage in progress... --------- Co-authored-by: Patryk Kopycinski <contact@patrykkopycinski.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
spong
added a commit
to elastic/kibana
that referenced
this issue
Jun 19, 2024
…ng Knowledge Base Entries (#184974) ## Summary This PR adds client hooks and basic REST API's for accessing and mutating Knowledge Base Entries. This is in support of @angorayc building out the new Knowledge Base settings interface. Change set includes: - [X] Refactors existing KB client hooks from `x-pack/packages/kbn-elastic-assistant/impl/knowledge_base` to be co-located next to the API methods where we put all our other hooks: `x-pack/packages/kbn-elastic-assistant/impl/assistant/api/knowledge_base` - [X] Refactors existing KB API calls and associated tests out of `kbn-elastic-assistant/impl/assistant/api/index.tsx` and into `x-pack/packages/kbn-elastic-assistant/impl/assistant/api/knowledge_base/api.tsx` - [X] Adds new `find_knowledge_base_entries_route.schema.yaml` OAS for the supporting `/internal/elastic_assistant/knowledge_base/entries/_find` route - [X] Refactors `SortOrder` out of existing OAS's into the shared `schemas/common_attributes.schema.yaml` ### Client Hooks & Routes Adds new `useKnowledgeBaseEntries()` hook and corresponding `/knowledge_base/entries/_find` route for returning paginated KB Entries to populate the KB table in settings. E.g. ``` ts const { assistantFeatures: { assistantKnowledgeBaseByDefault: enableKnowledgeBaseByDefault }, http, toasts, } = useAssistantContext(); const { data: kbEntries, isLoading: isLoadingEntries } = useKnowledgeBaseEntries({ http }); ``` ###### Sample Response ``` json { "perPage": 20, "page": 1, "total": 145, "data": [ { "timestamp": "2024-06-05T21:19:56.482Z", "id": "CtBF6o8BSQy1Bdxt2FHz", "createdAt": "2024-06-05T21:19:56.482Z", "createdBy": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0", "updatedAt": "2024-06-05T21:19:56.482Z", "updatedBy": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0", "users": [ { "id": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0", "name": "elastic" } ], "metadata": { "kbResource": "security_labs", "source": "/Users/garrettspong/dev/kibana-main/x-pack/plugins/elastic_assistant/server/knowledge_base/security_labs/2022_elastic_global_threat_report_announcement.mdx", "required": false }, "namespace": "default", "text": "[Source Content Here]", "vector": { "modelId": ".elser_model_2", "tokens": { "2": 0.06595266, ... } } }, ... ] } ``` Response is the full newly created `entry`. Same format for the entry as above in the `_find` API, and the `KnowledgeBaseEntries` cache is invalidated. Adds new `useCreateKnowledgeBaseEntry()` hook and corresponding `/knowledge_base/entries` route for creating new KB Entries ``` ts const entry: KnowledgeBaseEntryCreateProps = { metadata: { kbResource: 'user', required: true, source: 'user', }, text: 'Useful information about the user', }; const { mutate: createEntry, isLoading: isCreatingEntry } = useCreateKnowledgeBaseEntry({ http, }); await createEntry(entry); ``` Adds new `useDeleteKnowledgeBaseEntries()` hook and corresponding `/knowledge_base/entries/_bulk_action` route for deleting existing KB Entries. I left a TODO to plumb through `delete_by_query` so we can add a filter bar to the table. Need to confirm if we can do pagination with similarity search as well. ``` ts const { mutate: deleteEntries, isLoading: isDeletingEntries } = useDeleteKnowledgeBaseEntries({ http, }); await deleteEntries({ ids: ['YOE_CZABSQy1BdxtAGbs'] }) ``` See `KnowledgeBaseEntryBulkCrudActionResponse` for response formats. `KnowledgeBaseEntries` cache is invalidated upon delete. ### Checklist Delete any items that are not applicable to this PR. - [ ] ~Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)~ - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials * Feature currently behind feature flag. Documentation to be added before flag is removed. Tracked in elastic/security-docs#5337 - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] API tests will need to be rounded out as we finalize functionality behind the feature flag --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
8 tasks
seanrathier
pushed a commit
to seanrathier/kibana
that referenced
this issue
Jun 21, 2024
…ng Knowledge Base Entries (elastic#184974) ## Summary This PR adds client hooks and basic REST API's for accessing and mutating Knowledge Base Entries. This is in support of @angorayc building out the new Knowledge Base settings interface. Change set includes: - [X] Refactors existing KB client hooks from `x-pack/packages/kbn-elastic-assistant/impl/knowledge_base` to be co-located next to the API methods where we put all our other hooks: `x-pack/packages/kbn-elastic-assistant/impl/assistant/api/knowledge_base` - [X] Refactors existing KB API calls and associated tests out of `kbn-elastic-assistant/impl/assistant/api/index.tsx` and into `x-pack/packages/kbn-elastic-assistant/impl/assistant/api/knowledge_base/api.tsx` - [X] Adds new `find_knowledge_base_entries_route.schema.yaml` OAS for the supporting `/internal/elastic_assistant/knowledge_base/entries/_find` route - [X] Refactors `SortOrder` out of existing OAS's into the shared `schemas/common_attributes.schema.yaml` ### Client Hooks & Routes Adds new `useKnowledgeBaseEntries()` hook and corresponding `/knowledge_base/entries/_find` route for returning paginated KB Entries to populate the KB table in settings. E.g. ``` ts const { assistantFeatures: { assistantKnowledgeBaseByDefault: enableKnowledgeBaseByDefault }, http, toasts, } = useAssistantContext(); const { data: kbEntries, isLoading: isLoadingEntries } = useKnowledgeBaseEntries({ http }); ``` ###### Sample Response ``` json { "perPage": 20, "page": 1, "total": 145, "data": [ { "timestamp": "2024-06-05T21:19:56.482Z", "id": "CtBF6o8BSQy1Bdxt2FHz", "createdAt": "2024-06-05T21:19:56.482Z", "createdBy": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0", "updatedAt": "2024-06-05T21:19:56.482Z", "updatedBy": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0", "users": [ { "id": "u_mGBROF_q5bmFCATbLXAcCwKa0k8JvONAwSruelyKA5E_0", "name": "elastic" } ], "metadata": { "kbResource": "security_labs", "source": "/Users/garrettspong/dev/kibana-main/x-pack/plugins/elastic_assistant/server/knowledge_base/security_labs/2022_elastic_global_threat_report_announcement.mdx", "required": false }, "namespace": "default", "text": "[Source Content Here]", "vector": { "modelId": ".elser_model_2", "tokens": { "2": 0.06595266, ... } } }, ... ] } ``` Response is the full newly created `entry`. Same format for the entry as above in the `_find` API, and the `KnowledgeBaseEntries` cache is invalidated. Adds new `useCreateKnowledgeBaseEntry()` hook and corresponding `/knowledge_base/entries` route for creating new KB Entries ``` ts const entry: KnowledgeBaseEntryCreateProps = { metadata: { kbResource: 'user', required: true, source: 'user', }, text: 'Useful information about the user', }; const { mutate: createEntry, isLoading: isCreatingEntry } = useCreateKnowledgeBaseEntry({ http, }); await createEntry(entry); ``` Adds new `useDeleteKnowledgeBaseEntries()` hook and corresponding `/knowledge_base/entries/_bulk_action` route for deleting existing KB Entries. I left a TODO to plumb through `delete_by_query` so we can add a filter bar to the table. Need to confirm if we can do pagination with similarity search as well. ``` ts const { mutate: deleteEntries, isLoading: isDeletingEntries } = useDeleteKnowledgeBaseEntries({ http, }); await deleteEntries({ ids: ['YOE_CZABSQy1BdxtAGbs'] }) ``` See `KnowledgeBaseEntryBulkCrudActionResponse` for response formats. `KnowledgeBaseEntries` cache is invalidated upon delete. ### Checklist Delete any items that are not applicable to this PR. - [ ] ~Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)~ - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials * Feature currently behind feature flag. Documentation to be added before flag is removed. Tracked in elastic/security-docs#5337 - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] API tests will need to be rounded out as we finalize functionality behind the feature flag --------- Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
|
I've synced with @benironside, so just an update here: only some of this new Knowledge Base work is shipping in |
spong
added a commit
to elastic/kibana
that referenced
this issue
Sep 10, 2024
…o support `IndexEntries` (#186566) ## Summary This is a follow-up to #184974 that updates the KB Entries API's (and underlying schema) to support `IndexEntries` in addition to `DocumentEntries`. `IndexEntries` are entries in the Knowledge Base that are not backed by an embedded raw text source, but rather by an entire Index or Data Stream. The user can set the data source name, the specific field to query (must be ELSER embeddings in this initial implementation), and a description for when the assistant should search this data source for Knowledge Base content. This essentially enables the user to create custom retrieval tools backed by their own data. The changes in this PR, as with the other recent KB enhancements, are behind the following feature flag: ``` xpack.securitySolution.enableExperimental: - 'assistantKnowledgeBaseByDefault' ``` however as code change is required to test the new mappings. For this you can update the `knowledgeBaseDataStream` in `x-pack/plugins/elastic_assistant/server/ai_assistant_service/index.ts` to ```ts this.knowledgeBaseDataStream = this.createDataStream({ resource: 'knowledgeBase', kibanaVersion: options.kibanaVersion, fieldMap: knowledgeBaseFieldMapV2, // Update this to the V2 mapping }); ``` Change set includes: - [X] ES Knowledge Base data stream schema and OAS has been updated to support `IndexEntries`. - [X] OAS schema files have been moved to the `/entries` sub-directory - [ ] Backend KB services have been updated to support `IndexEntries` - [X] Storage methods updated - [ ] Retrieval methods updated (will round out these endpoint when working the UI next) --- With these API changes, I've also introduced a few sample `*.http` files for easier development/testing. These files are supported out of the box in JetBrains IDE's or in VSCode with the [httpyac](https://httpyac.github.io/) (and many other) extensions. Since the configuration for these files includes a `-` in the name, that's why you'll see a few @elastic/kibana-operations files updated. You can read more about `http` files [here](https://www.jetbrains.com/help/webstorm/http-client-in-product-code-editor.html) and for the spec see this repo [here](https://github.com/JetBrains/http-request-in-editor-spec/blob/master/spec.md). If we find these useful, we could add support to our [OpenAPI Generator](https://openapi-generator.tech/docs/generators/jetbrains-http-client) to create these automatically. They currently live co-located next to the OAS and generated schema files here: ``` x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/bulk_crud_knowledge_base_entries_route.http x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/crud_knowledge_base_entries_route.http ``` and the main config here: ``` x-pack/packages/kbn-elastic-assistant-common/env/http-client.env.json ``` The `x-pack/packages/kbn-elastic-assistant-common/.gitignore` has been updated to ignore `http-client.private.env.json` files locally, which is how you can override the config as you'd like. This is helpful to add variables like `basePath` as below: ``` { "dev": { "basePath": "/kbn" } } ``` To use them, just open the corresponding `*.http` for the API you want to test, and click `Send`, and the response will open in another tab. Here is what that looks like for creating one of the new `IndexEntry` KB documents that have been introduced in this PR: <p align="center"> <img width="500" src="https://github.com/user-attachments/assets/c9e70d1a-28d2-4eb3-9853-ab6d8e1c7acf" /> </p> ### Checklist Delete any items that are not applicable to this PR. - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials * Feature currently behind feature flag. Documentation to be added before flag is removed. Tracked in elastic/security-docs#5337 - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
gergoabraham
pushed a commit
to gergoabraham/kibana
that referenced
this issue
Sep 13, 2024
…o support `IndexEntries` (elastic#186566) ## Summary This is a follow-up to elastic#184974 that updates the KB Entries API's (and underlying schema) to support `IndexEntries` in addition to `DocumentEntries`. `IndexEntries` are entries in the Knowledge Base that are not backed by an embedded raw text source, but rather by an entire Index or Data Stream. The user can set the data source name, the specific field to query (must be ELSER embeddings in this initial implementation), and a description for when the assistant should search this data source for Knowledge Base content. This essentially enables the user to create custom retrieval tools backed by their own data. The changes in this PR, as with the other recent KB enhancements, are behind the following feature flag: ``` xpack.securitySolution.enableExperimental: - 'assistantKnowledgeBaseByDefault' ``` however as code change is required to test the new mappings. For this you can update the `knowledgeBaseDataStream` in `x-pack/plugins/elastic_assistant/server/ai_assistant_service/index.ts` to ```ts this.knowledgeBaseDataStream = this.createDataStream({ resource: 'knowledgeBase', kibanaVersion: options.kibanaVersion, fieldMap: knowledgeBaseFieldMapV2, // Update this to the V2 mapping }); ``` Change set includes: - [X] ES Knowledge Base data stream schema and OAS has been updated to support `IndexEntries`. - [X] OAS schema files have been moved to the `/entries` sub-directory - [ ] Backend KB services have been updated to support `IndexEntries` - [X] Storage methods updated - [ ] Retrieval methods updated (will round out these endpoint when working the UI next) --- With these API changes, I've also introduced a few sample `*.http` files for easier development/testing. These files are supported out of the box in JetBrains IDE's or in VSCode with the [httpyac](https://httpyac.github.io/) (and many other) extensions. Since the configuration for these files includes a `-` in the name, that's why you'll see a few @elastic/kibana-operations files updated. You can read more about `http` files [here](https://www.jetbrains.com/help/webstorm/http-client-in-product-code-editor.html) and for the spec see this repo [here](https://github.com/JetBrains/http-request-in-editor-spec/blob/master/spec.md). If we find these useful, we could add support to our [OpenAPI Generator](https://openapi-generator.tech/docs/generators/jetbrains-http-client) to create these automatically. They currently live co-located next to the OAS and generated schema files here: ``` x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/bulk_crud_knowledge_base_entries_route.http x-pack/packages/kbn-elastic-assistant-common/impl/schemas/knowledge_base/entries/crud_knowledge_base_entries_route.http ``` and the main config here: ``` x-pack/packages/kbn-elastic-assistant-common/env/http-client.env.json ``` The `x-pack/packages/kbn-elastic-assistant-common/.gitignore` has been updated to ignore `http-client.private.env.json` files locally, which is how you can override the config as you'd like. This is helpful to add variables like `basePath` as below: ``` { "dev": { "basePath": "/kbn" } } ``` To use them, just open the corresponding `*.http` for the API you want to test, and click `Send`, and the response will open in another tab. Here is what that looks like for creating one of the new `IndexEntry` KB documents that have been introduced in this PR: <p align="center"> <img width="500" src="https://github.com/user-attachments/assets/c9e70d1a-28d2-4eb3-9853-ab6d8e1c7acf" /> </p> ### Checklist Delete any items that are not applicable to this PR. - [X] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials * Feature currently behind feature flag. Documentation to be added before flag is removed. Tracked in elastic/security-docs#5337 - [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Wanted to create this epic to cover all the new work around the Security Assistant Knowledge Base for
8.15. The main effort can be summarized as adding support for 'custom Knowledge Base content' (both adding raw content and linking indices/data streams), but also includes improvements to how the KB is setup, managed in settings, and some new content we're planning on shipping, like exports of the Elastic Security Labs content.Background & resources
Open internal issues for tracking:
[UX] Knowledge Base design: Stack management and AI Assistant UIs #9392
[Epic] AI Assistant - Incorporate Latest Elastic Security Labs' Research into KB #8043
[Epic] AI Assistant and Insights - Allow users to import custom KB articles #8737
PR's thus far:
[Security Assistant] Automatically Install Knowledge Base #182763
[Security Assistant] Migrates to LangGraph and adds KB Tools #184554
[Security Assistant] Adds Security Labs Knowledge Base content #184885
[Security Assistant] Enables automatic setup of Knowledge Base and LangGraph code paths for 8.15 #188168
Point of contact: @spong @jamesspi @YulNaumenko
Test environments: TBD, but available by enabling the
assistantKnowledgeBaseByDefaultfeature flagWhich documentation set does this change impact?
ESS and serverless
ESS release
8.15
Serverless release
Soon after
8.15FF, so near mid-JulyFeature differences
Should have fully parity, but if anything ELSER/KB setup might end up being automatic in Serverless (so no 'Install Knowledge Base' button within the assistant).
API docs impact
We'll be introducing a whole new API for managing KB documents. This API is intended to be public at some point, but may ship as experimental or internal initially so we have flexibility to make modifications.
Initial OpenAPI Specs were added in [Security Assistant] Automatically Install Knowledge Base #182763, but they are still subject to change.
Prerequisites, privileges, feature flags
assistantKnowledgeBaseByDefaultxpack.securitySolution.enableExperimentalfeature flagThe text was updated successfully, but these errors were encountered: