diff --git a/docs/AI-for-security/ai-for-security.asciidoc b/docs/AI-for-security/ai-for-security.asciidoc index 04df8ebf73..a08daab378 100644 --- a/docs/AI-for-security/ai-for-security.asciidoc +++ b/docs/AI-for-security/ai-for-security.asciidoc @@ -1,5 +1,5 @@ [[ai-for-security]] -= AI for security += AI for Security :frontmatter-description: Learn to use AI capabilities in {elastic-sec}. :frontmatter-tags-products: [security] @@ -9,6 +9,8 @@ You can use {elastic-sec}'s built-in AI tools to speed up your work and augment your team's capabilities. The pages in this section describe <>, which answers questions and enhances your workflows throughout {elastic-sec}, and <>, which speeds up the triage process by finding patterns and identifying attacks spanning multiple alerts. include::ai-security-assistant.asciidoc[leveloffset=+1] +include::knowledge-base.asciidoc[leveloffset=+2] + include::attack-discovery.asciidoc[leveloffset=+1] include::connector-guides-landing-pg.asciidoc[leveloffset=+1] diff --git a/docs/AI-for-security/ai-security-assistant.asciidoc b/docs/AI-for-security/ai-security-assistant.asciidoc index 8c11681738..218bc94953 100644 --- a/docs/AI-for-security/ai-security-assistant.asciidoc +++ b/docs/AI-for-security/ai-security-assistant.asciidoc @@ -24,7 +24,7 @@ WARNING: The Elastic AI Assistant is designed to enhance your analysis with smar * To set up AI Assistant, you need the **Actions and Connectors : All** {kibana-ref}/kibana-privileges.html[privilege]. -* You need an account with a third-party generative AI provider, which AI Assistant uses to generate responses. Supported providers are OpenAI, Azure OpenAI Service, and Amazon Bedrock. +* You need a <>, which AI Assistant uses to generate responses. -- [discrete] @@ -68,7 +68,7 @@ You can also chat with AI Assistant from several particular pages in {elastic-se * <>: Select the *Incompatible fields* tab, then click *Chat*. (This is only available for fields marked red, indicating they're incompatible). * <>: Select the *Security Assistant* tab. -NOTE: Each user's chat history and custom quick prompts are automatically saved, so you can leave {elastic-sec} and return to pick up a conversation later. +NOTE: Each user's chat history (up to the 99 most recent conversations) and custom Quick Prompts are automatically saved, so you can leave {elastic-sec} and return to a conversation later. Chat history appears to the left of the AI Assistant chat window, and on the **Conversations** tab of the **AI Assistant settings** menu. To access the settings menu, use the global search field to search for "AI Assistant for Security". [discrete] [[interact-with-assistant]] @@ -76,19 +76,17 @@ NOTE: Each user's chat history and custom quick prompts are automatically saved, Use these features to adjust and act on your conversations with AI Assistant: -* Select a _system prompt_ at the beginning of a conversation to establish how detailed and technical you want AI Assistant's answers to be. +* (Optional) Select a _System Prompt_ at the beginning of a conversation by using the **Select Prompt** menu. System Prompts provide context to the model, informing its response. To create a System Prompt, open the System Prompts dropdown menu and click *+ Add new System Prompt...*. +* (Optional) Select a _Quick Prompt_ at the bottom of the chat window to get help writing a prompt for a specific purpose, such as summarizing an alert or converting a query from a legacy SIEM to {elastic-sec}. + [role="screenshot"] -image::images/system-prompt.gif[The system prompt drop-down menu,90%] +image::images/quick-prompts.png[Quick Prompts highlighted below a conversation,90%] + -System prompts provide context to the model, informing its response. To create a custom system prompt, open the system prompts dropdown menu and click *+ Add new system prompt...*. - -* Select a _quick prompt_ at the bottom of the chat window to get help writing a prompt for a specific purpose, such as summarizing an alert or converting a query from a legacy SIEM to {elastic-sec}. +* System Prompts and Quick Prompts can also be configured from the corresponding tabs on the **Security AI settings** page. + -[role="screenshot"] -image::images/quick-prompts.png[Quick prompts highlighted below a conversation,90%] +image::images/assistant-settings-system-prompts.png[The Security AI settings menu's System Prompts tab,90%] + -Quick prompt availability varies based on context — for example, the **Alert summarization** quick prompt appears when you open AI Assistant while viewing an alert. To customize existing quick prompts and create new ones, click *Add Quick prompt*. +* Quick Prompt availability varies based on context—for example, the **Alert summarization** Quick Prompt appears when you open AI Assistant while viewing an alert. To customize existing Quick Prompts and create new ones, click *Add Quick Prompt*. * In an active conversation, you can use the inline actions that appear on messages to incorporate AI Assistant's responses into your workflows: @@ -104,22 +102,16 @@ TIP: AI Assistant can remember particular information you tell it to remember. F [discrete] [[configure-ai-assistant]] == Configure AI Assistant -The *Settings* menu (image:images/icon-settings.png[Settings icon,17,17]) allows you to configure default conversations, quick prompts, system prompts, and data anonymization. - -[role="screenshot"] -image::images/assistant-settings-menu.png[AI Assistant's settings menu, open to the Conversations tab] - -The *Settings* menu has the following tabs: +The *Security AI settings* page allows you to configure AI Assistant. To access it, use the global search field to search for "AI Assistant for Security". -* **Conversations:** When you open AI Assistant from certain pages, such as Timeline or Alerts, it defaults to the relevant conversation type. Choose the default system prompt for each conversation type, the connector, and model (if applicable). The **Streaming** setting controls whether AI Assistant's responses appear word-by-word (streamed), or as a complete block of text. Streaming is currently only available for OpenAI models. -* **Quick Prompts:** Modify existing quick prompts or create new ones. To create a new quick prompt, type a unique name in the *Name* field, then press *enter*. Under *Prompt*, enter or update the quick prompt's text. -* **System Prompts:** Edit existing system prompts or create new ones. To create a new system prompt, type a unique name in the *Name* field, then press *enter*. Under *Prompt*, enter or update the system prompt's text. Under *Contexts*, select where the system prompt should appear. -+ -NOTE: To delete a custom prompt, open the *Name* drop-down menu, hover over the prompt you want to delete, and click the *X* that appears. You cannot delete the default prompts. +It has the following tabs: +* **Conversations:** When you open AI Assistant from certain pages, such as **Timeline** or **Alerts**, it defaults to the relevant conversation type. For each conversation type, choose the default System Prompt, the default connector, and the default model (if applicable). The **Streaming** setting controls whether AI Assistant's responses appear word-by-word (streamed), or as a complete block of text. Streaming is currently only available for OpenAI models. +* **Connectors:** Manage all LLM connectors. +* **System Prompts:** Edit existing System Prompts or create new ones. To create a new System Prompt, type a unique name in the *Name* field, then press *enter*. Under *Prompt*, enter or update the System Prompt's text. Under *Contexts*, select where the System Prompt should appear. +* **Quick Prompts:** Modify existing Quick Prompts or create new ones. To create a new Quick Prompt, type a unique name in the *Name* field, then press *enter*. Under *Prompt*, enter or update the Quick Prompt's text. * **Anonymization:** Select fields to include as plaintext, to obfuscate, and to not send when you provide events to AI Assistant as context. <>. - -* **Knowledge base:** Provide additional context to AI Assistant so it can answer questions about {esql} and alerts in your environment. <>. +* **Knowledge base:** Provide additional context to AI Assistant. <>. [discrete] [[ai-assistant-anonymization]] @@ -131,7 +123,9 @@ NOTE: To delete a custom prompt, open the *Name* drop-down menu, hover over the To modify Anonymization settings, you need the **Elastic AI Assistant: All** privilege, with **Customize sub-feature privileges** enabled. -- -The **Anonymization** tab of the AI Assistant settings menu allows you to define default data anonymization behavior for events you send to AI Assistant. Fields with **Allowed** toggled on are included in events provided to AI Assistant. **Allowed** fields with **Anonymized** set to **Yes** are included, but with their values obfuscated. +The **Anonymization** tab of the Security AI settings menu allows you to define default data anonymization behavior for events you send to AI Assistant. Fields with **Allowed** toggled on are included in events provided to AI Assistant. **Allowed** fields with **Anonymized** set to **Yes** are included, but with their values obfuscated. + +NOTE: You can access anonymization settings directly from the **Attack Discovery** page by clicking the settings (image:images/icon-settings.png[Settings icon,17,17]) button next to the model selection dropdown menu. [role="screenshot"] image::images/assistant-anonymization-menu.png[AI Assistant's settings menu, open to the Anonymization tab] @@ -143,49 +137,17 @@ The *Show anonymized* toggle controls whether you see the obfuscated or plaintex When you include a particular event as context, such as an alert from the Alerts page, you can adjust anonymization behavior for the specific event. Be sure the anonymization behavior meets your specifications before sending a message with the event attached. [discrete] -[[ai-assistant-knowledge-base]] +[[ai-assistant-page-knowledge-base]] === Knowledge base -beta::[] - -The **Knowledge base** tab of the AI Assistant settings menu allows you to enable AI Assistant to answer questions about the Elastic Search Query Language ({esql}), and about alerts in your environment. To use knowledge base, you must <>. - -[discrete] -[[rag-for-esql]] -==== Knowledge base for {esql} -NOTE: {esql} is enabled by default in {kib}. It can be -disabled using the `enableESQL` setting from the -{kibana-ref}/advanced-options.html[Advanced Settings]. This will hide the {esql} user interface from various applications. However, users will be able to access existing {esql} artifacts like saved searches and visualizations. - -IMPORTANT: {esql} queries generated by AI Assistant might require additional validation. To ensure they're correct, refer to the {ref}/esql-language.html[{esql} documentation]. - -When this feature is enabled, AI Assistant can help you write an {esql} query for a particular use case, or answer general questions about {esql} syntax and usage. To enable AI Assistant to answer questions about {esql}: - -. Turn on the knowledge base by clicking **Setup**. If the **Setup** button doesn't appear, knowledge base is already enabled. -. Click *Save*. The knowledge base is now active. A quick prompt for {esql} queries becomes available, which provides a good starting point for your {esql} conversations and questions. - -NOTE: AI Assistant's knowledge base gets additional context from {ml-docs}/ml-nlp-elser.html#download-deploy-elser[Elastic Learned Sparse EncodeR (ELSER)]. - -[discrete] -[[rag-for-alerts]] -==== Knowledge base for alerts -When this feature is enabled, AI Assistant will receive multiple alerts as context for each of your prompts. It will receive alerts from the last 24 hours that have a status of `open` or `acknowledged`, ordered first by risk score, then by recency. Building block alerts are excluded. This enables it to answer questions about multiple alerts in your environment, rather than just the individual alerts you choose to include as context. - -To enable RAG for alerts: - -. Turn on the knowledge base by clicking **Setup**. If the **Setup** button doesn't appear, knowledge base is already enabled. -. Use the slider to select the number of alerts to send to AI Assistant. Click **Save**. -+ -[role="screenshot"] -image::images/knowledge-base-settings.png["AI Assistant's settings menu open to the Knowledge Base tab",75%] - -NOTE: Including a large number of alerts may cause your request to exceed the maximum token length of your third-party generative AI provider. If this happens, try selecting a lower number of alerts to send. +The **Knowledge base** tab of the **Security AI settings** page allows you to enable AI Assistant to remember specified information, and use it as context to improve response quality. To learn more, refer to <>. [discrete] [[ai-assistant-queries]] +[[rag-for-esql]] ### Get the most from your queries -Elastic AI Assistant helps you take full advantage of the {elastic-sec} platform to improve your security operations. Its ability to assist you depends on the specificity and detail of your questions. The more context and detail you provide, the more tailored and useful its responses will be. +Elastic AI Assistant allows you to take full advantage of the {elastic-sec} platform to improve your security operations. It can help you write an {esql} query for a particular use case, or answer general questions about how to use the platform. Its ability to assist you depends on the specificity and detail of your questions. The more context and detail you provide, the more tailored and useful its responses will be. To maximize its usefulness, consider using more detailed prompts or asking for additional information. For instance, after asking for an {esql} query example, you could ask a follow-up question like, “Could you give me some other examples?” You can also ask for clarification or further exposition, for example "Please provide comments explaining the query you just gave." diff --git a/docs/AI-for-security/attack-discovery.asciidoc b/docs/AI-for-security/attack-discovery.asciidoc index 7b0d32431a..01bceba437 100644 --- a/docs/AI-for-security/attack-discovery.asciidoc +++ b/docs/AI-for-security/attack-discovery.asciidoc @@ -56,7 +56,7 @@ It may take from a few seconds up to several minutes to generate discoveries, de IMPORTANT: Attack discovery is in technical preview and will only analyze opened and acknowleged alerts from the past 24 hours. By default it only analyzes up to 20 alerts within this timeframe, but you can expand this up to 100 by going to **AI Assistant → Settings (image:images/icon-settings.png[Settings icon,17,17]) → Knowledge Base** and updating the **Alerts** setting. -image::images/knowledge-base-settings.png["AI Assistant's settings menu open to the Knowledge Base tab",75%] +image::images/knowledge-base-assistant-settings-kb-tab.png["AI Assistant's settings menu open to the Knowledge Base tab",75%] IMPORTANT: Attack discovery uses the same data anonymization settings as <>. To configure which alert fields are sent to the LLM and which of those fields are obfuscated, use the Elastic AI Assistant settings. Consider the privacy policies of third-party LLMs before sending them sensitive data. diff --git a/docs/AI-for-security/images/assistant-anonymization-menu.png b/docs/AI-for-security/images/assistant-anonymization-menu.png index e942269e61..de53cfdfc9 100644 Binary files a/docs/AI-for-security/images/assistant-anonymization-menu.png and b/docs/AI-for-security/images/assistant-anonymization-menu.png differ diff --git a/docs/AI-for-security/images/assistant-basic-view.png b/docs/AI-for-security/images/assistant-basic-view.png index 4251f73ea2..8e66aa8be5 100644 Binary files a/docs/AI-for-security/images/assistant-basic-view.png and b/docs/AI-for-security/images/assistant-basic-view.png differ diff --git a/docs/AI-for-security/images/assistant-settings-menu.png b/docs/AI-for-security/images/assistant-settings-menu.png deleted file mode 100644 index 728e61f944..0000000000 Binary files a/docs/AI-for-security/images/assistant-settings-menu.png and /dev/null differ diff --git a/docs/AI-for-security/images/assistant-settings-system-prompts.png b/docs/AI-for-security/images/assistant-settings-system-prompts.png new file mode 100644 index 0000000000..b28456d8e1 Binary files /dev/null and b/docs/AI-for-security/images/assistant-settings-system-prompts.png differ diff --git a/docs/AI-for-security/images/knowledge-base-add-index-config.png b/docs/AI-for-security/images/knowledge-base-add-index-config.png new file mode 100644 index 0000000000..3fcb91977b Binary files /dev/null and b/docs/AI-for-security/images/knowledge-base-add-index-config.png differ diff --git a/docs/AI-for-security/images/knowledge-base-assistant-menu-dropdown.png b/docs/AI-for-security/images/knowledge-base-assistant-menu-dropdown.png new file mode 100644 index 0000000000..785c79e75c Binary files /dev/null and b/docs/AI-for-security/images/knowledge-base-assistant-menu-dropdown.png differ diff --git a/docs/AI-for-security/images/knowledge-base-assistant-settings-kb-tab.png b/docs/AI-for-security/images/knowledge-base-assistant-settings-kb-tab.png new file mode 100644 index 0000000000..74c799dd39 Binary files /dev/null and b/docs/AI-for-security/images/knowledge-base-assistant-settings-kb-tab.png differ diff --git a/docs/AI-for-security/images/knowledge-base-assistant-setup-button.png b/docs/AI-for-security/images/knowledge-base-assistant-setup-button.png new file mode 100644 index 0000000000..d861ba361a Binary files /dev/null and b/docs/AI-for-security/images/knowledge-base-assistant-setup-button.png differ diff --git a/docs/AI-for-security/images/knowledge-base-rbac.png b/docs/AI-for-security/images/knowledge-base-rbac.png new file mode 100644 index 0000000000..84b950504b Binary files /dev/null and b/docs/AI-for-security/images/knowledge-base-rbac.png differ diff --git a/docs/AI-for-security/images/knowledge-base-settings.png b/docs/AI-for-security/images/knowledge-base-settings.png deleted file mode 100644 index 0f907cdf6f..0000000000 Binary files a/docs/AI-for-security/images/knowledge-base-settings.png and /dev/null differ diff --git a/docs/AI-for-security/images/quick-prompts.png b/docs/AI-for-security/images/quick-prompts.png index 2adfa57f15..55d6ad2f24 100644 Binary files a/docs/AI-for-security/images/quick-prompts.png and b/docs/AI-for-security/images/quick-prompts.png differ diff --git a/docs/AI-for-security/knowledge-base.asciidoc b/docs/AI-for-security/knowledge-base.asciidoc new file mode 100644 index 0000000000..2f5414822b --- /dev/null +++ b/docs/AI-for-security/knowledge-base.asciidoc @@ -0,0 +1,138 @@ +[[ai-assistant-knowledge-base]] += AI Assistant Knowledge Base + +AI Assistant's Knowledge Base feature enables AI Assistant to recall specific documents and other specified information. This information, which can include everything from the location of your datacenters to the latest threat research, provides additional context that can improve the quality of AI Assistant's responses to your queries. This topic describes how to enable and add information to Knowledge Base. + +NOTE: When you upgrade from {elastic-sec} version 8.15 to a newer version, information previously stored by AI Assistant will be lost. + +.Requirements +[sidebar] +-- + +* To use Knowledge Base, you need the `Elastic AI Assistant: All` privilege. To edit global Knowledge Base entries (information that will affect the AI Assistant experience for other users in the {kib} space), you need the `Allow Changes to Global Entries` privilege. +* You must <> with a minimum ML node size of 4 GB. + +-- + +[discrete] +[[knowledge-base-rbac]] +== Role-based access control (RBAC) for Knowledge Base + +The `Elastic AI Assistant: All` role privilege allows you to use AI Assistant and access its settings. It has two sub-privileges, `Field Selection and Anonymization`, which allows you to customize which alert fields are sent to AI Assistant and Attack Discovery, and `Knowledge Base`, which allows you to edit and create new Knowledge Base entries. + +image::images/knowledge-base-rbac.png[Knowledge base's RBAC settings,60%] + +[discrete] +[[enable-knowledge-base]] +== Enable Knowledge Base + +There are two ways to enable Knowledge Base. + +NOTE: You must individually enable Knowledge Base for each {kib} space where you want to use it. + +[discrete] +=== Option 1: Enable Knowledge Base from an AI Assistant conversation + +Open a conversation with AI Assistant, select a large language model, then click **Setup Knowledge Base**. If the button doesn't appear, Knowledge Base is already enabled. + +image::images/knowledge-base-assistant-setup-button.png[An AI Assistant conversation showing the Setup Knowledge Base button] + +Knowledge base setup may take several minutes. It will continue in the background if you close the conversation. After setup is complete, you can access Knowledge Base settings from AI Assistant's conversation settings menu (access the conversation settings menu by clicking the three dots button next to the model selection dropdown). + +image::images/knowledge-base-assistant-menu-dropdown.png[AI Assistant's dropdown menu with the Knowledge Base option highlighted] + +[discrete] +=== Option 2: Enable Knowledge Base from the Security AI settings + +. To open Security AI settings, use the {kibana-ref}/introduction.html#kibana-navigation-search[global search field] to find "AI Assistant for Security." +. On the **Knowledge Base** tab, click **Setup Knowledge Base**. If the button doesn't appear, Knowledge Base is already enabled. + +image::images/knowledge-base-assistant-settings-kb-tab.png[AI Assistant's settings menu open to the Knowledge Base tab] + +[discrete] +[[rag-for-alerts]] +== Knowledge base for alerts +When Knowledge Base is enabled, AI Assistant receives `open` or `acknowledged` alerts from your environment from the last 24 hours. It uses these as context for each of your prompts. This enables it to answer questions about multiple alerts in your environment rather than just about individual alerts you choose to send it. It receives alerts ordered by risk score, then by the most recently generated. Building block alerts are excluded. + +To enable Knowledge Base for alerts: + +. Ensure that knowledge base is <>. +. Use the slider on the Security AI settings' Knowledge Base tab to select the number of alerts to send to AI Assistant. Click **Save**. + +NOTE: Including a large number of alerts may cause your request to exceed the maximum token length of your third-party generative AI provider. If this happens, try selecting a lower number of alerts to send. + +[discrete] +[[knowledge-base-add-knowledge]] +== Add knowledge + +To view all knowledge base entries, go to the Security AI settings and select the **Knowledge Base** tab. You can add individual documents or entire indices containing multiple documents. Each entry in the Knowledge Base (a document or index) has a **Sharing** setting of `private` or `global`. Private entries apply to the current user only and do not affect other users in the {kib} space, whereas global entries affect all users. Each entry can also have a `Required knowledge` setting, which means it will be included as context for every message sent to AI Assistant. + +NOTE: When you enable Knowledge Base, it comes pre-populated with articles from https://www.elastic.co/security-labs[Elastic Security Labs], current through September 30, 2024, which allows AI Assistant to leverage Elastic's security research during your conversations. This enables it to answer questions such as, “Are there any new tactics used against Windows hosts that I should be aware of when investigating my alerts?” + +[discrete] +[[knowledge-base-add-knowledge-document]] +=== Add an individual document + +Add an individual document to Knowledge Base when you want AI Assistant to remember a specific piece of information. + +. To open Security AI settings, use the {kibana-ref}/introduction.html#kibana-navigation-search[global search field] to find "AI Assistant for Security." Select the **Knowledge Base** tab. +. Click **New → Document** and give it a name. +. Under **Sharing**, select whether this knowledge should be **Global** or **Private**. +. Write the knowledge AI Assistant should remember in the **Markdown text** field. +. In the **Markdown text** field, enter the information you want AI Assistant to remember. +. If it should be **Required knowledge**, select the option. Otherwise, leave it blank. +Alternatively, you can simply send a message to AI Assistant that instructs it to "Remember" the information. For example, "Remember that I changed my password today, October 24, 2024", or "Remember we always use the Threat Hunting Timeline template when investigating potential threats". Entries created in this way are private to you. By default they are not required knowledge, but you can make them required by instructing AI Assistant to "Always remember", for example "Always remember to address me as madam", or "Always remember that our primary data center is located in Austin, Texas". + +Refer to the following video for an example of adding a document to Knowledge Base from the settings menu. + +======= +++++ + + +
+++++ +======= + +[discrete] +[[knowledge-base-add-knowledge-index]] +=== Add an index + +Add an index as a knowledge source when you want new information added to that index to automatically inform AI Assistant's responses. Common security examples include asset inventories, network configuration information, on-call matrices, threat intelligence reports, and vulnerability scans. + +IMPORTANT: Indices added to Knowledge Base must have at least one field mapped as {ref}/semantic-text.html[semantic text]. + +. To open Security AI settings, use the {kibana-ref}/introduction.html#kibana-navigation-search[global search field] to find "AI Assistant for Security." Select the **Knowledge Base** tab. +. Click **New → Index**. +. Name the knowledge source. +. Under **Sharing**, select whether this knowledge should be **Global** or **Private**. +. Under **Index**, enter the name of the index you want to use as a knowledge source. +. Under **Field**, enter the names of one or more semantic text fields within the index. +. Under **Data Description**, describe when this information should be used by AI Assistant. +. Under **Query Instruction**, describe how AI Assistant should query this index to retrieve relevant information. +. Under **Output Fields**, list the fields which should be sent to AI Assistant. If none are listed, all fields will be sent. + +image::images/knowledge-base-add-index-config.png[Knowledge base's Edit index entry menu,80%] + +Refer to the following video for an example of adding an index to Knowledge Base. + +======= +++++ + + +
+++++ +=======