[Security Solution] Elastic Security Assistant

[spong]

[Security Solution] Elastic Security Assistant

The Elastic Security Assistant has entered the chat, integrating generative AI and large language models (LLMs) into the workflows of Elastic Security users.

Bring your alerts, events, rules, and data quality checks into the conversation.

Kapture.2023-06-02.at.14.56.09.mp4

This PR merges a feature branch developed by @spong and @andrew-goldstein , seeded by @jamesspi 's prototype of the assistant. Connectivity to LLMs is provided the Generative AI Connector , developed by @stephmilovic . This PR includes:

• A new reusable Kibana package containing the assistant: x-pack/packages/kbn-elastic-assistant
  • See the How to embed the Assistant in other parts of Kibana for details
• Assistant integration into Elastic Security Solution workflows (e.g. alerts, cases, Timeline, rules, data quality)

An assistant trained on the Elastic stack and Elastic Security

The Generative AI Connector connects the assistant to OpenAI and Azure OpenAI models trained with knowledge of the Elastic stack and the Elastic Security solution, including:

• The Elastic open Detection Rules
• The Elastic Common Schema (ECS)
• Elastic query languages, including KQL, EQL, and the Elastic Query DSL
• Elasticsearch API documentation

This training enables the assistant to offer fully interactive chat experiences that include:

• alert summarization
• interactive query generation
• workflow suggestions
• generating ingestion configurations that conform to the Elastic Common Schema
• your imagination

using context from Elastic Security.

Take action from your conversations

The Actions (from assistant response):

• Send KQL to Timeline
• Send EQL to Timeline
• Send Elasticsearch DSL to Timeline
• Send Note to timeline
• Create new case
• Add to existing case
• Copy to clipboard

Feature Flag

This functionality has been introduced behind the assistantEnabled feature flag. Please add this configuration to your kibana.yml or Kibana Cloud User Settings configuration to enable:

xpack.securitySolution.enableExperimental: ['assistantEnabled']

Components architecture diagram

How to embed the Assistant in other parts of Kibana

Follow the general instructions in x-pack/packages/kbn-elastic-assistant/index.ts to integrate the assistant into a Kibana app.

Step 1 - Wrap your Kibana app in the AssistantProvider component

// Step 1: Wrap your Kibana app in the `AssistantProvider` component. This typically
// happens in the root of your app. Optionally provide a custom title for the assistant:

/** provides context (from the app) to the assistant, and injects Kibana services, like `http` */
export { AssistantProvider } from './impl/assistant_context';

Step 2: Add the AssistantOverlay component to your app

// Step 2: Add the `AssistantOverlay` component to your app. This component displays the assistant
// overlay in a modal, bound to a shortcut key:

/** modal overlay for Elastic Assistant conversations */
export { AssistantOverlay } from './impl/assistant/assistant_overlay';

// In addition to the `AssistantOverlay`, or as an alternative, you may use the `Assistant` component
// to display the assistant without the modal overlay:

/** this component renders the Assistant without the modal overlay to, for example, render it in a Timeline tab */
export { Assistant } from './impl/assistant';

Step 3: Wherever you want to bring context into the assistant, use the any combination of the following

// Step 3: Wherever you want to bring context into the assistant, use the any combination of the following
// components and hooks:
// - `NewChat` component
// - `NewChatById` component
// - `useAssistantOverlay` hook

/**
 * `NewChat` displays a _New chat_ icon button, providing all the context
 * necessary to start a new chat. You may optionally style the button icon,
 * or override the default _New chat_ text with custom content, like `🪄✨`
 *
 * USE THIS WHEN: All the data necessary to start a new chat is available
 * in the same part of the React tree as the _New chat_ button.
 */
export { NewChat } from './impl/new_chat';

/**
 * `NewChatByID` displays a _New chat_ icon button by providing only the `promptContextId`
 * of a context that was (already) registered by the `useAssistantOverlay` hook. You may
 * optionally style the button icon, or override the default _New chat_ text with custom
 * content, like {'🪄✨'}
 *
 * USE THIS WHEN: all the data necessary to start a new chat is NOT available
 * in the same part of the React tree as the _New chat_ button. When paired
 * with the `useAssistantOverlay` hook, this option enables context to be be
 * registered where the data is available, and then the _New chat_ button can be displayed
 * in another part of the tree.
 */
export { NewChatById } from './impl/new_chat_by_id';

/**
 * `useAssistantOverlay` is a hook that registers context with the assistant overlay, and
 * returns an optional `showAssistantOverlay` function to display the assistant overlay.
 * As an alterative to using the `showAssistantOverlay` returned from this hook, you may
 * use the `NewChatById` component and pass it the `promptContextId` returned by this hook.
 *
 * USE THIS WHEN: You want to register context in one part of the tree, and then show
 * a _New chat_ button in another part of the tree without passing around the data, or when
 * you want to build a custom `New chat` button with features not not provided by the
 * `NewChat` component.
 */
export { useAssistantOverlay } from './impl/assistant/use_assistant_overlay';

Co-authored-by: Garrett Spong garrett.spong@elastic.co
Co-authored-by: Andrew Macri andrew.macri@elastic.co

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[YulNaumenko]

This changes is not related to the PR context.

[YulNaumenko]

Could be excluded from PR as well

[YulNaumenko]

Where this is used and how this changes is related to the current PR?

[spong]

It's used in the useConversationStore wrapper to store state in localstorage. _setValue needed to be extend to support providing the prev value as a function param when updating:

kibana/x-pack/plugins/security_solution/public/assistant/use_conversation_store/index.tsx

Lines 19 to 26 in 542f3e5

	export const useConversationStore = (): UseConversationStore => {
	const [conversations, setConversations] = useLocalStorage<Record<string, Conversation>>({
	defaultValue: BASE_SECURITY_CONVERSATIONS,
	key: LOCAL_STORAGE_KEY,
	isInvalidDefault: (valueFromStorage) => {
	return !valueFromStorage;
	},
	});

[YulNaumenko]

Tested locally and all is good. Have some minor comment related to the code and some to the user workflows. It could be done as a follow up work.
Amazing work!!!

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 542f3e5
• Storybooks Preview

Failed CI Steps

• FTR Configs #21

Test Failures

• [job] [logs] FTR Configs #21 / Alerting alerts_as_data alerts as data flapping should set flapping and flapping_history for flapping alerts that settle on active

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	3912	4103	+191

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
@kbn/elastic-assistant	-	55	+55

Any counts in public APIs

Total count of every any typed public API. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats any for more detailed information.

id	before	after	diff
@kbn/elastic-assistant	-	4	+4

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	9.4MB	10.7MB	⚠️ +1.4MB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
@kbn/elastic-assistant	-	3	+3
stackConnectors	0	1	+1
total			+4

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
securitySolution	51.0KB	51.4KB	+359.0B

Unknown metric groups

API count

id	before	after	diff
@kbn/elastic-assistant	-	75	+75

ESLint disabled line counts

id	before	after	diff
@kbn/elastic-assistant	-	15	+15
enterpriseSearch	19	21	+2
securitySolution	414	418	+4
total			+21

Total ESLint disabled count

id	before	after	diff
@kbn/elastic-assistant	-	15	+15
enterpriseSearch	20	22	+2
securitySolution	498	502	+4
total			+21

History

• 💔 Build #132245 failed 52657fa
• 💔 Build #132199 failed ff5d0dd
• 💔 Build #132182 failed 71f0da8
• 💔 Build #132135 failed 6154e21
• 💔 Build #132057 failed 93bbcdb

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @spong @andrew-goldstein

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.8	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 156933

Questions ?

Please refer to the Backport tool documentation

[spong]

💚 All backports created successfully

Status	Branch	Result
✅	8.8

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[nicpenning]

This is looking rad. Is this something that can be tested via feature flag as mentioned above in 8.8.1 or come later in 8.9.0?

Also, will this require OpenAI and/or Azure OpenAI subscriptions to operate? Looking to understand some pre-requisites so when this drops we can immediately test out.

This looks like a force multiplier for us analysts out there.

[jamesspi]

@nicpenning , Answered you on the community slack :)