Skip to content

Commit

Permalink
[Security Solution] [Attack discovery] Fixes zero connectors and zero…
Browse files Browse the repository at this point in the history
… alerts empty states (#182904)

## [Security Solution] [Attack discovery] Fixes zero connectors and zero alerts empty states

### Summary

This PR fixes usability issues in _Attack discovery_ by displaying an [Empty prompt](https://eui.elastic.co/#/display/empty-prompt) for the "zero connectors" and "zero alerts" states.

- When there are zero connectors configured, the empty prompt in the following screenshot is displayed:

![no_connectors_empty_prompt](https://github.com/elastic/kibana/assets/4459398/caf794ef-6659-4037-b252-860c08929b93)

- When there are zero open alerts in the last 24 hours to send to the LLM, the empty prompt in the following screenshot is displayed:

![no_alerts_to_analyze](https://github.com/elastic/kibana/assets/4459398/6df1f3a7-e94d-4271-935d-bd0eddaf6e83)

The fix for the "no alerts" state required returning an additional stat, the number of alerts sent as context to the LLM:

```
alertsContextCount
```

The `alertsContextCount` stat is now included in telemetry.

### Desk testing

The `Test setup` section describes how to reproduce the states necessary to desk test this PR in an existing environment.

The `Steps to verify` section updates the test environment from zero to one connector. The connector will then be used to test the zero alerts state.

#### Test setup

Testing this fix requires no alerts, and no connectors. This section describes how to reset an existing environment (both the deployment and the browser) to test these states.

1. Navigate to Security > Alerts

2. If there are alerts in the last 24 hours, create and login to a new space, because zero alerts are required.

3. Navigate to Stack Management > Connectors

4. Delete any `OpenAI` or `Bedrock` connectors (tagged with `Generative AI for Security`)

5. Remove any pre-configured connectors from `kibana.dev.yml`

6. Clear local storage (to remove any trace of previously selected connectors)

7. Close all browser tabs with a current session to Kibana (to clear session storage)

8. Restart Kibana server

#### Steps to verify

1. Navigate to Security > Alerts

**Expected result**

- There are zero open alerts in the last 24 hours

2. Navigate to Security > Attack discovery

**Expected result**

- The following empty prompt is displayed:

![no_connectors_empty_prompt](https://github.com/elastic/kibana/assets/4459398/caf794ef-6659-4037-b252-860c08929b93)

3. Click `OpenAI`

**Expected result**

- The OpenAI connector modal is displayed

4. Enter the new connector details, and then click `Save`

**Expected results**

- A toast is displayed, confirming the new connector was successfully created
- The newly-created connector is selected in the connector selector
- The `Up to 20 alerts will be analyzed` empty state in the following screenshot is displayed:

![up_to_n_alerts_will_be_analyzed](https://github.com/elastic/kibana/assets/4459398/cb611bb8-6eb1-4ba2-9c0e-5563dcf0d0ca)

5. Click `Generate`

**Expected result**

- The `No alerts to analyze` empty state (for zero alerts sent as context to the LLM) is displayed:

![no_alerts_to_analyze](https://github.com/elastic/kibana/assets/4459398/6df1f3a7-e94d-4271-935d-bd0eddaf6e83)

6. Generate some alerts

7. Navigate to Security > Alerts

**Expected result**

- Alerts in the last 24 hours are now available (as shown by the Alerts page)

8. Once again, navigate to Security > Attack discovery

**Expected result**

- The `Up to 20 alerts will be analyzed` empty state is displayed

9. Once again, click `Generate`

**Expected results**

- The `Attack discovery in progress` loading callout is displayed
- Attack discoveries are created for the alerts (when applicable)

(cherry picked from commit 6f4d423)
  • Loading branch information
andrew-goldstein committed May 8, 2024
1 parent a71c51c commit 875b0ae
Show file tree
Hide file tree
Showing 30 changed files with 3,673 additions and 66 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@ export type AttackDiscoveryPostRequestBodyInput = z.input<typeof AttackDiscovery
export type AttackDiscoveryPostResponse = z.infer<typeof AttackDiscoveryPostResponse>;
export const AttackDiscoveryPostResponse = z.object({
connector_id: z.string().optional(),
alertsContextCount: z.number().optional(),
attackDiscoveries: z.array(AttackDiscovery).optional(),
replacements: Replacements.optional(),
status: z.string().optional(),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,8 @@ paths:
properties:
connector_id:
type: string
alertsContextCount:
type: number
attackDiscoveries:
type: array
items:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,33 @@ describe('useConnectorSetup', () => {
});
});

it('should NOT set the api config for each conversation when a new connector is saved and updateConversationsOnSaveConnector is false', async () => {
await act(async () => {
const { result, waitForNextUpdate } = renderHook(
() =>
useConnectorSetup({
...defaultProps,
updateConversationsOnSaveConnector: false, // <-- don't update the conversations
}),
{
wrapper: ({ children }) => <TestProviders>{children}</TestProviders>,
}
);
await waitForNextUpdate();
const { getByTestId, queryByTestId, rerender } = render(result.current.prompt, {
wrapper: TestProviders,
});
expect(getByTestId('connectorButton')).toBeInTheDocument();
expect(queryByTestId('skip-setup-button')).not.toBeInTheDocument();
fireEvent.click(getByTestId('connectorButton'));

rerender(result.current.prompt);
fireEvent.click(getByTestId('modal-mock'));

expect(setApiConfig).not.toHaveBeenCalled();
});
});

it('should show skip button if message has presentation data', async () => {
await act(async () => {
const { result, waitForNextUpdate } = renderHook(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,13 +36,15 @@ export interface ConnectorSetupProps {
isFlyoutMode?: boolean;
onSetupComplete?: () => void;
onConversationUpdate: ({ cId, cTitle }: { cId: string; cTitle: string }) => Promise<void>;
updateConversationsOnSaveConnector?: boolean;
}

export const useConnectorSetup = ({
conversation: defaultConversation,
isFlyoutMode,
onSetupComplete,
onConversationUpdate,
updateConversationsOnSaveConnector = true,
}: ConnectorSetupProps): {
comments: EuiCommentProps[];
prompt: React.ReactElement;
Expand Down Expand Up @@ -180,27 +182,38 @@ export const useConnectorSetup = ({

const onSaveConnector = useCallback(
async (connector: ActionConnector) => {
const config = getGenAiConfig(connector);
// persist only the active conversation
const updatedConversation = await setApiConfig({
conversation,
apiConfig: {
...conversation.apiConfig,
connectorId: connector.id,
actionTypeId: connector.actionTypeId,
provider: config?.apiProvider,
model: config?.defaultModel,
},
});
if (updateConversationsOnSaveConnector) {
// this side effect is not required for Attack discovery, because the connector is not used in a conversation
const config = getGenAiConfig(connector);
// persist only the active conversation
const updatedConversation = await setApiConfig({
conversation,
apiConfig: {
...conversation.apiConfig,
connectorId: connector.id,
actionTypeId: connector.actionTypeId,
provider: config?.apiProvider,
model: config?.defaultModel,
},
});

if (updatedConversation) {
onConversationUpdate({ cId: updatedConversation.id, cTitle: updatedConversation.title });
if (updatedConversation) {
onConversationUpdate({ cId: updatedConversation.id, cTitle: updatedConversation.title });

refetchConnectors?.();
setIsConnectorModalVisible(false);
}
} else {
refetchConnectors?.();
setIsConnectorModalVisible(false);
}
},
[conversation, onConversationUpdate, refetchConnectors, setApiConfig]
[
conversation,
onConversationUpdate,
refetchConnectors,
setApiConfig,
updateConversationsOnSaveConnector,
]
);

const handleClose = useCallback(() => {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -143,12 +143,13 @@ export const postAttackDiscoveryRoute = (
});
}

const parsedAttackDiscoveries = JSON.parse(rawAttackDiscoveries);
const { alertsContextCount, attackDiscoveries } = JSON.parse(rawAttackDiscoveries);

return response.ok({
body: {
alertsContextCount,
attackDiscoveries,
connector_id: connectorId,
attackDiscoveries: parsedAttackDiscoveries,
replacements: latestReplacements,
},
});
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,13 +44,15 @@ describe('useAttackDiscoveryTelemetry', () => {
model: 'gpt-4',
durationMs: 8000,
alertsCount: 20,
alertsContextCount: 25,
configuredAlertsCount: 30,
});
expect(reportAttackDiscoveriesGenerated).toHaveBeenCalledWith({
actionTypeId: '.gen-ai',
model: 'gpt-4',
durationMs: 8000,
alertsCount: 20,
alertsContextCount: 25,
configuredAlertsCount: 30,
});
});
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import type { UseAttackDiscovery } from '../use_attack_discovery';
export const getMockUseAttackDiscoveriesWithCachedAttackDiscoveries = (
fetchAttackDiscoveries: () => Promise<void>
): UseAttackDiscovery => ({
alertsContextCount: 20,
approximateFutureTime: null,
cachedAttackDiscoveries: {
claudeV3SonnetUsEast1: {
Expand Down Expand Up @@ -507,6 +508,7 @@ export const getMockUseAttackDiscoveriesWithCachedAttackDiscoveries = (
export const getMockUseAttackDiscoveriesWithNoAttackDiscoveries = (
fetchAttackDiscoveries: () => Promise<void>
): UseAttackDiscovery => ({
alertsContextCount: null,
approximateFutureTime: null,
cachedAttackDiscoveries: {},
fetchAttackDiscoveries,
Expand All @@ -520,6 +522,7 @@ export const getMockUseAttackDiscoveriesWithNoAttackDiscoveries = (
export const getMockUseAttackDiscoveriesWithNoAttackDiscoveriesLoading = (
fetchAttackDiscoveries: () => Promise<void>
): UseAttackDiscovery => ({
alertsContextCount: null,
approximateFutureTime: new Date('2024-04-15T17:13:29.470Z'), // <-- estimated generation completion time
cachedAttackDiscoveries: {},
fetchAttackDiscoveries,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,7 @@ const EmptyPromptComponent: React.FC<Props> = ({
<EuiFlexItem grow={false}>
<EuiLink
data-test-subj="learnMore"
href="https://www.elastic.co/guide/en/security/master/attack-discovery.html"
href="https://www.elastic.co/guide/en/security/current/attack-discovery.html"
target="_blank"
>
{i18n.LEARN_MORE}
Expand Down

0 comments on commit 875b0ae

Please sign in to comment.