Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution][Investigations] - Add kibana.alert.url #155069

Merged
merged 11 commits into from Apr 25, 2023

Conversation

michaelolo24
Copy link
Contributor

@michaelolo24 michaelolo24 commented Apr 17, 2023

Summary

This PR introduces the field kibana.alert.url to the alerts generated by al alert rule types. Functionality was added in this PR for 8.8 to allow users to link directly to the alert flyout. To be able to provide users with this field via our connectors, we are adding the url under the field kibana.alert.url.

To test, create an alert of any type and you should see this field set in the alert flyout:
image

The url provided is a redirect path that contains the necessary information (space, id, index, and timestamp) to be able to redirect the user to a filtered alert page for the given alert and the detail flyout opened. This allows us to retain flexibility in the future for any changes that may occur with the alert flyout or an alert page. More on that can be found in the earlier pr: #148800

Testing

  1. The kibana.alert.url field makes use of the publicBaseUrl configuration which must be set in your kibana.dev.yml for this field to be generated. Add the following to your yaml file. Note that if you use a basePath, it will have to be appended to the end of your publicBaseUrl path.
server.publicBaseUrl: 'http://localhost:5601'

with basePath:

server.basePath: '/someBasePath'
server.publicBaseUrl: 'http://localhost:5601/someBasePath'
  1. Generate data and enable any rule type to get alerts.
  2. Go to the alert page, click expand detail, and search for kibana.alert.url in the table.
  3. Visit that url and you should see a filtered alert page with the details flyout opened

***Caveat - when grouping is enabled, the details flyout will not open as the table that it is attached to is not actually loaded at that point in time. When the table is loaded by either disabling grouping or opening the group, the details flyout will open

@michaelolo24 michaelolo24 added Team:Threat Hunting Security Solution Threat Hunting Team release_note:feature Makes this part of the condensed release notes Team:Threat Hunting:Investigations Security Solution Investigations Team 8.8 candidate labels Apr 17, 2023
@michaelolo24 michaelolo24 force-pushed the alert-details-url-update branch 2 times, most recently from b234ce5 to 85ae638 Compare April 18, 2023 13:15
@@ -143,11 +148,20 @@ export const buildAlertRoot = (
alertTimestampOverride
);
const alertId = generateAlertId(doc);
const alertUrl = getAlertDetailsUrl({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the individual building block alerts for EQL sequences have alert URLs as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, added them

lists,
logger,
config,
publicBaseUrl,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the use of publicBaseUrl require customers to configure the core.http.basePath.publicBaseUrl setting in order for alert URLs to work?

Rules created through the UI have a

meta: {from: "1m", kibana_siem_app_url: "http://localhost:5601/jde/app/security"}

object where the kibana_siem_app_url is populated from the browser to provide results_link functionality. Should we use that param in conjunction with the publicBaseUrl?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note from discussing on zoom: kibana_siem_app_url does not get populated for prebuilt rules at the moment. We may want to update the prebuilt rule installation to populate that field for prebuilt rules. cc @banderror

Copy link
Contributor

@banderror banderror Apr 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may want to update the prebuilt rule installation to populate that field for prebuilt rules

I'm missing the context @marshallmain -- could you please explain what this field is used for? I don't know what the "results_link functionality" is.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's added to the actions context to provide users of 3rd party systems with a link back to the alerts table. However, since the Kibana server does not always know the public facing URL of Kibana we rely on this kibana_siem_app_url rule parameter that is set by the UI when custom rules are created.

@@ -49,6 +49,9 @@ const ALERT_LAST_DETECTED = `${ALERT_NAMESPACE}.last_detected` as const;
// kibana.alert.reason - human readable reason that this alert is active
const ALERT_REASON = `${ALERT_NAMESPACE}.reason` as const;

// kibana.alert.url - url which will take the user directly to a view of the alert
const ALERT_URL = `${ALERT_NAMESPACE}.url` as const;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add ALERT_URL to an alert schema for 8.8, similar to how we updated the BaseFields for 8.4 (x-pack/plugins/security_solution/common/detection_engine/schemas/alerts/8.4.0/index.ts) when adding ALERT_RULE_INDICES?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am doing it another PR.

@@ -21,6 +21,7 @@ const {
applyDeltaToColumnWidth,
changeViewMode,
removeColumn,
toggleDetailPanel,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was added as a bugfix. The flyout kept re-opening on refresh of the page after being closed. This was due to the fact that the opening and closing of the flyout wasn't tracked for updating the localstorage config.

@@ -31,7 +31,7 @@ const AlertsContainerComponent: React.FC = () => {
<Switch>
<Route path={ALERTS_PATH} exact component={AlertsRoute} />
{/* Redirect to the alerts page filtered for the given alert id */}
<Route path={`${ALERTS_PATH}/:alertId`} component={AlertDetailsRedirect} />
<Route path={`${ALERTS_PATH}/redirect/:alertId`} component={AlertDetailsRedirect} />
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the redirect subpath, to be explicit about the path functionality, but also preserve any future paths we may want to use such as /alert or /:id, etc...

>
{i18n.SHARE_ALERT}
</EuiButtonEmpty>
<EuiFlexItem grow={false}>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was a visual bug fix for the position of the Share Alert Details button here: #155030

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not for this PR, but while desk testing, we noted that:

  • URLs generated by the kibana.alert.url feature in this PR do NOT encode the : characters in the &timestamp=2023-04-24T18:10:36.225Z URL parameter, per the example below:
http://localhost:5601/s/space-madness/app/security/alerts/redirect/aa7c1d26c66a7a549233c000bc8c240979a273abad6b50ada5b19f7dec86553a?index=.alerts-security.alerts-space-madness&timestamp=2023-04-24T18:10:36.225Z

Above: A URL generated with the kibana.alert.url feature in this PR

However, URLs generated by the Share Alert Deatils button encode the : characters in the &timestamp=2023-04-24T18%3A10%3A36.225Z URL parameter, per the example below:

http://localhost:5601/s/space-madness/app/security/alerts/redirect/aa7c1d26c66a7a549233c000bc8c240979a273abad6b50ada5b19f7dec86553a?index=.alerts-security.alerts-space-madness&timestamp=2023-04-24T18%3A10%3A36.225Z

Above: A URL for the same alert, generated with the Share Alert Details button

Since it doesn't feel necessary to encode the : characters, the differences in behavior may be worth investigating / normalizing.

});
if (isPreviewAlert) return null;
const url = getAppUrl({ path: alertDetailPath });
// We use window.location.origin instead of http.basePath as the http.basePath has to be configured in config dev yml
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While we expect publicBaseUrl to be configured by most of our users, it's not guaranteed to always be configured. Since we have guaranteed access to the location.origin on the UI, we use that in place of core.http.publicBaseUrl,

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I desk tested this with a specific alert, in a non-default space, and verified that:

✅ when Kibana is run without a base path, a valid link (with no base path) is copied:

http://localhost:5601/s/space-madness/app/security/alerts/redirect/8c6f83aeefa05a130769c3bf2419d6923714554916ce4b990966c7b9d1459261?index=.alerts-security.alerts-space-madness&timestamp=2023-04-24T20%3A54%3A25.941Z

✅ when Kibana is run with a base path, a valid link for the same alert (that includes the base path) is copied:

http://localhost:5601/jgy/s/space-madness/app/security/alerts/redirect/8c6f83aeefa05a130769c3bf2419d6923714554916ce4b990966c7b9d1459261?index=.alerts-security.alerts-space-madness&timestamp=2023-04-24T20%3A54%3A25.941

}) => {
const alertDetailPath = buildAlertDetailPath({ alertId, index, timestamp });
const alertDetailPathWithAppPath = `${APP_PATH}${alertDetailPath}`;
return basePath
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

core.http.publicBaseUrl isn't guaranteed to be set, though it's highly likely that it will be. Given that condition, we will not produce a url if the basePath is not set

@michaelolo24 michaelolo24 marked this pull request as ready for review April 21, 2023 18:44
@michaelolo24 michaelolo24 requested review from a team as code owners April 21, 2023 18:44
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

Copy link
Member

@pmuellr pmuellr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ResponseOps changes (one comment in one file) LGTM :-)

Copy link
Member

@maryam-saeidi maryam-saeidi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AO changes LGTM

@@ -94,7 +94,7 @@ const ALERT_RULE_TAGS = `${ALERT_RULE_NAMESPACE}.tags` as const;
// kibana.alert.rule_type_id - rule type id for rule that generated this alert
const ALERT_RULE_TYPE_ID = `${ALERT_RULE_NAMESPACE}.rule_type_id` as const;

// kibana.alert.url - allow our user to go back to the details url in kibana
// kibana.alert.url - url which will redirect users to the alert page filtered for the given alert
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the alert page filtered for the given alert

nit: Since the AO team is working on introducing a dedicated page for each alert, this comment will not be accurate in the future. How about changing it to something like:

url which will redirect users to a page related to the given alert

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, will do, thanks @maryam-saeidi . Are there any tickets for the planned work @maryam-saeidi ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we have different epics for different rule types, for example:

NewTermsFields840,
} from '../8.4.0';

/* DO NOT MODIFY THIS SCHEMA TO ADD NEW FIELDS. These types represent the alerts that shipped in 8.6.0.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider replacing 8.6.0 with 8.8.0 throughout this comment

*/

export type GenericAlert880 = AlertWithCommonFields800<
BaseFields840 & { [ALERT_URL]: string | undefined }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since ALERT_URL is common to all the alert types, can you define it in a new interface BaseFields880 in this file and have the rule-specific alert interfaces extend BaseFields880 instead of the 8.4 versions of those interfaces?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, originally when adding these schemas I was torn between whether we should extend prior versions when adding new fields (avoiding duplication but adding some complexity) or simply copy the old version and add the new fields. I think with the way the schemas have evolved it might be easier to read if we copy the previous version and accept the duplication of existing fields in the new schema.

I imagine labeling each field with when it was added could help with the duplication, e.g.

export interface BaseFields880 {
  [TIMESTAMP]: string; // ADDED IN: 8.0.0
  [SPACE_IDS]: string[]; // ADDED IN: 8.0.0
  [EVENT_KIND]: 'signal'; // ADDED IN: 8.0.0
  ...
  [ALERT_RULE_VERSION]: number; // ADDED IN: 8.0.0
  'kibana.alert.rule.risk_score': number; // ADDED IN: 8.0.0
  'kibana.alert.rule.severity': string; // ADDED IN: 8.0.0
  'kibana.alert.rule.building_block_type': string | undefined; // ADDED IN: 8.0.0
  [ALERT_RULE_INDICES]: string[]; // ADDED IN: 8.4.0
  [ALERT_URL]: string | undefined; // ADDED IN: 8.8.0
  [key: string]: SearchTypes; // ADDED IN: 8.0.0
}

For this PR it's completely fine to continue extending old versions of the interfaces, but I wanted to note that we don't have to stick with that design going forward and if it makes more sense to you to copy all the fields over then that's a reasonable approach too (and one we may switch to in the future). Of course interfaces within a particular version will still extend each other, e.g. EqlShellFields880 extends BaseFields880.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, I wanted to do that, but since it doesn't look like the UUID is generated at that point, it made it a bit harder since the url depends on that. Since I didn't have access to the ALERT_UUID in build_alert.ts, I just followed the same pattern as how ALERT_UUID was set in the 8.4.0 file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ruleExecutionLogger
);

const alertUrl = getAlertDetailsUrl({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like we call getAlertDetailsUrl every time we call buildBulkBody, can we move the details URL logic inside buildBulkBody? That change will line up nicely with adding the ALERT_URL to BaseFieldsLatest since buildBulkBody returns BaseFieldsLatest.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, I wanted to do that, but it also relied on having the alert id, which for eql alerts was generated outside of buildBulkBody. I can migrate that logic too into buildBulkBody, but wasn't sure if there was a reason it was done separately, so I just left them all externally similar to how the kibana.alert.uuid is set

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed on slack, I've updated the code so that the build_alert.ts has the logic for creating the url, and also sets the id. As suggested, I've updated all of the types and added a placeholder id for the eql alerts that's later overridden: c916500

@andrew-goldstein
Copy link
Contributor

Perhaps not for this PR, but the following steps / video demonstrates an issue where, when the Alerts table grouping feature is enabled on the Alerts page, the flyout doesn't appear in a newly-loaded tab until table grouping is disabled (in the new tab).

To reproduce (with a local Elasticsearch started with yarn es snapshot), and a local Kibana development environment:

  1. Enable the kibana.alert.url feature by adding one of the following configurations to config/kibana.dev.yml:

If you run kibana with yarn start --no-base-path:

server.publicBaseUrl: 'http://localhost:5601'

or

If you run kibana with yarn start (and a base path):

server.basePath: '/foo'
server.publicBaseUrl: 'http://localhost:5601/foo'
  1. Navigate to the Security > Alerts page

  2. Set the alerts table grouping to Group alerts by: none

  3. While in the $KIBANA_HOME directory, execute the following command to generate alerts:

cd x-pack/plugins/security_solution && yarn test:generate && cd -
  1. On the alerts page, click the Manage alerts button

  2. In the Rules page, disable, then re-enable the Endpoint Security rule, to trigger immediate execution of the rule

  3. Once again, navigate to the Security > Alerts page

Expected result

  • Newly-generated alerts appear in the Alerts table
  1. Click the View details action button for any newly-generated alert

  2. Click the Table tab in the flyout

  3. Enter the following text in the search filter: kibana.alert.url

  4. Using the mouse, manually select the Value of the kibana.alert.url field, and copy it to the clipboard

  5. Paste the URL in a new browser tab

Expected results

  • The new browser tab opens to the Alerts page
  • The search bar of the Alerts page is pre-filled with the of the event, e.g. _id: 213e4f537c7e22602dbeef062b17df55a6e7be6af39b724d39dfb33d8c359e6d, and the date range is pre-selected to include the alert
  • The alert details flyout automatically opens
  1. Close the new browser tab

  2. In the (existing) Alerts page, change the Alerts table grouping to Group alerts by: Rule name

  3. Expand the Endpoint Security group group

  4. Once again, click the View details action for an alert in the table

  5. Click the Table tab in the flyout

  6. Once again, enter the following text in the search filter: kibana.alert.url

  7. Using the mouse, manually select the Value of the kibana.alert.url field, and copy it to the clipboard

  8. Paste the URL in a new browser tab

Expected results

  • The new browser tab opens to the Alerts page
  • The search bar of the Alerts page is pre-filled with the of the event, e.g. _id: 213e4f537c7e22602dbeef062b17df55a6e7be6af39b724d39dfb33d8c359e6d, and the date range is pre-selected to include the alert
  • The alert details flyout automatically opens

Actual results

  • The new browser tab opens to the Alerts page
  • The search bar of the Alerts page is pre-filled with the of the event, e.g. _id: 213e4f537c7e22602dbeef062b17df55a6e7be6af39b724d39dfb33d8c359e6d, and the date range is pre-selected to include the alert
  • The alert details flyout does NOT automatically open // unexpected
  1. Once again, change the alerts table grouping to Group alerts by: none

Expected results

  • The table is once again grouped by nothing
  • The alerts table flyout does NOT appear (it should have already opened during page load, but didn't)

Actual results

  • The table is once again grouped by nothing
  • The alerts table flyout, which should have opened when the page opened, finally opens // unexpected
repro.mov

}) => {
const alertDetailPath = buildAlertDetailPath({ alertId, index, timestamp });
const alertDetailPathWithAppPath = `${APP_PATH}${alertDetailPath}`;
return basePath
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider updating the PR description to note that, as a result of this bathPath check, the kibana.alert.url feature in this PR must be enabled by adding one of the following configurations to config/kibana.dev.yml:

If you run kibana with yarn start --no-base-path:

server.publicBaseUrl: 'http://localhost:5601'

or

If you run kibana with yarn start (and a base path):

server.basePath: '/foo'
server.publicBaseUrl: 'http://localhost:5601/foo'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added already 😄

Copy link
Contributor

@andrew-goldstein andrew-goldstein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @michaelolo24 for this feature! 🙏

I successfully desk tested it with various combinations of:

server.publicBaseUrl configured without a base path (in kibana.dev.yml), e.g.

server.publicBaseUrl: 'http://localhost:5601'

server.publicBaseUrl configured with a base path:

server.basePath: '/foo'
server.publicBaseUrl: 'http://localhost:5601/foo'

✅ the default space
✅ a non-default space

In all cases, the redirect URL behaved as-expected apart from one observation that may be addressed in a follow-up PR.

LGTM 🚀

@michaelolo24
Copy link
Contributor Author

Perhaps not for this PR, but the following steps / video demonstrates an issue where, when the Alerts table grouping feature is enabled on the Alerts page, the flyout doesn't appear in a newly-loaded tab until table grouping is disabled (in the new tab).

Thanks for this note. We were able to discuss a bit over zoom, but currently the flyout logic is embedded within the redux state for the loaded alerts page table. When grouping is enabled, the alerts table is loaded by default as all groups are default collapsed. The fix for this will most likely be expanding the single group by default. I can discuss what this will entail with the team managing grouping 😄

@kibana-ci
Copy link
Collaborator

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id before after diff
securitySolution 3847 3848 +1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 9.1MB 9.1MB +307.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
securitySolution 60.0KB 60.2KB +192.0B
Unknown metric groups

ESLint disabled line counts

id before after diff
enterpriseSearch 17 19 +2
securitySolution 397 400 +3
total +5

Total ESLint disabled count

id before after diff
enterpriseSearch 18 20 +2
securitySolution 477 480 +3
total +5

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@logeekal
Copy link
Contributor

logeekal commented Apr 25, 2023

@michaelolo24

  1. Not exactly related to this functionality. But in case of building block alerts, they are not visible directly unless show building block alerts is checked. Do we have a plan to mitigate that? This is the same problem we face in redirection from Dashboard --> Alerts Page.

  2. Do we need to do any migration as well to include kibana alert url for old alerts as well?

@michaelolo24
Copy link
Contributor Author

michaelolo24 commented Apr 25, 2023

@michaelolo24

1. Not exactly related to this functionality. But in case of building block alerts, they are not visible directly unless `show building block alerts`  is checked. Do we have a plan to mitigate that? This is the same problem we face in redirection from `Dashboard --> Alerts Page`.

2. Do we need to do any migration as well to include kibana alert url for old alerts as well?

Hey @logeekal thanks for bringing this up

  1. I saw that after my most recent updates and I wanted to talk to the team about that later today. I think we can do something with query params to be able to set the property in redux so it can be pulled out in use_data_table_filters.ts, but I wonder if it's worth just defaulting building block alerts to be enabled when the table loads? I'm just really weary of adding additional parameters to the url. I think the way we solve this for this PR and your work should be the same though ideally. In the meantime, while it's not the best experience, we can let user's know they may need to update their building block alerts selection if the alerts are not visible.

  2. That would most likely require re-indexing, which I'm not sure is the best idea as how do you determine how far back you go to update alerts? All old alerts will have the Copy Alert button in their respective alert flyout to get the link that way, but only 8.8+ alerts will have the kibana.alert.url field set. It's primarily to provide the url to users via their external connectors, so it wouldn't make as much sense for older alerts on that end either.

Copy link
Contributor

@marshallmain marshallmain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
8.8 candidate backport:skip This commit does not require backporting release_note:feature Makes this part of the condensed release notes Team:Threat Hunting:Investigations Security Solution Investigations Team Team:Threat Hunting Security Solution Threat Hunting Team v8.8.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet