[SIEM][Detection Engine] - Update DE to work with new exceptions schema #69715

yctercero · 2020-06-23T16:56:21Z

Summary

The purpose of this PR is to update the detection engine to work with the new exception list schema. Previously, exceptions lived directly in the rule, now they are stored separately, and the rule only stores a reference to said exception lists.

Changes are concentrated in two areas. First x-pack/plugins/lists:

Updates the EntryList schema so it now takes a list id and type (as needed for the detections engine logic) as opposed to before where it was just a string for the id
Updates the lists plugin unit tests to account for the updated EntryList schema
Exposes the exceptions list client (in the lists plugin) to be used by the detection engine

Second in x-pack/plugins/security_solution:

Updates the detection engine exceptions_list schema. Previously, exceptions sat directly on the rule itself, now that exceptions are stored separately, the exceptions_list property needs to only keep reference to the exceptions list
Updates schema unit tests
Updates routes to use new exceptions list schema
Updates route unit tests
Removes old exception list scripts that are no longer necessary

To Do

Update big loop logic (issue described below)
Update UI to now dynamically read from rule exception_list [SIEM][Detection Engine] - Update UI to read rule exceptions_list #69939

Right now we have logic in build_exceptions_query.ts that takes exceptions that don't include large value lists and appends them to the main KQL query which is then translated into the elastic query dsl. We also have logic in filter_events_wtih_list.ts that uses inverse logic to filter events search results against any exceptions with large value lists. On their own, each of these does what is expected. However, after all expansion of the exceptions list functionality, it is clear that these two separate pieces of logic would not suffice.

This PR does not address that issue (it's in a follow up PR), but thought it worth noting here. It gets confusing with the boolean logic, but to try to simplify the issue we came upon with our existing solution is the following:

Imagine we had a rule, with exceptions_list: [{ id: 'exception_list', namespace_type: 'single'}]. The exception_list has 2 exception list items:

// exception list item 1
a && b && c(large value list)

// exception list item 2
d && e(large value list)

The initial search would query (query && a && b) || (query && d). Then the results of that search would get filtered through the large value lists logic, so it would filter for (c || e). What this would result in, is signals that were filtered for:

(query && a && b) || (query && d) || c || e

when what we want is:

(query && a && b && c) || (query && d && e)

These are not logically equivalent. Our current logic would work if there was a single exception item, but as soon as there is more than one as shown in the simplified example above, the logic breaks down. This issue is being addressed in another PR.

Testing

To turn on lists plugin - in kibana.dev.yml

# Enable lists feature
xpack.lists.enabled: true
xpack.lists.listIndex: '.lists-yara'
xpack.lists.listItemIndex: '.items-yara'

Add export ELASTIC_XPACK_SIEM_LISTS_FEATURE=true to your bash file.

Use the scripts in x-pack/plugins/lists/server/scripts to create some sample exception lists and items. You can use the following:

If you've previously played around with lists, run ./hard_reset.sh (this will delete any lists you've created).

Create large value list:

Create large value list ./post_list.sh
Create large value list item ./post_list_item.sh (I modified the value to be "value": "10.4.2.140")

Create exception list:

Create exception list ./post_exception_list.sh
Create exception list item ./post_exception_list_item.sh ./exception_lists/new/exception_list_item_with_list.json

Use the scripts in x-pack/plugins/security_solution/server/lib/detection_engine/scripts to create rule:

Run ./post_rule.sh ./rules/queries/query_with_list.json (Makes reference to the exception list created in step above)

In the Alerts table, you should see something like the following where you only see events where the event.module is zeek and source.ip is 10.4.2.140 (or whatever ip you specified).

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

…updated schema

yctercero · 2020-06-23T16:57:19Z

x-pack/plugins/lists/common/schemas/types/default_namespace.ts

@@ -7,7 +7,7 @@
 import * as t from 'io-ts';
 import { Either } from 'fp-ts/lib/Either';

-const namespaceType = t.keyof({ agnostic: null, single: null });
+export const namespaceType = t.keyof({ agnostic: null, single: null });


NOTE: Exported to use in detection engine.

yctercero · 2020-06-23T16:59:10Z

x-pack/plugins/lists/common/schemas/types/entries.ts

@@ -52,7 +52,7 @@ export type EntryExists = t.TypeOf<typeof entriesExists>;

 export const entriesNested = t.exact(
  t.type({
-    entries: t.array(t.union([entriesMatch, entriesMatchAny, entriesList, entriesExists])),
+    entries: t.array(entriesMatch),


NOTE: Looking at KQL nested support, looks like it would only support our match and not any of the other operators. Please feel free to comment if that's wrong.

yctercero · 2020-06-23T17:00:20Z

x-pack/plugins/lists/common/schemas/types/entries.ts

@@ -34,9 +34,9 @@ export type EntryMatchAny = t.TypeOf<typeof entriesMatchAny>;
 export const entriesList = t.exact(
  t.type({
    field: t.string,
+    list: t.exact(t.type({ id: t.string, type })),


NOTE: Changed this to list from value, SOs were screaming when trying to index value as string | string[] | object so updated.

yctercero · 2020-06-23T17:10:39Z

x-pack/plugins/security_solution/public/common/components/exceptions/helpers.tsx

@@ -87,6 +88,16 @@ export const getFormattedEntries = (entries: EntriesArray): FormattedEntry[] =>
  return formattedEntries.flat();
 };

+export const getEntryValue = (entry: Entry): string | string[] | null => {
+  if (entriesList.is(entry)) {
+    return entry.list.id;


NOTE: This is temporary, didn't want to dig into changing UI code in this PR that's already large. Thought maybe we wanted to add name to the the EntryList for ease of use here, but that would also then require any updates to large lists to search for references to it in exception list items.

yctercero · 2020-06-23T17:23:21Z

x-pack/plugins/security_solution/server/lib/detection_engine/signals/signal_rule_alert_type.ts

@@ -140,6 +136,18 @@ export const signalRulesAlertType = ({
        await ruleStatusService.error(gapMessage, { gap: gapString });
      }
      try {
+        const { listClient, exceptionsClient } = await getListsClient({


NOTE: Now needing to load exception items, where as before they lived on the rule.

yctercero · 2020-06-23T17:23:36Z

x-pack/plugins/security_solution/server/lib/detection_engine/signals/signal_rule_alert_type.ts

@@ -214,18 +222,6 @@ export const signalRulesAlertType = ({
            result.bulkCreateTimes.push(bulkCreateDuration);
          }
        } else {
-          let listClient: ListClient | undefined;


NOTE: Moved into util seen above.

elasticmachine · 2020-06-23T17:39:01Z

Pinging @elastic/siem (Team:SIEM)

x-pack/plugins/lists/server/scripts/exception_lists/new/exception_list_item_with_list.json

x-pack/plugins/security_solution/server/lib/detection_engine/signals/build_exceptions_query.ts

x-pack/plugins/security_solution/server/lib/detection_engine/signals/filter_events_with_list.ts

spong

Check out, tested locally and reviewed code -- LGMT! 👍

And many thanks for all the added tests here as well @yctercero! 🙂

dhurley14

LGTM!

…ma (elastic#69715) * Updates list entry schema, exposes exception list client, updates tests * create new de list schema and unit tests * updated route unit tests and types to match new list schema * updated existing DE exceptions code so it should now work as is with updated schema * test and types cleanup * cleanup * update unit test * updates per feedback

* master: (45 commits) [QA] Unskip functional tests (elastic#69760) [SIEM][Detection Engine] - Update DE to work with new exceptions schema (elastic#69715) Fixes elastic#69639: Ignore url.url fields above 2048 characters (elastic#69863) PR: Provide limit warnings to user when API limits are reached. (elastic#69590) [Maps] Remove broken button (elastic#69853) Makes usage collection methods available on start (elastic#69836) [SIEM][CASE] Improve Jira's labelling (elastic#69892) [Logs UI] Access ML via the programmatic plugin API (elastic#68905) [ML] DF Analytics: Creation wizard part 3 (elastic#69456) Update Resolver generator script documentation (elastic#69912) [ML] Changes View results button text on new job page (elastic#69809) Add master branch to backport config (elastic#69893) [Ingest Manager] Kibana, not EPR, controls removable packages (elastic#69761) unskips 'Events columns' test (elastic#69684) [ML] Changes the ML overview empty analytics panel text (elastic#69801) [DOCS] Emphasizes where File Data Visualizer is located. (elastic#69812) add the `exactRoute` property to app registration (elastic#69772) Bump backport to 5.4.6 (elastic#69880) [Logs UI] ML log integration splash screen (elastic#69288) Clean up TSVB type client code to conform to the schema (elastic#68519) ...

…ma (#69715) (#69932) ## Summary The purpose of this PR is to update the detection engine to work with the new exception list schema. Previously, exceptions lived directly in the rule, now they are stored separately, and the rule only stores a reference to said exception lists. Changes are concentrated in two areas. First `x-pack/plugins/lists`: - Updates the `EntryList` schema so it now takes a list `id` and `type` (as needed for the detections engine logic) as opposed to before where it was just a string for the `id` - Updates the lists plugin unit tests to account for the updated `EntryList` schema - Exposes the exceptions list client (in the lists plugin) to be used by the detection engine Second in `x-pack/plugins/security_solution`: - Updates the detection engine `exceptions_list` schema. Previously, exceptions sat directly on the rule itself, now that exceptions are stored separately, the `exceptions_list` property needs to only keep reference to the exceptions list - Updates schema unit tests - Updates routes to use new exceptions list schema - Updates route unit tests - Removes old exception list scripts that are no longer necessary

kibanamachine · 2020-06-25T15:51:18Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 373fe8c

Build metrics

✅ unchanged

History

💚 Build #56156 succeeded 373fe8c
💚 Build #55712 succeeded 53536f8
💔 Build #55612 failed a2f6286

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: (90 commits) [Encrypted Saved Objects] Adds support for migrations in ESO (elastic#69513) [SIEM] Replace WithSource with useWithSource hook (elastic#68722) [Endpoint]EMT-451: add ability to filter endpoint metadata based on presence of unenrolled events (elastic#69708) rename old siem kibana config to securitySolution (elastic#69874) Remove unused Resolver code (elastic#69914) [Observability] Fixing dynamic return type based on the appName (elastic#69894) [SECURITY SOLUTION][INGEST] Task/endpoint list tests (elastic#69419) Fixes special clicks and 3rd party icon sizes in nav (elastic#69767) [APM] Catch annotations index permission error and log warning (elastic#69881) [Endpoint][Ingest Manager] minor code cleanup (elastic#69844) [Logs UI] Logs ui context menu (elastic#69915) Index pattern serialize and de-serialize (elastic#68844) [QA] Unskip functional tests (elastic#69760) [SIEM][Detection Engine] - Update DE to work with new exceptions schema (elastic#69715) Fixes elastic#69639: Ignore url.url fields above 2048 characters (elastic#69863) PR: Provide limit warnings to user when API limits are reached. (elastic#69590) [Maps] Remove broken button (elastic#69853) Makes usage collection methods available on start (elastic#69836) [SIEM][CASE] Improve Jira's labelling (elastic#69892) [Logs UI] Access ML via the programmatic plugin API (elastic#68905) ...

elasticmachine · 2021-09-23T14:32:54Z

Pinging @elastic/security-solution (Team: SecuritySolution)

yctercero added 5 commits June 22, 2020 16:22

Updates list entry schema, exposes exception list client, updates tests

3b19da1

create new de list schema and unit tests

3388918

updated route unit tests and types to match new list schema

e977f00

updated existing DE exceptions code so it should now work as is with …

55e29c9

…updated schema

test and types cleanup

65e47ca

yctercero commented Jun 23, 2020

View reviewed changes

cleanup

a2f6286

yctercero self-assigned this Jun 23, 2020

yctercero added Team:SIEM v7.9.0 v8.0.0 release_note:enhancement labels Jun 23, 2020

yctercero marked this pull request as ready for review June 23, 2020 17:38

yctercero requested review from a team as code owners June 23, 2020 17:38

yctercero added 2 commits June 23, 2020 17:00

Merge branch 'master' of github.com:yctercero/kibana into de_lists

6b728fe

update unit test

53536f8

spong reviewed Jun 24, 2020

View reviewed changes

x-pack/plugins/lists/server/scripts/exception_lists/new/exception_list_item_with_list.json Outdated Show resolved Hide resolved

dhurley14 reviewed Jun 24, 2020

View reviewed changes

x-pack/plugins/security_solution/server/lib/detection_engine/signals/build_exceptions_query.ts Outdated Show resolved Hide resolved

dhurley14 reviewed Jun 24, 2020

View reviewed changes

x-pack/plugins/security_solution/server/lib/detection_engine/signals/filter_events_with_list.ts Outdated Show resolved Hide resolved

spong approved these changes Jun 24, 2020

View reviewed changes

yctercero added 2 commits June 24, 2020 19:34

Merge branch 'master' of github.com:yctercero/kibana into de_lists

1713e7f

updates per feedback

373fe8c

dhurley14 approved these changes Jun 25, 2020

View reviewed changes

yctercero merged commit f7acbbe into elastic:master Jun 25, 2020

yctercero mentioned this pull request Jun 25, 2020

[7.x] [SIEM][Detection Engine] - Update DE to work with new exceptions schema (#69715) #69932

Merged

yctercero mentioned this pull request Jun 25, 2020

[SIEM][Detection Engine] - Update UI to read rule exceptions_list #69939

Closed

4 tasks

yctercero mentioned this pull request Jul 7, 2020

[Security Solution][Exceptions] - Tie server and client code together #70918

Merged

4 tasks

yctercero deleted the de_lists branch October 14, 2020 12:00

MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SIEM][Detection Engine] - Update DE to work with new exceptions schema #69715

[SIEM][Detection Engine] - Update DE to work with new exceptions schema #69715

yctercero commented Jun 23, 2020 •

edited

yctercero Jun 23, 2020

yctercero Jun 23, 2020

yctercero Jun 23, 2020

yctercero Jun 23, 2020

yctercero Jun 23, 2020

yctercero Jun 23, 2020

elasticmachine commented Jun 23, 2020

spong left a comment

dhurley14 left a comment

kibanamachine commented Jun 25, 2020

elasticmachine commented Sep 23, 2021

[SIEM][Detection Engine] - Update DE to work with new exceptions schema #69715

[SIEM][Detection Engine] - Update DE to work with new exceptions schema #69715

Conversation

yctercero commented Jun 23, 2020 • edited

Summary

To Do

Testing

Checklist

For maintainers

yctercero Jun 23, 2020

Choose a reason for hiding this comment

yctercero Jun 23, 2020

Choose a reason for hiding this comment

yctercero Jun 23, 2020

Choose a reason for hiding this comment

yctercero Jun 23, 2020

Choose a reason for hiding this comment

yctercero Jun 23, 2020

Choose a reason for hiding this comment

yctercero Jun 23, 2020

Choose a reason for hiding this comment

elasticmachine commented Jun 23, 2020

spong left a comment

Choose a reason for hiding this comment

dhurley14 left a comment

Choose a reason for hiding this comment

kibanamachine commented Jun 25, 2020

💚 Build Succeeded

Build metrics

History

elasticmachine commented Sep 23, 2021

yctercero commented Jun 23, 2020 •

edited