[Security Solution][Exceptions] Implement exceptions for ML rules #84006

marshallmain · 2020-11-20T23:54:14Z

Summary

Adds exceptions for ML rules through the UI and implements them in the backend. Also implements the ability to use value list exceptions on arrays of values like user.name: ["root", "some_user"] which is important for ML rules. Documents created by the ML plugin use arrays for most fields, even when they only contain a single value.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

For maintainers

This was checked for breaking API changes and was labeled appropriately

marshallmain · 2020-11-23T15:45:32Z

@elasticmachine merge upstream

rylnd

Added some initial comments, mostly around typings. Generally this looks great, but I'd like to pair with you on desk-testing this if possible; I wanna see what you've been using to generate anomaly data!

rylnd · 2020-11-25T20:31:11Z

x-pack/plugins/security_solution/public/common/containers/source/index.tsx

@@ -197,8 +197,10 @@ export const useFetchIndex = (
  );

  useEffect(() => {
-    if (!isEmpty(indexNames) && !isEqual(previousIndexesName.current, indexNames)) {
+    if (!isEqual(previousIndexesName.current, indexNames)) {


This change might be redundant with mine

rylnd · 2020-11-25T21:04:48Z

x-pack/plugins/security_solution/server/lib/detection_engine/signals/bulk_create_ml_signals.ts

@@ -15,6 +14,7 @@ import { RuleTypeParams, RefreshTypes } from '../types';
 import { singleBulkCreate, SingleBulkCreateResponse } from './single_bulk_create';
 import { AnomalyResults, Anomaly } from '../../machine_learning';
 import { BuildRuleMessage } from './rule_messages';
+import { SearchResponse } from '../../types';


I see that we're continuing to diverge from the official typings, but I couldn't find an explanation as to why (I was able to revert these without type errors). If we have an unsupported use case we should probably file an issue upstream.

I think I was playing around with the types as well and forgot to switch this one back...got curious about the duplicates

Ah I got an error when changing it back and tracked it down. The search response for KQL query rules is typed based on the custom SearchResponse type because it is usinghits.total which looks to be typed (incorrectly) as number in the official SearchResponse type. To share the "filter against lists" code with the KQL rules then the ML bulk create logic has to accept our custom SearchResponse.

Need to investigate further why the upstream SearchResponse.hits.total type doesn't match the expected type based on https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html

Looks like it was a number before 7.0: https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking-changes-7.0.html#hits-total-now-object-search-response

The types package is out of date by over a year.

There's an issue to include up-to-date types with the client, but nothing has been merged. We may be better off preferring to use our own SearchResponse type in general over the unmaintained SearchResponse from ES 5 - until there are centrally maintained response types.

Yeah, this has been an issue for me in the past too. I think we're on our own until those are updated...

rylnd · 2020-11-25T21:27:40Z

x-pack/plugins/security_solution/server/lib/detection_engine/signals/filter_events_with_list.ts

+  // TS does not allow .every to be called on val as-is, even though every type in the union
+  // is an array. https://github.com/microsoft/TypeScript/issues/36390
+  // eslint-disable-next-line @typescript-eslint/no-explicit-any
+  return (val as any[]).every((subVal: string | number | boolean | object) =>


If we get rid of the as any[] and leave subVal implicitly typed, could we change the comment to a @ts-expect-error instead?

rylnd · 2020-11-25T21:29:44Z

x-pack/plugins/security_solution/server/lib/detection_engine/signals/filter_events_with_list.ts

@@ -71,13 +80,19 @@ export const createSetToFilterAgainst = async ({
  return matchedListItemsSet;
 };

-export const filterEventsAgainstList = async ({
+export const filterEventsAgainstList = async <T>({


rylnd · 2020-11-25T21:33:31Z

x-pack/plugins/security_solution/server/lib/detection_engine/signals/filter_events_with_list.ts

+      if (isStringableType(valueField)) {
+        acc.add(valueField.toString());
+      } else if (isStringableArray(valueField)) {
+        valueField.forEach((subVal: string | number | boolean) => acc.add(subVal.toString()));


Annotation of subVal here shouldn't be necessary, but maybe I'm missing something...

You're right, I think I added this before changing the return type of isStringableArray from val is string[] | number[] | boolean[] to Array<string | number | boolean and forgot to change it

rylnd · 2020-11-25T21:48:49Z

x-pack/plugins/security_solution/server/lib/detection_engine/signals/signal_rule_alert_type.ts

@@ -273,15 +283,16 @@ export const signalRulesAlertType = ({
          });
          // The legacy ES client does not define failures when it can be present on the structure, hence why I have the & { failures: [] }


It looks like our local, artisanal SearchResponse type now makes this comment/code obsolete, so that answers part of my earlier question. I'd still like to be able to extend the upstream type rather than diverge entirely, though, if at all possible.

rylnd · 2020-11-25T21:59:40Z

x-pack/plugins/security_solution/server/lib/machine_learning/index.ts

@@ -49,6 +52,17 @@ export const getAnomalies = async (
                },
              },
            ],
+            must_not: buildExceptionFilter({


I see in other places we're conditionally adding this filter based on the length of exceptionItems; does specifying must_not: undefined work in the general case here?

Yeah, the logic around those conditions is to prevent an empty object from showing up in the queries, like

{ bool: {} }

Here if the exception filter comes back undefined then must_not will be omitted from the final query that gets sent to ES, but the filter inside bool will always exist so we won't end up with any empty objects.

…l-exceptions

marshallmain · 2020-11-26T00:00:36Z

@elasticmachine merge upstream

marshallmain · 2020-11-30T15:31:57Z

@elasticmachine merge upstream

spong · 2020-11-30T21:55:33Z

Looks like the exception modal is falling back to using the securitySolution:defaultIndex indices (for autocomplete/validation) since we're not setting index on ML Rules when we create them. There's two touch points here, one via Alert Actions, and the other from Rule Details:

Alert Actions:

kibana/x-pack/plugins/security_solution/public/detections/components/alerts_table/timeline_actions/alert_context_menu.tsx

Lines 77 to 87 in 87f817e

    
           const ruleIndices = useMemo((): string[] => { 
        
             if ( 
        
               ecsRowData.signal?.rule && 
        
               ecsRowData.signal.rule.index && 
        
               ecsRowData.signal.rule.index.length > 0 
        
             ) { 
        
               return ecsRowData.signal.rule.index; 
        
             } else { 
        
               return DEFAULT_INDEX_PATTERN; 
        
             } 
        
           }, [ecsRowData]);

Rule Details Action:

kibana/x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/details/index.tsx

Line 557 in 87f817e

ruleIndices={rule?.index ?? DEFAULT_INDEX_PATTERN}

I think we can go ahead and fix this for future rules by setting the index when creating the rule, as the underlying ml-results-index shouldn't change after that (pretty sure you've gotta re-create the job to change the index). That said, the index doesn't seem to be provided in any of the ML calls we make on the rule creation page, so we may need to fetch it, as there's no guarantee they're using a dedicated index / they could be using a custom results index.

That'll fix it for future rules, but existing ones will still have this issue. As opposed to modifying the above touch points, perhaps we query for the ML index within initial load of the AddExceptionModal? Thoughts?

spong

Checked out and desk tested a bunch of different scenarios (individual exception items, value-lists, against array fields, etc) and exceptions seem to be working great! 🙌 Only issue I encountered was in adding exceptions via the modal as we don't send the ml index to the modal so field validation won't pass (more details in my other comment). Going to request changes for fixing this, but once fixed this LGTM! 👍

I mentioned a spot or two for how we could fix the issue, but let me know if you have any other ideas here, and happy to pair on this one as well since it's a bit more UI-related, just let me know! 🙂

marshallmain · 2020-12-01T06:15:45Z

@spong I'm leaning towards fetching the ML index on initial load. That'll work for existing rules and new rules and also avoid the possibility of the index stored in the rule getting out of sync with the ML job if the ML team adds the ability to change the anomaly index of a job in the future. I'll try to implement that tomorrow.

…tocomplete

peluja1012 · 2020-12-02T12:15:34Z

.../plugins/security_solution/public/common/components/exceptions/add_exception_modal/index.tsx

+    }
+  }, [jobs, ruleIndices]);
+
+  const [isIndexPatternLoading, { indexPatterns }] = useFetchIndex(memoRuleIndices);


You'll want to add this logic to the Edit Modal component as well edit_exception_moddal/index.tsx

marshallmain · 2020-12-02T15:58:49Z

@elasticmachine merge upstream

kibanamachine · 2020-12-02T17:39:02Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 61299d5

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	2093	2095	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	8.0MB	8.0MB	+5.3KB

History

💚 Build #91386 succeeded fe820c3
💚 Build #91273 succeeded 0b3c064
💚 Build #90740 succeeded 87f817e
💚 Build #90358 succeeded 5e5a9b0
💚 Build #89672 succeeded 50306cb

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

spong

Checked out the latest changes and verified all touch points of the exceptions modal now populate with the fields/values from the corresponding ml index. Thanks for all the goodness here @marshallmain! LGTM! 🙂 🎉

Note: Stumbled across a similar issue on rule create/edit with the override suggestions -- I've opened #84836 for that.

…astic#84006) * Implement exceptions for ML rules * Remove unused import * Better implicit types * Retrieve ML rule index pattern for exception field suggestions and autocomplete * Add ML job logic to edit exception modal * Remove unnecessary logic change Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

…4006) (#84841) * Implement exceptions for ML rules * Remove unused import * Better implicit types * Retrieve ML rule index pattern for exception field suggestions and autocomplete * Add ML job logic to edit exception modal * Remove unnecessary logic change Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

* master: (40 commits) fix: 🐛 don't add separator befor group on no main items (elastic#83166) [Security Solution][Detections] Implements indicator match rule cypress test (elastic#84323) [APM] Add APM agent config options (elastic#84678) Fixed a11y issue on rollup jobs table selection (elastic#84567) [Discover] Refactor getContextUrl to separate file (elastic#84503) [Embeddable] Export CSV action for Lens embeddables in dashboard (elastic#83654) [TSVB] [Cleanup] Remove extra dateFormat props (elastic#84749) [Lens] Migrate legacy es client and remove total hits as int (elastic#84340) Improve logging pipeline in @kbn/legacy-logging (elastic#84629) Catch @hapi/podium errors (elastic#84575) [Discover] Unskip date histogram test (elastic#84727) Rename server.xsrf.whitelist to server.xsrf.allowlist (elastic#84791) [Enterprise Search] Fix schema errors button (elastic#84842) [APM] Removes react-sticky dependency in favor of using CSS (elastic#84589) [Maps] Always initialize routes on server-startup (elastic#84806) [Fleet] EPM support to handle uploaded file paths (elastic#84708) [Snapshot Restore] Fix initial policy form state (elastic#83928) Upgrade Node.js to version 14 (elastic#83425) [Security Solution] Keep Endpoint policies up to date with license changes (elastic#83992) [Security Solution][Exceptions] Implement exceptions for ML rules (elastic#84006) ...

* master: (236 commits) fix: 🐛 don't add separator befor group on no main items (elastic#83166) [Security Solution][Detections] Implements indicator match rule cypress test (elastic#84323) [APM] Add APM agent config options (elastic#84678) Fixed a11y issue on rollup jobs table selection (elastic#84567) [Discover] Refactor getContextUrl to separate file (elastic#84503) [Embeddable] Export CSV action for Lens embeddables in dashboard (elastic#83654) [TSVB] [Cleanup] Remove extra dateFormat props (elastic#84749) [Lens] Migrate legacy es client and remove total hits as int (elastic#84340) Improve logging pipeline in @kbn/legacy-logging (elastic#84629) Catch @hapi/podium errors (elastic#84575) [Discover] Unskip date histogram test (elastic#84727) Rename server.xsrf.whitelist to server.xsrf.allowlist (elastic#84791) [Enterprise Search] Fix schema errors button (elastic#84842) [APM] Removes react-sticky dependency in favor of using CSS (elastic#84589) [Maps] Always initialize routes on server-startup (elastic#84806) [Fleet] EPM support to handle uploaded file paths (elastic#84708) [Snapshot Restore] Fix initial policy form state (elastic#83928) Upgrade Node.js to version 14 (elastic#83425) [Security Solution] Keep Endpoint policies up to date with license changes (elastic#83992) [Security Solution][Exceptions] Implement exceptions for ML rules (elastic#84006) ...

Implement exceptions for ML rules

8525db7

marshallmain requested review from a team as code owners November 20, 2020 23:54

marshallmain added the release_note:enhancement label Nov 20, 2020

Merge branch 'master' into ml-exceptions

09682d5

spong requested a review from a team November 21, 2020 01:00

Remove unused import

7a20f78

Merge branch 'master' into ml-exceptions

50306cb

rylnd reviewed Nov 25, 2020

View reviewed changes

marshallmain added 2 commits November 25, 2020 18:59

Better implicit types

c5ad7c0

Merge branch 'ml-exceptions' of github.com:marshallmain/kibana into m…

abbc51a

…l-exceptions

Merge branch 'master' into ml-exceptions

5e5a9b0

Merge branch 'master' into ml-exceptions

87f817e

spong mentioned this pull request Nov 30, 2020

[Security Solution] Re-validate rule exceptions when updating pre-packaged rules #84385

Closed

spong requested changes Dec 1, 2020

View reviewed changes

Retrieve ML rule index pattern for exception field suggestions and au…

0b3c064

…tocomplete

peluja1012 reviewed Dec 2, 2020

View reviewed changes

marshallmain added 2 commits December 2, 2020 10:20

Add ML job logic to edit exception modal

d52702c

Remove unnecessary logic change

fe820c3

Merge branch 'master' into ml-exceptions

61299d5

spong approved these changes Dec 2, 2020

View reviewed changes

marshallmain merged commit d47c70c into elastic:master Dec 2, 2020

marshallmain deleted the ml-exceptions branch December 2, 2020 22:14

marshallmain mentioned this pull request Dec 2, 2020

[7.x] [Security Solution][Exceptions] Implement exceptions for ML rules (#84006) #84841

Merged

peluja1012 mentioned this pull request Dec 3, 2020

[Security Solution][Detections] Add support for exceptions to ML Rules #75820

Closed

2 tasks

dplumlee mentioned this pull request Dec 15, 2020

[Security Solution] Update/unskip event viewer action tests to match new format #75934

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Exceptions] Implement exceptions for ML rules #84006

[Security Solution][Exceptions] Implement exceptions for ML rules #84006

marshallmain commented Nov 20, 2020

marshallmain commented Nov 23, 2020

rylnd left a comment

rylnd Nov 25, 2020

rylnd Nov 25, 2020

marshallmain Nov 25, 2020

marshallmain Nov 25, 2020 •

edited

rylnd Nov 30, 2020

marshallmain Dec 1, 2020

madirey Dec 6, 2020

rylnd Nov 25, 2020

rylnd Nov 25, 2020

rylnd Nov 25, 2020

marshallmain Nov 25, 2020

rylnd Nov 25, 2020

rylnd Nov 25, 2020

marshallmain Nov 25, 2020

marshallmain commented Nov 26, 2020

marshallmain commented Nov 30, 2020

spong commented Nov 30, 2020

spong left a comment

marshallmain commented Dec 1, 2020

peluja1012 Dec 2, 2020

marshallmain commented Dec 2, 2020

kibanamachine commented Dec 2, 2020

spong left a comment

		@@ -273,15 +283,16 @@ export const signalRulesAlertType = ({
		});
		// The legacy ES client does not define failures when it can be present on the structure, hence why I have the & { failures: [] }

[Security Solution][Exceptions] Implement exceptions for ML rules #84006

[Security Solution][Exceptions] Implement exceptions for ML rules #84006

Conversation

marshallmain commented Nov 20, 2020

Summary

Checklist

For maintainers

marshallmain commented Nov 23, 2020

rylnd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marshallmain Nov 25, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

marshallmain commented Nov 26, 2020

marshallmain commented Nov 30, 2020

spong commented Nov 30, 2020

spong left a comment

Choose a reason for hiding this comment

marshallmain commented Dec 1, 2020

Choose a reason for hiding this comment

marshallmain commented Dec 2, 2020

kibanamachine commented Dec 2, 2020

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

History

spong left a comment

Choose a reason for hiding this comment

marshallmain Nov 25, 2020 •

edited