[Security Solution] Add the ability to filter by index pattern to the rules management table #128245

xcrzx · 2022-03-22T10:36:15Z

Partially addresses: #125596, https://github.com/elastic/security-team/issues/1972
Resolves: #126166

Summary

Added the ability to sort detection rules by index pattern and MITRE ATT&CK tactics or techniques.

1. Filter by index pattern

2. Filter by MITRE ATT&CK tactic or technique name

3. Filter by MITRE ATT&CK tactic or technique id

Other changes

In the in-memory implementation, rules are not filtered on the client-side anymore. Instead, we fetch filtered rules from the server and only sort and paginate them in memory. This change was necessary to get rid of discrepancies between the two implementations. See #126166 for more detail.

elasticmachine · 2022-03-24T13:44:55Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2022-03-24T13:44:56Z

Pinging @elastic/security-solution (Team: SecuritySolution)

xcrzx · 2022-03-24T17:01:29Z

@yiyangliu9286 As a follow-up to our discussion, could you please review the changes to the rules table header and filters?

I removed the "All rules" title
Extended the search bar to the full width
Moved the "last updated" label to the right side

I kept the refresh button as it was before as there's no ready-to-use UI component at the moment (or I am missing something 🤔). So I would propose implementing it separately and updating all our tables at once.

Also, I kept the Custom/Elastic rules toggle in its place as there are not too many other filters, so enough space. But please let me know if you think we should move that toggle.

yiyangliu9286 · 2022-03-24T17:14:12Z

@yiyangliu9286 As a follow-up to our discussion, could you please review the changes to the rules table header and filters?

I removed the "All rules" title

Extended the search bar to the full width

Moved the "last updated" label to the right side

I kept the refresh button as it was before as there's no ready-to-use UI component at the moment (or I am missing something 🤔). So I would propose implementing it separately and updating all our tables at once.

Also, I kept the Custom/Elastic rules toggle in its place as there are not too many other filters, so enough space. But please let me know if you think we should move that toggle.

Hi @xcrzx! Thanks so much for this quick implementation updates, this looks awesome!

I kept the refresh button as it was before as there's no ready-to-use UI component at the moment (or I am missing something 🤔). So I would propose implementing it separately and updating all our tables at once.

I agree that we can keep the current table actions as is because the new table utility bar component haven't been made to an EUI component yet (as far as I know of). One small question, do you think if it makes sense to implement the "Auto refresh" and the "Refresh" button as part of this release's changes, since the EuiAutoRefreshButton component is ready to use (i know that in our case, we need some customization to the existing component). Or we should make this as an UX/UI enhancement as part of the 8.3 cycle? I am happy for either decision based on your time and capacity.

banderror · 2022-03-24T18:31:06Z

One small question, do you think if it makes sense to implement the "Auto refresh" and the "Refresh" button as part of this release's changes, since the EuiAutoRefreshButton component is ready to use (i know that in our case, we need some customization to the existing component). Or we should make this as an UX/UI enhancement as part of the 8.3 cycle?

Let's work on it separately and ship in a different PR to keep our PRs single-purpose.

@yiyangliu9286 It would be super helpful if we had a ticket describing the design changes that we'd like to apply to the Rule Management and Monitoring tables. While it's clear how we could move filters from one place on the table to another, I'd be willing to sync on the new table refresh button in more detail: its behavior and UI states.

banderror · 2022-03-24T18:36:29Z

I kept the refresh button as it was before as there's no ready-to-use UI component at the moment (or I am missing something 🤔).

@xcrzx If <EuiAutoRefresh> is not available in Kibana, it could be that EUI folks have merged it to the EUI repo, but haven't upgraded EUI in Kibana yet. The next upgrade is planned to be done in 8.3 (#128313).

xcrzx · 2022-03-25T09:27:43Z

@xcrzx If <EuiAutoRefresh> is not available in Kibana, it could be that EUI folks have merged it to the EUI repo, but haven't upgraded EUI in Kibana yet. The next upgrade is planned to be done in 8.3 (#128313).

Thanks, @banderror! I couldn't find it in the docs for some reason. It looks like we can use it but need some exploration on how we wire it up with other components. Let's do that separately, as there's not much time left before FF.

kibana-ci · 2022-03-28T21:01:13Z

💛 Build succeeded, but was flaky

Test Failures

[job] [logs] Security Solution Tests / value lists user with restricted access role "before each" hook for "Does not allow a t1 analyst user to upload a value list"

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	4.8MB	4.8MB	-252.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`securitySolution`	441	440	-1

Total ESLint disabled count

id	before	after	diff
`securitySolution`	510	509	-1

History

💔 Build #34035 failed a9c358d
💚 Build #33883 succeeded 5d600d1
💔 Build #33748 failed 04ba1d2
💔 Build #33595 failed c282980
💛 Build #33477 was flaky 8ef2d28

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @xcrzx

spong · 2022-03-28T22:50:08Z

x-pack/plugins/security_solution/cypress/integration/detection_rules/sorting.spec.ts

+        // Check that the rules table shows at least one row
+        cy.get(RULES_TABLE).find(RULES_ROW).should('have.length.gte', 1);
+        // Check that the rules table doesn't show the rule from the first page
+        cy.get(RULES_TABLE).should('not.contain', ruleNameFirstPage);


cc @MadameSheema -- not sure if you're still tracking cypress changes, but here's a few more. Let me know if you'd like to stop receiving these pings! 🙂

spong · 2022-03-28T22:57:13Z

x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/translations.ts

@@ -408,7 +401,7 @@ export const SEARCH_RULES = i18n.translate(
 export const SEARCH_PLACEHOLDER = i18n.translate(
  'xpack.securitySolution.detectionEngine.rules.allRules.searchPlaceholder',
  {
-    defaultMessage: 'e.g. rule name',
+    defaultMessage: 'Search by rule name, index pattern, or MITRE ATT&CK tactic or technique',


May want the ™ as we're adding it elsewhere in the UI:

Suggested change

defaultMessage: 'Search by rule name, index pattern, or MITRE ATT&CK tactic or technique',

defaultMessage: 'Search by rule name, index pattern, or MITRE ATT&CK™ tactic or technique',

spong · 2022-03-28T23:06:40Z

x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/utils.ts

+const SEARCHABLE_RULE_PARAMS = [
+  'alert.attributes.name',
+  'alert.attributes.params.index',
+  'alert.attributes.params.threat.tactic.id',
+  'alert.attributes.params.threat.tactic.name',
+  'alert.attributes.params.threat.technique.id',
+  'alert.attributes.params.threat.technique.name',
+];


Was subtechnique not added as a searchable param? You know if there was any particular reason?

Good catch, thanks, @spong! I'll add subtechnique to the searchable params as well.

spong · 2022-03-28T23:09:34Z

x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/utils.ts

+  const filters: string[] = [];
+
+  if (showCustomRules) {
+    filters.push(`alert.attributes.tags: "${INTERNAL_IMMUTABLE_KEY}:false"`);
+  }


This function is far more readable -- thank you @xcrzx!

spong · 2022-03-28T23:10:31Z

x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/utils.ts

+
+  if (tags.length > 0) {
+    filters.push(
+      `alert.attributes.tags:(${tags.map((tag) => `"${escapeKuery(tag)}"`).join(' AND ')})`


++ for bringing in the common escapeKuery 👍

spong · 2022-03-28T23:23:50Z

...ty_solution/public/detections/pages/detection_engine/rules/all/rules_table/use_find_rules.ts

+    {
+      refetchInterval,
+      enabled: isInMemorySorting,
+      keepPreviousData: true, // Use this option so that the state doesn't jump between "success" and "loading" on page change


Thank-you for the comment here 👍

spong · 2022-03-28T23:35:17Z

x-pack/plugins/security_solution/public/detections/pages/detection_engine/rules/translations.ts

@@ -408,7 +401,7 @@ export const SEARCH_RULES = i18n.translate(
 export const SEARCH_PLACEHOLDER = i18n.translate(
  'xpack.securitySolution.detectionEngine.rules.allRules.searchPlaceholder',
  {
-    defaultMessage: 'e.g. rule name',
+    defaultMessage: 'Search by rule name, index pattern, or MITRE ATT&CK tactic or technique',


Also noticed a small UX item here: if a user copies the whole tactic/technique from Rule Details, ie. Defense Evasion (TA0005) and pastes that in they won't get any results since id/name are independent fields -- they would need to search Defense Evasion or TA0005. Not too much we could do here outside updating the placeholder text on the search bar to include an example maybe?

Yeah, I agree 👍 Providing an example would make this UI a bit friendlier to our users.

spong

Checked out, tested locally, and performed code review. LGTM @xcrzx! 👍

Couple small nits, and a potential copy change that may need to be updated, but everything else looks and tested great! Thanks for the additional cleanup you've provided along the way as well! 🙂

xcrzx · 2022-03-29T09:16:23Z

Thanks for your review, @spong! I'm going to add a tour showing the new capabilities in the next PR, so I will also include changes to copy and filter by subtechnique there.

xcrzx self-assigned this Mar 22, 2022

xcrzx force-pushed the index-pattern-filter branch 7 times, most recently from 0581c6a to 8ef2d28 Compare March 24, 2022 13:18

xcrzx marked this pull request as ready for review March 24, 2022 13:44

xcrzx requested a review from a team as a code owner March 24, 2022 13:44

xcrzx force-pushed the index-pattern-filter branch from 8ef2d28 to c282980 Compare March 24, 2022 16:49

yiyangliu9286 self-requested a review March 24, 2022 17:14

yiyangliu9286 approved these changes Mar 24, 2022

View reviewed changes

xcrzx force-pushed the index-pattern-filter branch 3 times, most recently from 5d600d1 to a9c358d Compare March 28, 2022 14:10

Add the ability to filter by index pattern to the rules management table

9f7d8f4

xcrzx force-pushed the index-pattern-filter branch from a9c358d to 9f7d8f4 Compare March 28, 2022 17:50

spong reviewed Mar 28, 2022

View reviewed changes

spong approved these changes Mar 28, 2022

View reviewed changes

xcrzx merged commit eb51ea6 into elastic:main Mar 29, 2022

xcrzx mentioned this pull request Mar 29, 2022

[Security Solution] Add a tour showing new rules search capabilities #128759

Merged

banderror mentioned this pull request Mar 30, 2022

[Security Solution] Add a tour showing new rules search capabilities #128849

Closed

xcrzx mentioned this pull request Mar 30, 2022

[Security Solution] Add a tour showing new rules search capabilities #128888

Merged

joepeeples mentioned this pull request Apr 7, 2022

[DOCS] Rules search and filtering: index pattern, MITRE ATT&CK elastic/security-docs#1819

Closed

4 tasks

joepeeples mentioned this pull request Apr 22, 2022

[DOCS] What's New in 8.2? elastic/security-docs#1827

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Add the ability to filter by index pattern to the rules management table #128245

[Security Solution] Add the ability to filter by index pattern to the rules management table #128245

xcrzx commented Mar 22, 2022 •

edited

elasticmachine commented Mar 24, 2022

elasticmachine commented Mar 24, 2022

xcrzx commented Mar 24, 2022 •

edited

yiyangliu9286 commented Mar 24, 2022

banderror commented Mar 24, 2022

banderror commented Mar 24, 2022

xcrzx commented Mar 25, 2022

kibana-ci commented Mar 28, 2022

ESLint disabled line counts

Total ESLint disabled count

spong Mar 28, 2022

spong Mar 28, 2022

spong Mar 28, 2022

xcrzx Mar 29, 2022

spong Mar 28, 2022

spong Mar 28, 2022

spong Mar 28, 2022

spong Mar 28, 2022

xcrzx Mar 29, 2022

spong left a comment

xcrzx commented Mar 29, 2022

	defaultMessage: 'Search by rule name, index pattern, or MITRE ATT&CK tactic or technique',
	defaultMessage: 'Search by rule name, index pattern, or MITRE ATT&CK™ tactic or technique',

[Security Solution] Add the ability to filter by index pattern to the rules management table #128245

[Security Solution] Add the ability to filter by index pattern to the rules management table #128245

Conversation

xcrzx commented Mar 22, 2022 • edited

Summary

1. Filter by index pattern

2. Filter by MITRE ATT&CK tactic or technique name

3. Filter by MITRE ATT&CK tactic or technique id

Other changes

elasticmachine commented Mar 24, 2022

elasticmachine commented Mar 24, 2022

xcrzx commented Mar 24, 2022 • edited

yiyangliu9286 commented Mar 24, 2022

banderror commented Mar 24, 2022

banderror commented Mar 24, 2022

xcrzx commented Mar 25, 2022

kibana-ci commented Mar 28, 2022

💛 Build succeeded, but was flaky

Test Failures

Metrics [docs]

Async chunks

ESLint disabled line counts

Total ESLint disabled count

History

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

spong left a comment

Choose a reason for hiding this comment

xcrzx commented Mar 29, 2022

xcrzx commented Mar 22, 2022 •

edited

xcrzx commented Mar 24, 2022 •

edited