Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security Solution] Alerts Show All Closed when We Want to See Only Open #159606

Closed
MakoWish opened this issue Jun 13, 2023 · 22 comments
Closed

[Security Solution] Alerts Show All Closed when We Want to See Only Open #159606

MakoWish opened this issue Jun 13, 2023 · 22 comments
Assignees
@logeekal
Labels
bug Fixes for quality problems that affect the customer experience fixed Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Threat Hunting:Investigations Security Solution Investigations Team Team:Threat Hunting Security Solution Threat Hunting Team v8.9.0

Comments

@MakoWish
Copy link

Describe the bug: Previously, you could view Open, Acknowledged, or Closed alerts. Now, if there are no Open alerts, you cannot select to only show Open alerts; effectively showing you all closed alerts.

Kibana/Elasticsearch Stack version: 8.8.1

Original install method (e.g. download page, yum, from source, etc.): Yum

Functional Area (e.g. Endpoint management, timelines, resolver, etc.): Detection Rule Alerts

Steps to reproduce:

  1. Have several Closed and Open alerts
  2. Select "Open" for the "Status"
  3. Close all Open alerts, and note the "Open" selection is struck-through and shows all closed alerts.

Current behavior: If there are no open alerts, you are shown all closed alerts.

Expected behavior: You should still have the ability to only show open alerts. If there are no open alerts, you should see 0, or "No results found." Showing closed alerts when you are only interested in seeing open alerts is very confusing.

Screenshots (if relevant):
no_open_shows_closed
@MakoWish MakoWish added bug Fixes for quality problems that affect the customer experience Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. triage_needed labels Jun 13, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@MadameSheema MadameSheema added Team:Threat Hunting Security Solution Threat Hunting Team Team:Threat Hunting:Investigations Security Solution Investigations Team labels Jun 13, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

@MadameSheema
Copy link
Member

@logeekal can you please take a look at the above when you have the chance? Thanks!

@MakoWish
Copy link
Author

MakoWish commented Jun 13, 2023 •
edited

To note, I also opened ticket 01386456 for this.

@logeekal
Copy link
Contributor

Sure. We are looking into it.

@logeekal logeekal mentioned this issue Jun 23, 2023
Merged
logeekal added a commit that referenced this issue Jul 4, 2023
@logeekal
[Security Solution][Fix] Alert Page Filter Controls should not ignore…
54e613b 
… Invalid Selections (#160374)

This PR changes how Alert Page Filter Controls Work. 

## Before

1. Filter Controls use to ignore invalid Selections. For example, if
User has selected `Open` as the filter, but there is actually no alert
with Status `Open`, filters would ignore that selection and would
proceed to show other alerts ( which are NOT `Open`) .

- It seemed it was confusing users. [This
bug](#159606) and [messages in
community
slack](https://elasticstack.slack.com/archives/CNRTGB9A4/p1686937708085629?thread_ts=1686841414.978319&cid=CNRTGB9A4)
are the examples. @paulewing also emphasized this.

   - Below video shows what I mean.
  

https://github.com/elastic/kibana/assets/7485038/01771587-e4e8-4331-9535-4ffa09877c02






## After

1. With this Change Control Filters no longer ignore invalid selection.
So if user has chosen to show only `Open` Alerts. Then that filter will
be taken into account even though no alert with `Open` exists and a
empty table will be show.

    - Here is a video to demonstrate it.
       

https://github.com/elastic/kibana/assets/7485038/62b17762-16c0-471f-8480-e9f46e2ca5ef
logeekal added a commit to logeekal/kibana that referenced this issue Jul 4, 2023
@logeekal
[Security Solution][Fix] Alert Page Filter Controls should not ignore…
3cc6e8a 
… Invalid Selections (elastic#160374)

This PR changes how Alert Page Filter Controls Work.

## Before

1. Filter Controls use to ignore invalid Selections. For example, if
User has selected `Open` as the filter, but there is actually no alert
with Status `Open`, filters would ignore that selection and would
proceed to show other alerts ( which are NOT `Open`) .

- It seemed it was confusing users. [This
bug](elastic#159606) and [messages in
community
slack](https://elasticstack.slack.com/archives/CNRTGB9A4/p1686937708085629?thread_ts=1686841414.978319&cid=CNRTGB9A4)
are the examples. @paulewing also emphasized this.

   - Below video shows what I mean.

https://github.com/elastic/kibana/assets/7485038/01771587-e4e8-4331-9535-4ffa09877c02

## After

1. With this Change Control Filters no longer ignore invalid selection.
So if user has chosen to show only `Open` Alerts. Then that filter will
be taken into account even though no alert with `Open` exists and a
empty table will be show.

    - Here is a video to demonstrate it.

https://github.com/elastic/kibana/assets/7485038/62b17762-16c0-471f-8480-e9f46e2ca5ef
(cherry picked from commit 54e613b)

# Conflicts:
#	x-pack/plugins/security_solution/cypress/e2e/investigations/alerts/detection_page_filters.cy.ts
logeekal added a commit to logeekal/kibana that referenced this issue Jul 4, 2023
@logeekal
[Security Solution][Fix] Alert Page Filter Controls should not ignore…
710ede7 
… Invalid Selections (elastic#160374)

This PR changes how Alert Page Filter Controls Work.

## Before

1. Filter Controls use to ignore invalid Selections. For example, if
User has selected `Open` as the filter, but there is actually no alert
with Status `Open`, filters would ignore that selection and would
proceed to show other alerts ( which are NOT `Open`) .

- It seemed it was confusing users. [This
bug](elastic#159606) and [messages in
community
slack](https://elasticstack.slack.com/archives/CNRTGB9A4/p1686937708085629?thread_ts=1686841414.978319&cid=CNRTGB9A4)
are the examples. @paulewing also emphasized this.

   - Below video shows what I mean.

https://github.com/elastic/kibana/assets/7485038/01771587-e4e8-4331-9535-4ffa09877c02

## After

1. With this Change Control Filters no longer ignore invalid selection.
So if user has chosen to show only `Open` Alerts. Then that filter will
be taken into account even though no alert with `Open` exists and a
empty table will be show.

    - Here is a video to demonstrate it.

https://github.com/elastic/kibana/assets/7485038/62b17762-16c0-471f-8480-e9f46e2ca5ef
(cherry picked from commit 54e613b)

# Conflicts:
#	x-pack/plugins/security_solution/cypress/e2e/investigations/alerts/detection_page_filters.cy.ts
logeekal added a commit that referenced this issue Jul 5, 2023
@logeekal
[8.9] [Security Solution][Fix] Alert Page Filter Controls should not …
6c67324 
…ignore Invalid Selections (#160374) (#161208)

# Backport

This will backport the following commits from `main` to `8.9`:
- [[Security Solution][Fix] Alert Page Filter Controls should not ignore
Invalid Selections
(#160374)](#160374)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Jatin
Kathuria","email":"jatin.kathuria@elastic.co"},"sourceCommit":{"committedDate":"2023-07-04T12:13:54Z","message":"[Security
Solution][Fix] Alert Page Filter Controls should not ignore Invalid
Selections (#160374)\n\nThis PR changes how Alert Page Filter Controls
Work. \r\n\r\n## Before\r\n\r\n1. Filter Controls use to ignore invalid
Selections. For example, if\r\nUser has selected `Open` as the filter,
but there is actually no alert\r\nwith Status `Open`, filters would
ignore that selection and would\r\nproceed to show other alerts ( which
are NOT `Open`) .\r\n\r\n- It seemed it was confusing users.
[This\r\nbug](#159606) and
[messages
in\r\ncommunity\r\nslack](https://elasticstack.slack.com/archives/CNRTGB9A4/p1686937708085629?thread_ts=1686841414.978319&cid=CNRTGB9A4)\r\nare
the examples. @paulewing also emphasized this.\r\n\r\n - Below video
shows what I mean.\r\n
\r\n\r\nhttps://github.com/elastic/kibana/assets/7485038/01771587-e4e8-4331-9535-4ffa09877c02\r\n\r\n\r\n\r\n\r\n\r\n\r\n##
After\r\n\r\n1. With this Change Control Filters no longer ignore
invalid selection.\r\nSo if user has chosen to show only `Open` Alerts.
Then that filter will\r\nbe taken into account even though no alert with
`Open` exists and a\r\nempty table will be show.\r\n\r\n - Here is a
video to demonstrate it.\r\n
\r\n\r\nhttps://github.com/elastic/kibana/assets/7485038/62b17762-16c0-471f-8480-e9f46e2ca5ef","sha":"54e613b1d9341497d40a2c671c527de18beec165","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Threat
Hunting:Investigations","ci:cloud-deploy","v8.9.0","v8.10.0"],"number":160374,"url":"#160374
Solution][Fix] Alert Page Filter Controls should not ignore Invalid
Selections (#160374)\n\nThis PR changes how Alert Page Filter Controls
Work. \r\n\r\n## Before\r\n\r\n1. Filter Controls use to ignore invalid
Selections. For example, if\r\nUser has selected `Open` as the filter,
but there is actually no alert\r\nwith Status `Open`, filters would
ignore that selection and would\r\nproceed to show other alerts ( which
are NOT `Open`) .\r\n\r\n- It seemed it was confusing users.
[This\r\nbug](#159606) and
[messages
in\r\ncommunity\r\nslack](https://elasticstack.slack.com/archives/CNRTGB9A4/p1686937708085629?thread_ts=1686841414.978319&cid=CNRTGB9A4)\r\nare
the examples. @paulewing also emphasized this.\r\n\r\n - Below video
shows what I mean.\r\n
\r\n\r\nhttps://github.com/elastic/kibana/assets/7485038/01771587-e4e8-4331-9535-4ffa09877c02\r\n\r\n\r\n\r\n\r\n\r\n\r\n##
After\r\n\r\n1. With this Change Control Filters no longer ignore
invalid selection.\r\nSo if user has chosen to show only `Open` Alerts.
Then that filter will\r\nbe taken into account even though no alert with
`Open` exists and a\r\nempty table will be show.\r\n\r\n - Here is a
video to demonstrate it.\r\n
\r\n\r\nhttps://github.com/elastic/kibana/assets/7485038/62b17762-16c0-471f-8480-e9f46e2ca5ef","sha":"54e613b1d9341497d40a2c671c527de18beec165"}},"sourceBranch":"main","suggestedTargetBranches":["8.9"],"targetPullRequestStates":[{"branch":"8.9","label":"v8.9.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"#160374
Solution][Fix] Alert Page Filter Controls should not ignore Invalid
Selections (#160374)\n\nThis PR changes how Alert Page Filter Controls
Work. \r\n\r\n## Before\r\n\r\n1. Filter Controls use to ignore invalid
Selections. For example, if\r\nUser has selected `Open` as the filter,
but there is actually no alert\r\nwith Status `Open`, filters would
ignore that selection and would\r\nproceed to show other alerts ( which
are NOT `Open`) .\r\n\r\n- It seemed it was confusing users.
[This\r\nbug](#159606) and
[messages
in\r\ncommunity\r\nslack](https://elasticstack.slack.com/archives/CNRTGB9A4/p1686937708085629?thread_ts=1686841414.978319&cid=CNRTGB9A4)\r\nare
the examples. @paulewing also emphasized this.\r\n\r\n - Below video
shows what I mean.\r\n
\r\n\r\nhttps://github.com/elastic/kibana/assets/7485038/01771587-e4e8-4331-9535-4ffa09877c02\r\n\r\n\r\n\r\n\r\n\r\n\r\n##
After\r\n\r\n1. With this Change Control Filters no longer ignore
invalid selection.\r\nSo if user has chosen to show only `Open` Alerts.
Then that filter will\r\nbe taken into account even though no alert with
`Open` exists and a\r\nempty table will be show.\r\n\r\n - Here is a
video to demonstrate it.\r\n
\r\n\r\nhttps://github.com/elastic/kibana/assets/7485038/62b17762-16c0-471f-8480-e9f46e2ca5ef","sha":"54e613b1d9341497d40a2c671c527de18beec165"}}]}]
BACKPORT-->
@PhilippeOberti
Copy link
Contributor

@MakoWish this should be fixed in 8.9 )(see this PR)

@karanbirsingh-qasource when you get a chance, could you verify that the issue has been correctly fixed?

@karanbirsingh-qasource
Copy link

Sure @PhilippeOberti thanks for the update.

For now issue is still occuring on 8.9 BC2 However we will recheck the same issue on 8.9 BC3 and share the update here.

Alerts.-.Kibana.Mozilla.Firefox.2023-07-06.09-58-06.mp4

@logeekal
Copy link
Contributor

@karanbirsingh-qasource Could you please try it on BC3? It should be fixed.

@karanbirsingh-qasource
Copy link

Hi @logeekal

The Issue has been Fixed on 8.9.0 BC3 ✔️ . On Closing all open alert filter retains to open state and we get a banner No result matching search criteria.

Build Details:

Version: 8.9.0
Commit: fc463b96275c55dc44524f79f617b0026b7f8667
Build: 64584

Screen-Cast:

Alerts.-.Kibana.Mozilla.Firefox.2023-07-12.16-36-51.mp4

Hence we are closing this issue and adding "QA:Validated" tag to it.

thanks !!

@MakoWish
Copy link
Author

What is the default filter when navigating to the Alerts page after this change? If there are no open alerts, will it by default show "No results match your search criteria" like it used to? Or if there are no open alerts to show, will it undesirably show all closed alerts? I only ever care about closed alerts if I need to look at something historically. 99.999% of the time, I could care less about closed alerts; that's why they're closed.

@logeekal
Copy link
Contributor

logeekal commented Jul 13, 2023 •
edited

@MakoWish

What is the default filter when navigating to the Alerts page after this change? If there are no open alerts, will it by default show "No results match your search criteria" like it used to?

Yes, default filter remains as Open and it will show No results match your search criteria.

Or if there are no open alerts to show, will it undesirably show all closed alerts?

This was previous behaviour which has been removed.

@MakoWish
Copy link
Author

MakoWish commented Aug 16, 2023 •
edited

There still seems to be an issue here. I have closed all instances of "Privileged Account Brute Force", but they are still showing even when I have the filter set to "Open".

closed_still_showing

And why are they highlighted a brownish-green color? Only alerts that are closed (and falsely showing here when they should not be) are highlighted like that. Alerts that are still open, and are properly showing, are not highlighted like that.

closed_versus_open_highlighting

@logeekal
Copy link
Contributor

Hello @MakoWish ,

highlighted alerts mean that they are building block alerts. Can you check what is the value of this table filter as shown below. If it is checked, try unchecking it. It could be possible that that building block alerts were not closed.

image

Additionally, I tried to reproduce this type of scenario but I could not. Could you please add kibana.alert.workflow_status column in table for clarity?

@MakoWish
Copy link
Author

MakoWish commented Aug 21, 2023 •
edited

The Privileged Account Brute Force rule is not actually a building block rule (as far as I can tell), so that is strange, but unchecking that box did hide those highlighted alerts.

It seems the root issue here has still not been resolved, however. I am still not seeing an option to select only "Open" alerts, and it is therefore showing me all alerts again.

no_filter_for_open

@MakoWish
Copy link
Author

To add... I also noticed last week that the "Toggle column in table" link is no longer working. It does not actually add the column to the table anymore. I have been meaning to open a support ticket for this, but figured I would mention it here as well since you are asking me to add that column.

toggle_not_working

@logeekal
Copy link
Contributor

The Privileged Account Brute Force rule is not actually a building block rule (as far as I can tell), so that is strange, but unchecking that box did hide those highlighted alerts.

Could you please check again the rule details, it has to be building block ( or may be it was in past ) which created those building block alerts.

It seems the root issue here has still not been resolved, however. I am still not seeing an option to select only "Open" alerts, and it is therefore showing me all alerts again.

This is a different issue and It seems that the previous issue where Open is selected and it shows all alerts does not exists. Correct me if I am wrong.

Now in this case, it seems Open was removed from the drop down using clear button. Drop-down only shows the status values that available in elastic. Currently, there is no Open Alert so you will not be able to remove/add open from the drop-down.

There are now 2 ways to choose OpenAlert:

  1. Once you have at least one open alert, you will be able to select Open in the drop down and it will always remain there, unless you remove it.
  2. Secondly, On the right side, there is a context menu ( three dots. ), click on that and then Reset controls. this will bring Open back. Please keep in mind that if you remove Open from the drop-down, you will only be able to add it again if there are any open alerts.

One good strategy is to keep Open always selected.

To add... I also noticed last week that the "Toggle column in table" link is no longer working. It does not actually add the column to the table anymore. I have been meaning to open a support ticket for this, but figured I would mention it here as well since you are asking me to add that column.

We already have ticket for it: #155243

@MakoWish
Copy link
Author

@logeekal

Could you please check again the rule details, it has to be building block ( or may be it was in past ) which created those building block alerts.

It sure looks like a standard EQL query to me. Those alerts that were highlighted in the above screenshot are not building blocks, but the actual alerts. They were not highlighted until I closed them. After I closed them, they remained in view but were highlighted.

Now in this case, it seems Open was removed from the drop down using clear button.

I didn't remove the filter, but I selected "Closed" to see if the alerts were showing highlighted there since I did actually close them. They were showing there as closed, but they were not highlighted. I then wanted to revert back to showing only "Open" alerts (none currently, but expecting to see those highlighted closed alerts still), but the option to select "Open" did not exist.

Drop-down only shows the status values that available in elastic. Currently, there is no Open Alert so you will not be able to remove/add open from the drop-down.

This needs to be changed. Not having a way to go back to showing only "Open" alerts is a problem. The previous design of having "tabs" for "Open", "Acknowledged", or "Closed" worked perfectly fine. I don't know why a decision was made to use the drop-down based only on available values. Every detection MUST have a status of Open, Acknowledged, or Closed, so why can that not be hard-coded into the drop-down?

One good strategy is to keep Open always selected.

Sorry, but saying not to check Ackowledged or Closed alerts is not a solution to being able to revert back to seeing only Open alerts.

Eric

@logeekal
Copy link
Contributor

logeekal commented Aug 22, 2023 •
edited

They were not highlighted until I closed them. After I closed them, they remained in view but were highlighted.

This should not happen, would you be able to open a bug with reproduction steps, we will happy to look into it.

I didn't remove the filter, but I selected "Closed" to see if the alerts were showing highlighted there since I did actually close them. They were showing there as closed, but they were not highlighted. I then wanted to revert back to showing only "Open" alerts (none currently, but expecting to see those highlighted closed alerts still), but the option to select "Open" did not exist.

This needs to be changed. Not having a way to go back to showing only "Open" alerts is a problem. The previous design of having "tabs" for "Open", "Acknowledged", or "Closed" worked perfectly fine. I don't know why a decision was made to use the drop-down based only on available values. Every detection MUST have a status of Open, Acknowledged, or Closed, so why can that not be hard-coded into the drop-down?

Sorry, but saying not to check Ackowledged or Closed alerts is not a solution to being able to revert back to seeing only Open alerts.

I understand what you are saying but I am not suggesting not to check Acknowleged or Closed Alerts. Consider below workflow. Here I was able to include/exclude closed alerts without removing Open.

Screen.Recording.2023-08-22.at.16.51.51.mov

This is a new feature and we are still working to refine the workflow and currently, we do not have an elegant solution to this problem that event if alerts are not available, we still show the status. Currently, all drop downs are created on the basis of the data available.

Edit:

I don't know why a decision was made to use the drop-down based only on available values. Every detection MUST have a status of Open, Acknowledged, or Closed, so why can that not be hard-coded into the drop-down?

This is because, drop downs are configurable and you can add any field to the drop down. We will see what is the best way to add hard coded values (eg. status and severity ) and it is on the roadmap but we will need to see how.

@MakoWish
Copy link
Author

You should honestly just revert back to the old method. It worked, it was easy, and it had no issues. This is still used on the individual rules, and again works perfectly fine, but it is no longer on the global "Alerts" page. This new method with the dropdown is just not working the way I believe it was intended.

old_options

@logeekal
Copy link
Contributor

logeekal commented Aug 22, 2023 •
edited

@MakoWish , Are you on Elastic Community slack? It will be great for you to voice your concerns there as there are other users as well who are using this feature in variety of ways and product members as well who will be happy to take any feedback.

Please raise the bug regarding highlighted fields with reproduction steps as well. Thanks.

@MakoWish
Copy link
Author

I am on the Slack channel. Is there an existing thread for this discussion?

@logeekal
Copy link
Contributor

after 8.9, not that I am aware of.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@logeekal logeekal

Labels
bug Fixes for quality problems that affect the customer experience fixed Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Threat Hunting:Investigations Security Solution Investigations Team Team:Threat Hunting Security Solution Threat Hunting Team v8.9.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@logeekal @elasticmachine @PhilippeOberti @MadameSheema @MakoWish @karanbirsingh-qasource