[Security Solution] [Endpoint] Allow filtering activity log with date range #104085

ashokaditya · 2021-07-01T11:54:55Z

Summary

Allow users to fetch activity logs where actions and responses have a timestamp >= start date, <= end date or with a date range where start and end dates are included.

Changes:
Frontend

Added date range filter within the activity log flyout tab
The date range picker sticks to the bottom of the flyout header
Asks for data whenever a start date, end date or both is selected
Ignores data fetch when end date is older than start date, i.e, if the date towards the end of the range is not greater than that in the start of the range. We call this invalid date ranges. Also doesn't make API request if paging is disabled.
Pagination is reset when dates are selected and progresses on scroll.
Pagination is not reset if date ranges are invalid.
Activity log shows nothing (not empty state) if date filters result in an empty list of actions and responses.
The date ranges are stored in the redux store under activityLog.paging

Backend

assumes that either start_date or end_date are present in the request query.
If start_date or end_date are present in the request query then elasticsearch query uses { range: { '@timestamp': { gte: startDate } } } and { range: { '@timestamp': { lte: endDate } } } respectively to query with those params.
The response returns the queried startDate, endDate or both, that is used to update the redux store in the frontend.

Clip

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

kevinlog · 2021-07-01T17:20:26Z

@ashokaditya - looks great! Since this is a new feature that isn't budgeted in the test plan, we'll delay it to 7.15. I just removed the 7.14 label.

fixes elastic/security-team/issues/1137

for some reason TS was complaining earlier with `undefined`

fixes elastic/security-team/issues/1137

refs elastic/security-team/issues/1137

fixes elastic/security-team/issues/1137

elasticmachine · 2021-07-06T11:53:42Z

Pinging @elastic/security-onboarding-and-lifecycle-mgt (Team:Onboarding and Lifecycle Mgt)

dasansol92

That looks awesome!! A few comments added but it looks like a very good job so far! 🔥

x-pack/plugins/security_solution/public/management/pages/endpoint_hosts/store/middleware.ts

...olution/public/management/pages/endpoint_hosts/view/details/components/date_range_picker.tsx

dasansol92 · 2021-07-06T12:04:02Z

...olution/public/management/pages/endpoint_hosts/view/details/components/date_range_picker.tsx

+import moment, { Moment } from 'moment';
+import { EuiFlexGroup, EuiFlexItem, EuiDatePicker, EuiDatePickerRange } from '@elastic/eui';
+
+import * as i18 from '../../translations';


Are you using all the translations there? Otherwise you should import just the needed ones

it's just the two translations, but they are deep nested within the export object.

...olution/public/management/pages/endpoint_hosts/view/details/components/date_range_picker.tsx

...urity_solution/public/management/pages/endpoint_hosts/view/details/endpoint_activity_log.tsx

refs f551b67

ashokaditya · 2021-07-07T09:03:39Z

@elasticmachine merge upstream

…ctions_log-1137

ashokaditya · 2021-07-08T08:12:32Z

@elasticmachine merge upstream

…ctions_log-1137

dasansol92

Few more comments added after the changes but in any case, this is looking 🔥

x-pack/plugins/security_solution/public/management/pages/endpoint_hosts/store/middleware.ts

...gement/pages/endpoint_hosts/view/details/components/activity_log_date_range_picker/index.tsx

…ponent review changes

ashokaditya · 2021-07-08T12:06:28Z

@elasticmachine merge upstream

…ctions_log-1137

dasansol92 · 2021-07-09T11:59:32Z

One last thing. I checked it locally and seems the date range inside the input is a little bit covered by the clear button.

Apart from that, when the search doesn't return any result the Nothing more to show text is outside the windows view and I've to scroll a bit to see that, so for a user perspective seems there is something wrong till it scroll down a bit:

ashokaditya · 2021-07-09T12:51:51Z

One last thing. I checked it locally and seems the date range inside the input is a little bit covered by the clear button.

Can't do much about that except create a ticket in the Eui repo for it. It's how the date picker works.

Apart from that, when the search doesn't return any result the Nothing more to show text is outside the windows view and I've to scroll a bit to see that, so for a user perspective seems there is something wrong till it scroll down a bit:

I see your point but I think this is okay. I'll make a tiny fix anyway so that we show this message right below the dates. Although I think it is okay as it is, and follows the same pattern when there's a small list of actions/responses then users have to scroll down to see that there's nothing more to see. I guess we can improve the UX or make it consistent later.

ashokaditya · 2021-07-09T12:54:03Z

@elasticmachine merge upstream

…ctions_log-1137

dasansol92 · 2021-07-09T13:05:45Z

One last thing. I checked it locally and seems the date range inside the input is a little bit covered by the clear button.

Can't do much about that except create a ticket in the Eui repo for it. It's how the date picker works.

I think this is because the inputs are too small, can we increase the input width?

Apart from that, when the search doesn't return any result the Nothing more to show text is outside the windows view and I've to scroll a bit to see that, so for a user perspective seems there is something wrong till it scroll down a bit:

I see your point but I think this is okay. I'll make a tiny fix anyway so that we show this message right below the dates. Although I think it is okay as it is, and follows the same pattern when there's a small list of actions/responses then users have to scroll down to see that there's nothing more to see. I guess we can improve the UX or make it consistent later.

Yeah, I think that make sense when there are few results and you have to scroll a bit, but when there is no results it feels weird having a full blank page and when you scroll then the message appears.

ashokaditya · 2021-07-09T13:40:20Z

One last thing. I checked it locally and seems the date range inside the input is a little bit covered by the clear button.

Can't do much about that except create a ticket in the Eui repo for it. It's how the date picker works.

I think this is because the inputs are too small, can we increase the input width?

set fullWidth={true} for the date range component will allow it to take the full width of the container. Let's do that.

review changes

ashokaditya · 2021-07-09T13:51:10Z

done @dasansol92

large screen

small screen

dasansol92

This looks awesome! Shiiiip this! 🔥 🚢 🥳
PD: About full width input, I found this mockups in figma with a full width input -> https://www.figma.com/file/BeTJy7bpHRPtHiNJtWU7Pj/OLM-7.12%2B?node-id=4835%3A558081

ashokaditya · 2021-07-09T13:59:59Z

This looks awesome! Shiiiip this! 🔥 🚢 🥳
PD: About full width input, I found this mockups in figma with a full width input -> https://www.figma.com/file/BeTJy7bpHRPtHiNJtWU7Pj/OLM-7.12%2B?node-id=4835%3A558081

Nice. I somehow missed that mock. 😅 I was looking at the earlier version where there's also a search bar.

kibanamachine · 2021-07-09T15:58:53Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	2205	2206	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	6.3MB	6.3MB	+6.0KB

History

💔 Build #137523 failed 3b52416
💚 Build #137217 succeeded 9af99dc
💔 Build #137206 failed 169d267
💔 Build #137182 failed 6f8e62d
💚 Build #137140 succeeded d9a27dd

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ashokaditya

… range (elastic#104085) * use date range in search query fixes elastic/security-team/issues/1137 * make any date selection fetch matching log fixes elastic/security-team/issues/1137 * use a single action for updating paging info and fetching data fixes elastic/security-team/issues/1137 * use consistent types for some reason TS was complaining earlier with `undefined` * reset date picker on tab load fixes elastic/security-team/issues/1137 * refactor date pickers into a component refs elastic/security-team/issues/1137 * clear dates on change of endpoint fixes elastic/security-team/issues/1137 * do not show empty state if date filtering results return empty data fixes elastic/security-team/issues/1137 * add tests fixes elastic/security-team/issues/1137 * review changes * update comment refs f551b67 * store invalidDateRange on redux store and decouple logic from the component review changes * fix test * fix lint * review changes * expand date picker to use the full width of the flyout review changes Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>

kibanamachine · 2021-07-09T16:26:51Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

… range (#104085) (#105093) * use date range in search query fixes elastic/security-team/issues/1137 * make any date selection fetch matching log fixes elastic/security-team/issues/1137 * use a single action for updating paging info and fetching data fixes elastic/security-team/issues/1137 * use consistent types for some reason TS was complaining earlier with `undefined` * reset date picker on tab load fixes elastic/security-team/issues/1137 * refactor date pickers into a component refs elastic/security-team/issues/1137 * clear dates on change of endpoint fixes elastic/security-team/issues/1137 * do not show empty state if date filtering results return empty data fixes elastic/security-team/issues/1137 * add tests fixes elastic/security-team/issues/1137 * review changes * update comment refs f551b67 * store invalidDateRange on redux store and decouple logic from the component review changes * fix test * fix lint * review changes * expand date picker to use the full width of the flyout review changes Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Ashokaditya <am.struktr@gmail.com>

ashokaditya added auto-backport Deprecated: Automatically backport this PR after it's merged release_note:feature Makes this part of the condensed release notes Team:Defend Workflows “EDR Workflows” sub-team of Security Solution v7.14.0 v7.15.0 v8.0.0 labels Jul 1, 2021

kevinlog removed the v7.14.0 label Jul 1, 2021

ashokaditya added 9 commits July 6, 2021 09:25

use date range in search query

a70c667

fixes elastic/security-team/issues/1137

make any date selection fetch matching log

041bde1

fixes elastic/security-team/issues/1137

use a single action for updating paging info and fetching data

716e77a

fixes elastic/security-team/issues/1137

use consistent types

2f1fd2e

for some reason TS was complaining earlier with `undefined`

reset date picker on tab load

9e7eb74

fixes elastic/security-team/issues/1137

refactor date pickers into a component

4f97e4b

refs elastic/security-team/issues/1137

clear dates on change of endpoint

b45ee96

fixes elastic/security-team/issues/1137

do not show empty state if date filtering results return empty data

bdeb266

fixes elastic/security-team/issues/1137

add tests

3c9dca1

fixes elastic/security-team/issues/1137

ashokaditya marked this pull request as ready for review July 6, 2021 11:53

ashokaditya requested a review from a team as a code owner July 6, 2021 11:53

dasansol92 reviewed Jul 6, 2021

View reviewed changes

review changes

f551b67

ashokaditya requested a review from dasansol92 July 6, 2021 16:58

update comment

94b2c4a

refs f551b67

Merge branch 'master' into feature/olm-allow_date_range_filtereting_a…

fcffbb8

…ctions_log-1137

ashokaditya self-assigned this Jul 7, 2021

Merge branch 'master' into feature/olm-allow_date_range_filtereting_a…

d9a27dd

…ctions_log-1137

dasansol92 reviewed Jul 8, 2021

View reviewed changes

store invalidDateRange on redux store and decouple logic from the com…

fe04361

…ponent review changes

ashokaditya requested a review from dasansol92 July 8, 2021 11:02

fix test

6f8e62d

kibanamachine and others added 2 commits July 8, 2021 08:06

Merge branch 'master' into feature/olm-allow_date_range_filtereting_a…

169d267

…ctions_log-1137

fix lint

9af99dc

review changes

11695c8

Merge branch 'master' into feature/olm-allow_date_range_filtereting_a…

3b52416

…ctions_log-1137

expand date picker to use the full width of the flyout

7ff3230

review changes

dasansol92 approved these changes Jul 9, 2021

View reviewed changes

ashokaditya merged commit 81f09a8 into elastic:master Jul 9, 2021

ashokaditya deleted the feature/olm-allow_date_range_filtereting_actions_log-1137 branch July 9, 2021 16:24

kibanamachine mentioned this pull request Jul 9, 2021

[7.x] [Security Solution] [Endpoint] Allow filtering activity log with date range (#104085) #105093

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] [Endpoint] Allow filtering activity log with date range #104085

[Security Solution] [Endpoint] Allow filtering activity log with date range #104085

ashokaditya commented Jul 1, 2021 •

edited

kevinlog commented Jul 1, 2021

elasticmachine commented Jul 6, 2021

dasansol92 left a comment

dasansol92 Jul 6, 2021

ashokaditya Jul 6, 2021

ashokaditya commented Jul 7, 2021

ashokaditya commented Jul 8, 2021

dasansol92 left a comment

ashokaditya commented Jul 8, 2021

dasansol92 commented Jul 9, 2021

ashokaditya commented Jul 9, 2021

ashokaditya commented Jul 9, 2021

dasansol92 commented Jul 9, 2021

ashokaditya commented Jul 9, 2021

ashokaditya commented Jul 9, 2021 •

edited

dasansol92 left a comment

ashokaditya commented Jul 9, 2021

kibanamachine commented Jul 9, 2021

kibanamachine commented Jul 9, 2021

[Security Solution] [Endpoint] Allow filtering activity log with date range #104085

[Security Solution] [Endpoint] Allow filtering activity log with date range #104085

Conversation

ashokaditya commented Jul 1, 2021 • edited

Summary

Checklist

Risk Matrix

kevinlog commented Jul 1, 2021

elasticmachine commented Jul 6, 2021

dasansol92 left a comment

Choose a reason for hiding this comment

dasansol92 Jul 6, 2021

Choose a reason for hiding this comment

ashokaditya Jul 6, 2021

Choose a reason for hiding this comment

ashokaditya commented Jul 7, 2021

ashokaditya commented Jul 8, 2021

dasansol92 left a comment

Choose a reason for hiding this comment

ashokaditya commented Jul 8, 2021

dasansol92 commented Jul 9, 2021

ashokaditya commented Jul 9, 2021

ashokaditya commented Jul 9, 2021

dasansol92 commented Jul 9, 2021

ashokaditya commented Jul 9, 2021

ashokaditya commented Jul 9, 2021 • edited

dasansol92 left a comment

Choose a reason for hiding this comment

ashokaditya commented Jul 9, 2021

kibanamachine commented Jul 9, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

History

kibanamachine commented Jul 9, 2021

💚 Backport successful

ashokaditya commented Jul 1, 2021 •

edited

ashokaditya commented Jul 9, 2021 •

edited