[ML] Delayed data test for Anomaly Detection jobs health rule type #107183

darnautov · 2021-07-29T15:13:47Z

Summary

Part of #101028

Adds the delayed data test for Anomaly Detection jobs health rule type.

Rule executor looks up for delayed_data events in the annotations index within the provided time interval.

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)

szabosteve

I left three minor suggestions.

x-pack/plugins/ml/public/alerting/jobs_health_rule/register_jobs_health_alerting_rule.ts

x-pack/plugins/ml/common/constants/alerts.ts

elasticmachine · 2021-08-02T11:31:08Z

Pinging @elastic/ml-ui (:ml)

szabosteve

UI text LGTM! Thanks!

…-data

walterra

Code LGTM

darnautov · 2021-08-04T14:38:18Z

@alvarezmelissa87 looks like you have some issues with the Kibana configuration, could be SSL related. Let me know if you need some help with the setup

benwtrent

minor things, but looking great

x-pack/plugins/ml/server/lib/alerts/jobs_health_service.ts

x-pack/plugins/ml/server/models/annotation_service/annotation.ts

benwtrent · 2021-08-04T15:24:44Z

x-pack/plugins/ml/server/lib/alerts/jobs_health_service.ts

+          const job = jobs.find((j) => j.job_id === v.job_id);
+          const datafeed = datafeeds?.find((d) => d.job_id === v.job_id);
+
+          const jobLookbackInterval = resolveLookbackInterval([job!], [datafeed!]);


👍 this is great.

Get them all in the same query, then filter down appropriately. good stuff.

qn895 · 2021-08-04T16:12:48Z

Small UI thing I found while testing - wildcard doesn't seem to work with prefixes.

Screen.Recording.2021-08-04.at.11.01.46.mov

darnautov · 2021-08-04T16:26:40Z

@qn895 hey! thanks for reviewing. We don't support wildcards for the jobs selection. Just a special case for all jobs (with * symbol)

…-data

x-pack/plugins/ml/common/constants/alerts.ts

szabosteve

Helper tooltips LGTM!

Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>

pheyos

Tested and LGTM

kibanamachine · 2021-08-05T14:31:48Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`ml`	5.9MB	6.0MB	+10.8KB

History

💚 Build #143192 succeeded 3afb856
💚 Build #143177 succeeded c57b07b
💚 Build #143026 succeeded 0968256
💚 Build #143000 succeeded 637cc90
💚 Build #142968 succeeded 1e8d5d5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @darnautov

…lastic#107183) * [ML] enable test selection * [ML] executor update for annotations * [ML] update unit tests * [ML] fix i18n * [ML] update schema * [ML] fix ts * [ML] account for docs count, update unit tests * [ML] update translation strings * [ML] add types * [ML] fetch the latest annotation sorted by modified_time * [ML] getDelayedDataAnnotations * [ML] update unit tests * [ML] set default number of docs to 1, update schema validation * [ML] getDelayedDataLookbackTimestamp * [ML] filter null values, update unit tests * [ML] account for query delay, refactor with memoize * [ML] update unit test * [ML] remove previousStartedAt * [ML] filter based on the job config * [ML] fix tests * [ML] add maps * [ML] combine filters * [ML] move range query inside of a filter * [ML] filter out jobs with missing datafeed * [ML] resolveLookbackInterval only from jobs with datafeeds * [ML] do not show an error on empty time interval * [ML] add help tooltips * [ML] update description for the datafeed check

kibanamachine · 2021-08-05T14:54:24Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

…107183) (#107762) * [ML] enable test selection * [ML] executor update for annotations * [ML] update unit tests * [ML] fix i18n * [ML] update schema * [ML] fix ts * [ML] account for docs count, update unit tests * [ML] update translation strings * [ML] add types * [ML] fetch the latest annotation sorted by modified_time * [ML] getDelayedDataAnnotations * [ML] update unit tests * [ML] set default number of docs to 1, update schema validation * [ML] getDelayedDataLookbackTimestamp * [ML] filter null values, update unit tests * [ML] account for query delay, refactor with memoize * [ML] update unit test * [ML] remove previousStartedAt * [ML] filter based on the job config * [ML] fix tests * [ML] add maps * [ML] combine filters * [ML] move range query inside of a filter * [ML] filter out jobs with missing datafeed * [ML] resolveLookbackInterval only from jobs with datafeeds * [ML] do not show an error on empty time interval * [ML] add help tooltips * [ML] update description for the datafeed check Co-authored-by: Dima Arnautov <dmitrii.arnautov@elastic.co>

…lastic#107183) * [ML] enable test selection * [ML] executor update for annotations * [ML] update unit tests * [ML] fix i18n * [ML] update schema * [ML] fix ts * [ML] account for docs count, update unit tests * [ML] update translation strings * [ML] add types * [ML] fetch the latest annotation sorted by modified_time * [ML] getDelayedDataAnnotations * [ML] update unit tests * [ML] set default number of docs to 1, update schema validation * [ML] getDelayedDataLookbackTimestamp * [ML] filter null values, update unit tests * [ML] account for query delay, refactor with memoize * [ML] update unit test * [ML] remove previousStartedAt * [ML] filter based on the job config * [ML] fix tests * [ML] add maps * [ML] combine filters * [ML] move range query inside of a filter * [ML] filter out jobs with missing datafeed * [ML] resolveLookbackInterval only from jobs with datafeeds * [ML] do not show an error on empty time interval * [ML] add help tooltips * [ML] update description for the datafeed check

darnautov added 2 commits July 29, 2021 11:34

[ML] enable test selection

bea342b

[ML] executor update for annotations

e4aee46

darnautov added :ml Feature:Anomaly Detection ML anomaly detection v8.0.0 auto-backport Deprecated: Automatically backport this PR after it's merged v7.15.0 Feature:Alerting/RuleTypes Issues related to specific Alerting Rules Types labels Jul 29, 2021

darnautov self-assigned this Jul 29, 2021

darnautov added 3 commits July 30, 2021 13:31

[ML] update unit tests

8e2c5d5

[ML] fix i18n

6979ad4

[ML] update schema

79e6bdf

darnautov requested review from peteharverson, szabosteve, pheyos and alvarezmelissa87 July 30, 2021 11:44

[ML] fix ts

856963a

szabosteve reviewed Jul 30, 2021

View reviewed changes

[ML] account for docs count, update unit tests

2979454

darnautov marked this pull request as ready for review August 2, 2021 11:31

darnautov requested a review from a team as a code owner August 2, 2021 11:31

[ML] update translation strings

8bc164e

darnautov requested a review from szabosteve August 2, 2021 11:33

szabosteve approved these changes Aug 2, 2021

View reviewed changes

darnautov added the release_note:feature Makes this part of the condensed release notes label Aug 2, 2021

darnautov added 2 commits August 2, 2021 14:17

Merge remote-tracking branch 'upstream/master' into ml-101028-delayed…

a81a180

…-data

[ML] add types

d53c91b

darnautov requested a review from walterra August 2, 2021 12:27

walterra approved these changes Aug 2, 2021

View reviewed changes

[ML] fix tests

1e8d5d5

darnautov requested a review from benwtrent August 4, 2021 14:37

benwtrent reviewed Aug 4, 2021

View reviewed changes

darnautov added 2 commits August 4, 2021 17:44

[ML] add maps

ef5130c

[ML] combine filters

9606a2f

darnautov requested a review from benwtrent August 4, 2021 15:55

[ML] move range query inside of a filter

637cc90

darnautov added 2 commits August 4, 2021 18:29

[ML] filter out jobs with missing datafeed

66e331a

[ML] resolveLookbackInterval only from jobs with datafeeds

0968256

benwtrent approved these changes Aug 4, 2021

View reviewed changes

darnautov added 3 commits August 5, 2021 10:32

[ML] do not show an error on empty time interval

c57b07b

[ML] add help tooltips

d6e3665

Merge remote-tracking branch 'upstream/master' into ml-101028-delayed…

52fcd40

…-data

szabosteve reviewed Aug 5, 2021

View reviewed changes

x-pack/plugins/ml/common/constants/alerts.ts Outdated Show resolved Hide resolved

szabosteve reviewed Aug 5, 2021

View reviewed changes

[ML] update description for the datafeed check

3afb856

Co-authored-by: István Zoltán Szabó <istvan.szabo@elastic.co>

pheyos approved these changes Aug 5, 2021

View reviewed changes

darnautov enabled auto-merge (squash) August 5, 2021 11:00

[ML] update default message

83d90ed

darnautov merged commit 39bd188 into elastic:master Aug 5, 2021

kibanamachine mentioned this pull request Aug 5, 2021

[7.x] [ML] Delayed data test for Anomaly Detection jobs health rule type (#107183) #107762

Merged

darnautov mentioned this pull request Aug 5, 2021

[ML] Operational alerting rule type for Anomaly detection jobs #101028

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Delayed data test for Anomaly Detection jobs health rule type #107183

[ML] Delayed data test for Anomaly Detection jobs health rule type #107183

darnautov commented Jul 29, 2021 •

edited

szabosteve left a comment

elasticmachine commented Aug 2, 2021

szabosteve left a comment

walterra left a comment

darnautov commented Aug 4, 2021

benwtrent left a comment

benwtrent Aug 4, 2021

qn895 commented Aug 4, 2021

darnautov commented Aug 4, 2021

szabosteve left a comment

pheyos left a comment

kibanamachine commented Aug 5, 2021

kibanamachine commented Aug 5, 2021

[ML] Delayed data test for Anomaly Detection jobs health rule type #107183

[ML] Delayed data test for Anomaly Detection jobs health rule type #107183

Conversation

darnautov commented Jul 29, 2021 • edited

Summary

Checklist

szabosteve left a comment

Choose a reason for hiding this comment

elasticmachine commented Aug 2, 2021

szabosteve left a comment

Choose a reason for hiding this comment

walterra left a comment

Choose a reason for hiding this comment

darnautov commented Aug 4, 2021

benwtrent left a comment

Choose a reason for hiding this comment

benwtrent Aug 4, 2021

Choose a reason for hiding this comment

qn895 commented Aug 4, 2021

darnautov commented Aug 4, 2021

szabosteve left a comment

Choose a reason for hiding this comment

pheyos left a comment

Choose a reason for hiding this comment

kibanamachine commented Aug 5, 2021

💚 Build Succeeded

Metrics [docs]

Async chunks

History

kibanamachine commented Aug 5, 2021

💚 Backport successful

darnautov commented Jul 29, 2021 •

edited