Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML] Handle delayed data audit messages better #73679

Closed
benwtrent opened this issue Jun 2, 2021 · 1 comment · Fixed by #75815
Closed

[ML] Handle delayed data audit messages better #73679

benwtrent opened this issue Jun 2, 2021 · 1 comment · Fixed by #75815
Assignees
@benwtrent
Labels
>enhancement :ml Machine learning Team:ML Meta label for the ML team

Comments

@benwtrent
Copy link
Member

If a job is running a long time and continually gets delayed data, we write audit messages for each new bucket that gets delayed.

For the annotations, we are able to update the previous annotation because we can keep track of the annotation ID there. But audit messages are traditionally append only, so we don't necessarily want to update the previous audit message.

It may be good to introduce some sort of audit message "throttling" so that we don't write a new audit message if every new bucket has delayed data (maybe some multiple of bucket length with some min threshold)?
@benwtrent benwtrent added >enhancement :ml Machine learning labels Jun 2, 2021
@elasticmachine elasticmachine added the Team:ML Meta label for the ML team label Jun 2, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

@benwtrent benwtrent self-assigned this Jul 28, 2021
benwtrent added a commit to benwtrent/elasticsearch that referenced this issue Jul 28, 2021
@benwtrent
[ML] throttle job audit msgs if delayed data occurs for consecutive b…
9d4741c 
…uckets

In a consistently delayed environment, it is possible that many buckets
in a row have delayed data detected. This can flood the audit
message page, causing noise that detracts from other useful information

This commit reduces the number of audit messages written as the number
of consecutive buckets with delayed data increases.

closes elastic#73679
@benwtrent benwtrent mentioned this issue Jul 28, 2021
Merged
benwtrent added a commit that referenced this issue Jul 29, 2021
@benwtrent
[ML] throttle job audit msgs if delayed data occurs for consecutive b…
c4f3b93 
…uckets (#75815)

In a consistently delayed environment, it is possible that many buckets
in a row have delayed data detected. This can flood the audit
message page, causing noise that detracts from other useful information

This commit reduces the number of audit messages written as the number
of consecutive buckets with delayed data increases.

closes #73679
elasticsearchmachine pushed a commit to elasticsearchmachine/elasticsearch that referenced this issue Jul 29, 2021
@benwtrent
[ML] throttle job audit msgs if delayed data occurs for consecutive b…
e559f33 
…uckets (elastic#75815)

In a consistently delayed environment, it is possible that many buckets
in a row have delayed data detected. This can flood the audit
message page, causing noise that detracts from other useful information

This commit reduces the number of audit messages written as the number
of consecutive buckets with delayed data increases.

closes elastic#73679
elasticsearchmachine added a commit that referenced this issue Jul 29, 2021
@elasticsearchmachine @benwtrent
[ML] throttle job audit msgs if delayed data occurs for consecutive b…
8bb02cf 
…uckets (#75815) (#75837)

In a consistently delayed environment, it is possible that many buckets
in a row have delayed data detected. This can flood the audit
message page, causing noise that detracts from other useful information

This commit reduces the number of audit messages written as the number
of consecutive buckets with delayed data increases.

closes #73679

Co-authored-by: Benjamin Trent <ben.w.trent@gmail.com>
ywangd pushed a commit to ywangd/elasticsearch that referenced this issue Jul 30, 2021
@benwtrent @ywangd
[ML] throttle job audit msgs if delayed data occurs for consecutive b…
dead681 
…uckets (elastic#75815)

In a consistently delayed environment, it is possible that many buckets
in a row have delayed data detected. This can flood the audit
message page, causing noise that detracts from other useful information

This commit reduces the number of audit messages written as the number
of consecutive buckets with delayed data increases.

closes elastic#73679
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benwtrent benwtrent

Labels
>enhancement :ml Machine learning Team:ML Meta label for the ML team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ML] throttle job audit msgs if delayed data occurs for consecutive buckets benwtrent/elasticsearch
2 participants
@benwtrent @elasticmachine