Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EventLog] change to use DLM instead of ILM #162886

Closed
pmuellr opened this issue Aug 1, 2023 · 2 comments · Fixed by #163210
Closed

[EventLog] change to use DLM instead of ILM #162886

pmuellr opened this issue Aug 1, 2023 · 2 comments · Fixed by #163210
Assignees
Labels
bug Fixes for quality problems that affect the customer experience Feature:EventLog Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)

Comments

@pmuellr
Copy link
Member

pmuellr commented Aug 1, 2023

The current version of the event log sets up ILM for the event log indices, but ILM is not available in serverless. DLM is.

I assume we want to use DLM for traditional as well. If not, we'll need a configuration switch to select whether to use ILM or DLM.

It's not clear to me when DLM will be available as well, but looks like it's available now: https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams-put-lifecycle.html

There's also a potential short-term issue we will probably want to address. In serverless today, when the event log initializes, it tries to create an ILM policy, but fails since ILM is not available. Resulting in a spammy console. We may want to disable ILM

@pmuellr pmuellr added bug Fixes for quality problems that affect the customer experience Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Feature:EventLog labels Aug 1, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@pmuellr
Copy link
Member Author

pmuellr commented Aug 2, 2023

Note that the current ILM policy calls for deleting event log docs after 90 days. Presumably, that means we'd set the DLM options as:

      "lifecycle": {                                        
        "data_retention": "90d"
      }

Should also be release-noted, since customers may have customized the ILM policy we've been using all this time, which would now be ignored. We could also note that that ILM policy can be deleted - or should we attempt to delete it ourselves? We could potentially just update it's _meta with a note as well ...

export function getIlmPolicy() {
return {
policy: {
_meta: {
description:
'ilm policy the Kibana event log, created initially by Kibana, but updated by the user, not Kibana',
managed: false,
},
phases: {
hot: {
actions: {
rollover: {
max_size: '50GB',
max_age: '30d',
// max_docs: 1, // you know, for testing
},
},
},
delete: {
min_age: '90d',
actions: {
delete: {},
},
},
},
},
};
}

@pmuellr pmuellr self-assigned this Aug 4, 2023
pmuellr added a commit that referenced this issue Aug 8, 2023
resolves #162886

The default continues to be 90 days for data detetention of event log
documents, and the rollover is now controlled by DLM, as described in
[Data stream lifecycle][].

[Data stream lifecycle]:
https://www.elastic.co/guide/en/elasticsearch/reference/8.9/data-stream-lifecycle.html

## Release note

Fixes the event log data stream to use Data stream lifecycle instead of
Index Lifecycle Management. If you had previously customized the Kibana
event log ILM policy, you should now update the lifecycle of the event
log data stream itself.
bryce-b pushed a commit to bryce-b/kibana that referenced this issue Aug 9, 2023
…c#163210)

resolves elastic#162886

The default continues to be 90 days for data detetention of event log
documents, and the rollover is now controlled by DLM, as described in
[Data stream lifecycle][].

[Data stream lifecycle]:
https://www.elastic.co/guide/en/elasticsearch/reference/8.9/data-stream-lifecycle.html

## Release note

Fixes the event log data stream to use Data stream lifecycle instead of
Index Lifecycle Management. If you had previously customized the Kibana
event log ILM policy, you should now update the lifecycle of the event
log data stream itself.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Fixes for quality problems that affect the customer experience Feature:EventLog Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

2 participants