Lower the look_ahead_time setting's maximum value. #103434
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Initially the index.look_head_time was both used to define the index.time_series.start_time and index.time_series.end_time. The former is now controlled by index.look_back_time and the previous default of 2 hour for index.look_ahead_time is too generous. As it also delays data being indexed to new index after rollover by 2 hours. This change changes the index.look_ahead_time setting from 2 hours to 30 minutes. This should still allow data that is ahead of the current time of Elasticsearch nodes, but reduces the delay that the new backing index receives write after rollover. Additionally, the max value is reduced to 2 hours instead of 7 days.
|
Hi @martijnvg, I've created a changelog YAML for you. |
Only keep the change the maximum allowed value.
martijnvg
changed the title
|
Hi @martijnvg, I've updated the changelog YAML for you. Note that since this PR is labelled |
|
Hi @martijnvg, I've updated the changelog YAML for you. Note that since this PR is labelled |
elasticsearchmachine
added
the
Team:Analytics
|
Pinging @elastic/es-analytics-geo (Team:Analytics) |
andreidan
approved these changes
Dec 19, 2023
|
@martijnvg according to this PR's labels, I need to update the changelog YAML, but I can't because the PR is closed. Please either update the changelog yourself on the appropriate branch, or adjust the labels. Specifically:
|
javanna
pushed a commit
to javanna/elasticsearch
that referenced
this pull request
Dec 21, 2023
Initially the index.look_head_time was both used to define the index.time_series.start_time and index.time_series.end_time. The former is now controlled by index.look_back_time and the maximum value of 7 days for index.look_ahead_time is too generous. As it also delays data being indexed to new index after rollover by up to 7 days. This PR changes the index.look_ahead_time setting's maximum allowed value from 7 days to 2 hours, which is equal to the index.look_ahead_time setting's default. A look ahead time of 2 hours is high enough to accept data that is ahead of the current time, but avoids configuring the index.look_ahead_time setting to a too high value that causes rolled over indices to not receive writes for a very long period. This is a breaking change, but configuring the index.look_ahead_time setting to a higher value than 2 hours will not fail. Instead 2 hours will be used a look ahead time.
navarone-feekery
pushed a commit
to navarone-feekery/elasticsearch
that referenced
this pull request
Dec 22, 2023
Initially the index.look_head_time was both used to define the index.time_series.start_time and index.time_series.end_time. The former is now controlled by index.look_back_time and the maximum value of 7 days for index.look_ahead_time is too generous. As it also delays data being indexed to new index after rollover by up to 7 days. This PR changes the index.look_ahead_time setting's maximum allowed value from 7 days to 2 hours, which is equal to the index.look_ahead_time setting's default. A look ahead time of 2 hours is high enough to accept data that is ahead of the current time, but avoids configuring the index.look_ahead_time setting to a too high value that causes rolled over indices to not receive writes for a very long period. This is a breaking change, but configuring the index.look_ahead_time setting to a higher value than 2 hours will not fail. Instead 2 hours will be used a look ahead time.
|
@martijnvg according to this PR's labels, I need to update the changelog YAML, but I can't because the PR is closed. Please either update the changelog yourself on the appropriate branch, or adjust the labels. Specifically:
|
martijnvg
added a commit
that referenced
this pull request
Jan 5, 2024
…30 minutes. (#103898) A followup from #103434 The impact of this change is that by default documents with @timestamp of 30 minutes or more in the future will be rejected. Prior to this change by default documents with @timestamp of 2 hours or more in the future were rejected. If the previous behaviour should be kept, then update the index.look_ahead_time setting to two hours before performing the upgrade. By reducing the default look ahead time, new metrics will sooner be indexed into the new backing index. With the current default it takes up to 2 hours before writes get indexed into the new backing index. This is after rollover has occurred. This delayed effect can result in previous backing index growing well beyond the rollover conditions. This change should reduce this effect.
jbaiera
pushed a commit
to jbaiera/elasticsearch
that referenced
this pull request
Jan 10, 2024
Initially the index.look_head_time was both used to define the index.time_series.start_time and index.time_series.end_time. The former is now controlled by index.look_back_time and the maximum value of 7 days for index.look_ahead_time is too generous. As it also delays data being indexed to new index after rollover by up to 7 days. This PR changes the index.look_ahead_time setting's maximum allowed value from 7 days to 2 hours, which is equal to the index.look_ahead_time setting's default. A look ahead time of 2 hours is high enough to accept data that is ahead of the current time, but avoids configuring the index.look_ahead_time setting to a too high value that causes rolled over indices to not receive writes for a very long period. This is a breaking change, but configuring the index.look_ahead_time setting to a higher value than 2 hours will not fail. Instead 2 hours will be used a look ahead time.
jbaiera
pushed a commit
to jbaiera/elasticsearch
that referenced
this pull request
Jan 10, 2024
…30 minutes. (elastic#103898) A followup from elastic#103434 The impact of this change is that by default documents with @timestamp of 30 minutes or more in the future will be rejected. Prior to this change by default documents with @timestamp of 2 hours or more in the future were rejected. If the previous behaviour should be kept, then update the index.look_ahead_time setting to two hours before performing the upgrade. By reducing the default look ahead time, new metrics will sooner be indexed into the new backing index. With the current default it takes up to 2 hours before writes get indexed into the new backing index. This is after rollover has occurred. This delayed effect can result in previous backing index growing well beyond the rollover conditions. This change should reduce this effect.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initially the index.look_head_time was both used to define the index.time_series.start_time and index.time_series.end_time.
The former is now controlled by index.look_back_time and the maximum value of 7 days for index.look_ahead_time is too generous. As it also delays data being indexed to new index after rollover by up to 7 days.
This PR changes the
index.look_ahead_timesetting's maximum allowed value from 7 days to 2 hours, which is equal to theindex.look_ahead_timesetting's default. A look ahead time of 2 hours is high enough to accept data that is ahead of the current time, but avoids configuring theindex.look_ahead_timesetting to a too high value that causes rolled over indices to not receive writes for a very long period.This is a breaking change, but configuring the
index.look_ahead_timesetting to a higher value than 2 hours will not fail. Instead 2 hours will be used a look ahead time.The
index.look_ahead_timesetting's default will be changed in a different change.