Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ILM introduce the check-ts-end-time-passed step #100179

Merged
merged 5 commits into from Oct 3, 2023

Conversation

andreidan
Copy link
Contributor

This introduces a new ILM step, the check-ts-end-time-passed step that'll wait for the index.time_series.end_time to lapse for TS indices before allowing ILM to proceed with the execution of the following actions:

  • downsample
  • forcemerge
  • readonly
  • searchable_snapshot
  • shrink

TSDS indices are allowed to receive writes until the configured index.time_series.end_time time passes. This makes sure ILM doesn't block these indices from accepting writes prematurely.

NOTE: documentation is added in a subsequent PR

Fixes #99696

This introduce a new ILM step, the `check-ts-end-time-passed` step
that'll wait for the `index.time_series.end_time` to lapse for TS
indices before allowing ILM to proceed with the execution of the
following actions:
* downsample
* forcemerge
* readonly
* searchable_snapshot
* shrink

TSDS indices are allowed to receive writes until the configured
`index.time_series.end_time` time passes. This makes sure ILM doesn't
block these indices from accepting writes prematurely.
@andreidan andreidan added >bug :Data Management/ILM+SLM Index and Snapshot lifecycle management v8.11.0 v8.10.3 labels Oct 3, 2023
@elasticsearchmachine elasticsearchmachine added the Team:Data Management Meta label for data/management team label Oct 3, 2023
@elasticsearchmachine
Copy link
Collaborator

Hi @andreidan, I've created a changelog YAML for you.

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@andreidan
Copy link
Contributor Author

@elasticmachine run elasticsearch-ci/part-1

@andreidan
Copy link
Contributor Author

andreidan commented Oct 3, 2023

@elasticmachine run elasticsearch-ci/part-2

Failure tracked already in #96723

Copy link
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left one testing comment. LGTM otherwise.

}
Instant configuredEndTime = IndexSettings.TIME_SERIES_END_TIME.get(indexMetadata.getSettings());
assert configuredEndTime != null : "a time series index must have an end time configured but [" + index.getName() + "] does not";
if (Instant.now().isBefore(configuredEndTime)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe provide current time as a lambda function? This would allow unit tests to not rely on current time.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes! Great shout. Done in a6ffc7d

Thanks Martijn

@andreidan
Copy link
Contributor Author

@elasticmachine run elasticsearch-ci/part-3

@andreidan andreidan added the auto-merge Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Oct 3, 2023
@elasticsearchmachine elasticsearchmachine merged commit 8bf8fc1 into elastic:main Oct 3, 2023
12 checks passed
@andreidan andreidan added the auto-backport-and-merge Automatically create backport pull requests and merge when ready label Oct 3, 2023
@andreidan andreidan deleted the ilm-wait-for-end-time branch October 3, 2023 13:43
andreidan added a commit to andreidan/elasticsearch that referenced this pull request Oct 3, 2023
This introduces a new ILM step, the `check-ts-end-time-passed` step
that'll wait for the `index.time_series.end_time` to lapse for TS
indices before allowing ILM to proceed with the execution of the
following actions: * downsample * forcemerge * readonly *
searchable_snapshot * shrink

TSDS indices are allowed to receive writes until the configured
`index.time_series.end_time` time passes. This makes sure ILM doesn't
block these indices from accepting writes prematurely.

NOTE: documentation is added in a subsequent PR

Fixes elastic#99696

(cherry picked from commit 8bf8fc1)
Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
andreidan added a commit to andreidan/elasticsearch that referenced this pull request Oct 3, 2023
This introduces a new ILM step, the `check-ts-end-time-passed` step
that'll wait for the `index.time_series.end_time` to lapse for TS
indices before allowing ILM to proceed with the execution of the
following actions: * downsample * forcemerge * readonly *
searchable_snapshot * shrink

TSDS indices are allowed to receive writes until the configured
`index.time_series.end_time` time passes. This makes sure ILM doesn't
block these indices from accepting writes prematurely.

NOTE: documentation is added in a subsequent PR

Fixes elastic#99696
@elasticsearchmachine
Copy link
Collaborator

💚 Backport successful

Status Branch Result
8.10

elasticsearchmachine pushed a commit that referenced this pull request Oct 3, 2023
This introduces a new ILM step, the `check-ts-end-time-passed` step
that'll wait for the `index.time_series.end_time` to lapse for TS
indices before allowing ILM to proceed with the execution of the
following actions: * downsample * forcemerge * readonly *
searchable_snapshot * shrink

TSDS indices are allowed to receive writes until the configured
`index.time_series.end_time` time passes. This makes sure ILM doesn't
block these indices from accepting writes prematurely.

NOTE: documentation is added in a subsequent PR

Fixes #99696
andreidan added a commit that referenced this pull request Oct 3, 2023
This introduces a new ILM step, the `check-ts-end-time-passed` step
that'll wait for the `index.time_series.end_time` to lapse for TS
indices before allowing ILM to proceed with the execution of the
following actions: * downsample * forcemerge * readonly *
searchable_snapshot * shrink

TSDS indices are allowed to receive writes until the configured
`index.time_series.end_time` time passes. This makes sure ILM doesn't
block these indices from accepting writes prematurely.

NOTE: documentation is added in a subsequent PR

Fixes #99696

(cherry picked from commit 8bf8fc1)

Signed-off-by: Andrei Dan <andrei.dan@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport-and-merge Automatically create backport pull requests and merge when ready auto-merge Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) >bug :Data Management/ILM+SLM Index and Snapshot lifecycle management Team:Data Management Meta label for data/management team v8.10.3 v8.11.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TSDS data streams can route writes to downsampled indices after rollover
3 participants