Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add awaitsfixed tests for running ILM actions during snapshot #37552

Closed
wants to merge 6 commits into from

Conversation

talevy
Copy link
Contributor

@talevy talevy commented Jan 16, 2019

These tests check that Delete, Freeze, and Shrink actions fail execution while
a snapshot is in progress.

these should stay awaitsfixed until #37541 is resolved.
The intention of this PR is to introduce tests that should pass once follow-up PRs are introduced that can un-awaits them.

@talevy talevy added >test Issues or PRs that are addressing/adding tests :Data Management/ILM+SLM Index and Snapshot lifecycle management labels Jan 16, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features

@talevy talevy removed the WIP label Jan 17, 2019
@talevy talevy requested a review from dakrone January 17, 2019 06:27
@talevy
Copy link
Contributor Author

talevy commented Jan 17, 2019

fun fact: not all IN_PROGRESS are the same. Setting max_snapshot_bytes_per_sec to 1b results in no progress being noticed, so the delete-snapshot action never succeeds in a timely manner. Setting this to 256b yields desired results within the time-bounds of assertBusy().

Copy link
Member

@dakrone dakrone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM, I left one minor comment

@talevy talevy changed the title add test for running ILM delete action during snapshot add awaitsfixed tests for running ILM actions during snapshot Jan 17, 2019
@talevy
Copy link
Contributor Author

talevy commented Jan 17, 2019

run gradle build tests 1
run gradle build tests 2

@talevy
Copy link
Contributor Author

talevy commented Jan 18, 2019

run gradle build tests 1
run gradle build tests 2

@talevy
Copy link
Contributor Author

talevy commented Jan 18, 2019

run gradle build tests 2

@talevy
Copy link
Contributor Author

talevy commented Jan 18, 2019

run gradle build tests 1
run gradle build tests 2

@talevy
Copy link
Contributor Author

talevy commented Jan 18, 2019

run gradle build tests 1

@talevy
Copy link
Contributor Author

talevy commented Jan 18, 2019

These tests were designed to be used by a follow-up PR that fixes these ILM action while snapshot issues. Closing this since changes to these tests will need to be done when new code for fixes are introduced. This PR can be viewed as a reference to tests that will be added in other PRs. Closing to be superseded by follow-up PRs

@talevy talevy closed this Jan 18, 2019
@talevy talevy deleted the ilm-snapshot-test branch January 18, 2019 17:56
dakrone added a commit that referenced this pull request Jan 23, 2019
Some steps, such as steps that delete, close, or freeze an index, may fail due to a currently running snapshot of the index. In those cases, rather than move to the ERROR step, we should retry the step when the snapshot has completed.

This change adds an abstract step (`AsyncRetryDuringSnapshotActionStep`) that certain steps (like the ones I mentioned above) can extend that will automatically handle a situation where a snapshot is taking place. When a `SnapshotInProgressException` is received by the listener wrapper, a `ClusterStateObserver` listener is registered to wait until the snapshot has completed, re-running the ILM action when no snapshot is occurring.

This also adds integration tests for these scenarios (thanks to @talevy in #37552).

Resolves #37541
dakrone added a commit that referenced this pull request Jan 23, 2019
Some steps, such as steps that delete, close, or freeze an index, may fail due to a currently running snapshot of the index. In those cases, rather than move to the ERROR step, we should retry the step when the snapshot has completed.

This change adds an abstract step (`AsyncRetryDuringSnapshotActionStep`) that certain steps (like the ones I mentioned above) can extend that will automatically handle a situation where a snapshot is taking place. When a `SnapshotInProgressException` is received by the listener wrapper, a `ClusterStateObserver` listener is registered to wait until the snapshot has completed, re-running the ILM action when no snapshot is occurring.

This also adds integration tests for these scenarios (thanks to @talevy in #37552).

Resolves #37541
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/ILM+SLM Index and Snapshot lifecycle management >test Issues or PRs that are addressing/adding tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants