Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add awaitsfixed tests for running ILM actions during snapshot #37552

Closed
wants to merge 6 commits into from

Conversation

Projects
None yet
3 participants
@talevy
Copy link
Contributor

commented Jan 16, 2019

These tests check that Delete, Freeze, and Shrink actions fail execution while
a snapshot is in progress.

these should stay awaitsfixed until #37541 is resolved.
The intention of this PR is to introduce tests that should pass once follow-up PRs are introduced that can un-awaits them.

@elasticmachine

This comment has been minimized.

Copy link

commented Jan 16, 2019

@talevy talevy force-pushed the talevy:ilm-snapshot-test branch from e2eca9f to 7aeec21 Jan 17, 2019

@talevy talevy removed the WIP label Jan 17, 2019

@talevy talevy requested a review from dakrone Jan 17, 2019

@talevy

This comment has been minimized.

Copy link
Contributor Author

commented Jan 17, 2019

fun fact: not all IN_PROGRESS are the same. Setting max_snapshot_bytes_per_sec to 1b results in no progress being noticed, so the delete-snapshot action never succeeds in a timely manner. Setting this to 256b yields desired results within the time-bounds of assertBusy().

talevy added some commits Jan 17, 2019

@dakrone
Copy link
Member

left a comment

This LGTM, I left one minor comment

@talevy talevy changed the title add test for running ILM delete action during snapshot add awaitsfixed tests for running ILM actions during snapshot Jan 17, 2019

@talevy

This comment has been minimized.

Copy link
Contributor Author

commented Jan 17, 2019

run gradle build tests 1
run gradle build tests 2

@talevy

This comment has been minimized.

Copy link
Contributor Author

commented Jan 18, 2019

run gradle build tests 1
run gradle build tests 2

@talevy

This comment has been minimized.

Copy link
Contributor Author

commented Jan 18, 2019

run gradle build tests 2

@talevy

This comment has been minimized.

Copy link
Contributor Author

commented Jan 18, 2019

run gradle build tests 1
run gradle build tests 2

@talevy

This comment has been minimized.

Copy link
Contributor Author

commented Jan 18, 2019

run gradle build tests 1

@talevy

This comment has been minimized.

Copy link
Contributor Author

commented Jan 18, 2019

These tests were designed to be used by a follow-up PR that fixes these ILM action while snapshot issues. Closing this since changes to these tests will need to be done when new code for fixes are introduced. This PR can be viewed as a reference to tests that will be added in other PRs. Closing to be superseded by follow-up PRs

@talevy talevy closed this Jan 18, 2019

@talevy talevy deleted the talevy:ilm-snapshot-test branch Jan 18, 2019

dakrone added a commit that referenced this pull request Jan 23, 2019

Retry ILM steps that fail due to SnapshotInProgressException (#37624)
Some steps, such as steps that delete, close, or freeze an index, may fail due to a currently running snapshot of the index. In those cases, rather than move to the ERROR step, we should retry the step when the snapshot has completed.

This change adds an abstract step (`AsyncRetryDuringSnapshotActionStep`) that certain steps (like the ones I mentioned above) can extend that will automatically handle a situation where a snapshot is taking place. When a `SnapshotInProgressException` is received by the listener wrapper, a `ClusterStateObserver` listener is registered to wait until the snapshot has completed, re-running the ILM action when no snapshot is occurring.

This also adds integration tests for these scenarios (thanks to @talevy in #37552).

Resolves #37541

dakrone added a commit that referenced this pull request Jan 23, 2019

Retry ILM steps that fail due to SnapshotInProgressException (#37624)
Some steps, such as steps that delete, close, or freeze an index, may fail due to a currently running snapshot of the index. In those cases, rather than move to the ERROR step, we should retry the step when the snapshot has completed.

This change adds an abstract step (`AsyncRetryDuringSnapshotActionStep`) that certain steps (like the ones I mentioned above) can extend that will automatically handle a situation where a snapshot is taking place. When a `SnapshotInProgressException` is received by the listener wrapper, a `ClusterStateObserver` listener is registered to wait until the snapshot has completed, re-running the ILM action when no snapshot is occurring.

This also adds integration tests for these scenarios (thanks to @talevy in #37552).

Resolves #37541
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.