Skip to content

Commit

Permalink
Stabilize SLM REST Tests (#46195)
Browse files Browse the repository at this point in the history
Unfortunately, #42791 destabilized SLM tests because those tests use
rate limiting the snapshot write rate to a very low value globally.
Now that the various files in a snapshot get uploaded in parallel
this can lead to a few threads in parallel way overshooting the low
value throughput value used by the rate limiter and then making it
wait for minutes which times out the tests that then try to abort
the snapshot (see #21759 for details, aborting a snapshot only
happens when writing bytes to the repository).

For now the old behavior of the test from before my changes can
be restored by moving to a single threaded snapshot pool but
we should find a better way of testing the SLM behaviour here in
a follow-up.
  • Loading branch information
original-brownbear committed Aug 31, 2019
1 parent 3499b58 commit 1e676ab
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions x-pack/plugin/ilm/qa/multi-node/build.gradle
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
import org.elasticsearch.gradle.test.RestIntegTestTask

apply plugin: 'elasticsearch.testclusters'
apply plugin: 'elasticsearch.standalone-rest-test'
apply plugin: 'elasticsearch.rest-test'
Expand Down Expand Up @@ -27,4 +25,8 @@ testClusters.integTest {
setting 'xpack.ml.enabled', 'false'
setting 'xpack.license.self_generated.type', 'trial'
setting 'indices.lifecycle.poll_interval', '1000ms'
// TODO: Find a way to run these tests with more than one snapshot pool thread. Currently we need to limit to one thread so that the
// rate limiting settings in SnapshotLifecycleIT doesn't result in blocked snapshot threads because multiple threads overshoot
// the limit simultaneously and the rate limiter then moves to wait minutes to make up for this.
setting 'thread_pool.snapshot.max', '1'
}

0 comments on commit 1e676ab

Please sign in to comment.