Snapshot restore operations throttle more than specified

[ywelsch]

Lucene's RateLimiter can do too much sleeping on small values (see also #6018).
The issue here is that calls to "pause" are not properly guarded in "restoreFile".

Instead of simply adding the guard, this commit uses the RateLimitingInputStream similar as for "snapshotFile".

[ywelsch]

@mikemccand can you have a look at this PR?

[mikemccand]

LGTM, this is a very bad bug! That small 4096 buffer means we are way over-throttling recovery :( We should push this back to 1.7.x?

[mikemccand]

Are we directly calling RateLimiter.pause anywhere else in ES?

[ywelsch]

I checked the other places, and they are all properly guarded. Maybe RateLimiter.pause could have an assertion?

[mikemccand]

Maybe RateLimiter.pause could have an assertion?

Well, it's called (correctly) by RateLimitingInputStream ... maybe we could ban ES from directly calling RateLimiter.pause using our forbidden APIs?

[ywelsch]

I'm very much in favour of back porting this to 1.7.x, as I saw a 10x slowdown on a production cluster.

I added your suggestions. Not closing the RateLimitingInputStream is ok, as it just delegates the close to PartSliceStream (which is now auto-closed).

Just banning RateLimiter.pause does not work as it is used in other places of ES (e.g. RecoverySourceHandler).

[mikemccand]

Just banning RateLimiter.pause does not work as it is used in other places of ES (e.g. RecoverySourceHandler).

OK thanks, and it looks like it does the right thing (checks getMinPauseCheckBytes).

[mikemccand]

LGTM, thanks @ywelsch!

[ywelsch]

@clintongormley which ES versions should we push this back to?

[clintongormley]

@ywelsch i'm good with 1.7.x and above