Move index sealing terminology to synced flush #11336

Merged
merged 3 commits into from May 29, 2015

Conversation

Projects
None yet
6 participants
@bleskes
Member

bleskes commented May 25, 2015

#10032 introduced the notion of sealing an index by marking it with a special read only marker, allowing for a couple of optimization to happen. The most important one was to speed up recoveries of shards where we know nothing has changed since they were online by skipping the file based sync phase. During the implementation we came up with a light notion which achieves the same recovery benefits but without the read only aspects which we dubbed synced flush. The fact that it was light weight and didn't put the index in read only mode, allowed us to do it automatically in the background which has great advantage. However we also felt the need to allow users to manually trigger this operation.

The implementation at #11179 added the sync flush internal logic and the manual (rest) rest API. The name of the API was modeled after the sealing terminology which may end up being confusing. This commit changes the API name to match the internal synced flush naming, namely `{index}/_flush/synced'.

On top of that it contains a couple other changes:

  • Remove all java client API. This feature is not supposed to be called programtically by applications but rather by admins.
  • Improve rest responses making structure similar to other (flush) API
  • Change IndexShard#getOperationsCount to exclude the internal +1 on open shard . it's confusing to get 1 while there are actually no ongoing operations
  • Some minor other clean ups

Closes #11251

Move index sealing terminology to synced flush
#10032 introduced the notion of sealing an index by marking it with a special read only marker, allowing for a couple of optimization to happen. The most important one was to speed up recoveries of shards where we know nothing has changed since they were online by skipping the file based sync phase. During the implementation we came up with a light notion which achieves the same recovery benefits but without the read only aspects which we dubbed synced flush. The fact that it was light weight and didn't put the index in read only mode, allowed us to do it automatically in the background which has great advantage. However we also felt the need to allow users to manually trigger this operation.

 The implementation at #11179 added the sync flush internal logic and the manual (rest) rest API. The name of the API was modeled after the sealing terminology which may end up being confusing. This commit changes the API name to match the internal synced flush naming, namely `{index}/_flush/synced'.

  On top of that it contains a couple other changes:
   - Remove all java client API. This feature is not supposed to be called programtically by applications but rather by admins.
   - Improve rest responses making structure similar to other (flush) API
   - Change IndexShard#getOperationsCount to exclude the internal +1 on open shard . it's confusing to get 1 while there are actually no ongoing operations
   - Some minor other clean ups
@bleskes

This comment has been minimized.

Show comment
Hide comment
@bleskes

bleskes May 25, 2015

Member

@brwe @clintongormley can you please review?

Member

bleskes commented May 25, 2015

@brwe @clintongormley can you please review?

docs/reference/indices/flush.asciidoc
+1. Synced flush is a best effort operation. Any ongoing indexing operations will cause
+the synced flush to fail. This means that some shards may be synced flushed while others aren't. See below for more.
+2. The `sync_id` marker is removed as soon as the shard is flushed again. Uncommitted
+operations in the transaction log do not remove the marker. That is because the marker is store as part

This comment has been minimized.

@clintongormley

clintongormley May 25, 2015

Member

the marker is storeD

@clintongormley

clintongormley May 25, 2015

Member

the marker is storeD

docs/reference/indices/flush.asciidoc
+--------------------------------------------------
+// AUTOSENSE
+
+The response contains details about how many shards were successfully synced-flushed and information about any failure.

This comment has been minimized.

@clintongormley

clintongormley May 25, 2015

Member

synced-flushed -> sync-flushed

@clintongormley

clintongormley May 25, 2015

Member

synced-flushed -> sync-flushed

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley May 25, 2015

Member

Docs look great! (two tiny changes)

Member

clintongormley commented May 25, 2015

Docs look great! (two tiny changes)

docs/reference/indices/flush.asciidoc
+[float]
+=== Synced Flush API
+
+The Synced Flush API allows an administrator to initiate a synced flush manually. This can particularly useful for

This comment has been minimized.

@bleskes

bleskes May 25, 2015

Member

This can BE particularly useful for

@bleskes

bleskes May 25, 2015

Member

This can BE particularly useful for

+
+// @Override
+// public void writeTo(StreamOutput out) throws IOException {
+// super.writeTo(out);

This comment has been minimized.

@brwe

brwe May 26, 2015

Contributor

remove?

@brwe

brwe May 26, 2015

Contributor

remove?

This comment has been minimized.

@bleskes

bleskes May 26, 2015

Member

oops. will do

@bleskes

bleskes May 26, 2015

Member

oops. will do

+ }
+ final int finalTotalNumberOfShards = totalNumberOfShards;
+ final CountDown countDown = new CountDown(numberOfShards);
+

This comment has been minimized.

@brwe

brwe May 26, 2015

Contributor

flush request will hang if there are no concrete indices. I think we have to check this case and then call listener.onResponse()

@brwe

brwe May 26, 2015

Contributor

flush request will hang if there are no concrete indices. I think we have to check this case and then call listener.onResponse()

@nik9000

This comment has been minimized.

Show comment
Hide comment
@nik9000

nik9000 May 26, 2015

Contributor

I don't mind the "seal" name - I just stopped thinking of it as a hermetic seal and started thinking of it as a wax seal on an envelope. You break it when you stuff more documents in it. I forget the other half of the metaphor about having to break the seal to read the documents.

Contributor

nik9000 commented May 26, 2015

I don't mind the "seal" name - I just stopped thinking of it as a hermetic seal and started thinking of it as a wax seal on an envelope. You break it when you stuff more documents in it. I forget the other half of the metaphor about having to break the seal to read the documents.

builder.endObject();
- return new BytesRestResponse(response.status(), builder);
+ return new BytesRestResponse(RestStatus.OK, builder);

This comment has been minimized.

@brwe

brwe May 26, 2015

Contributor

ok for now but we need to figure out what to return (#11251)

@brwe

brwe May 26, 2015

Contributor

ok for now but we need to figure out what to return (#11251)

docs/reference/indices/flush.asciidoc
+=== Synced Flush
+
+Elasticsearch tracks the indexing activity of each shards. Shards that have not
+received any indexing operations for, by default, 30m are automatically marked as inactive. This presents

This comment has been minimized.

@nik9000

nik9000 May 26, 2015

Contributor

s/for, by default, 30m/for 30 minutes (configurable)/

The commas caused my momentary blindness and I didn't read "30m" as 30 minutes - I read it as 30 million and though I guess I'll figure out how that makes sense later on in the document.

@nik9000

nik9000 May 26, 2015

Contributor

s/for, by default, 30m/for 30 minutes (configurable)/

The commas caused my momentary blindness and I didn't read "30m" as 30 minutes - I read it as 30 million and though I guess I'll figure out how that makes sense later on in the document.

This comment has been minimized.

@bleskes

bleskes May 26, 2015

Member

haha. Appreciate your trust :)

On 26 May 2015, at 16:20, Nik Everett notifications@github.com wrote:

In docs/reference/indices/flush.asciidoc:


+// AUTOSENSE
+
+[[indices-synced-flush]]
+=== Synced Flush
+
+Elasticsearch tracks the indexing activity of each shards. Shards that have not
+received any indexing operations for, by default, 30m are automatically marked as inactive. This presents

s/for, by default, 30m/for 30 minutes (configurable)/

The commas caused my momentary blindness and I didn't read "30m" as 30 minutes - I read it as 30 million and though I guess I'll figure out how that makes sense later on in the document.


Reply to this email directly or view it on GitHub.

@bleskes

bleskes May 26, 2015

Member

haha. Appreciate your trust :)

On 26 May 2015, at 16:20, Nik Everett notifications@github.com wrote:

In docs/reference/indices/flush.asciidoc:


+// AUTOSENSE
+
+[[indices-synced-flush]]
+=== Synced Flush
+
+Elasticsearch tracks the indexing activity of each shards. Shards that have not
+received any indexing operations for, by default, 30m are automatically marked as inactive. This presents

s/for, by default, 30m/for 30 minutes (configurable)/

The commas caused my momentary blindness and I didn't read "30m" as 30 minutes - I read it as 30 million and though I guess I'll figure out how that makes sense later on in the document.


Reply to this email directly or view it on GitHub.

docs/reference/indices/flush.asciidoc
+=== Synced Flush API
+
+The Synced Flush API allows an administrator to initiate a synced flush manually. This can particularly useful for
+a planned (rolling) cluster restart where one can stop indexing and doesn't want to wait for the default 30m to pass

This comment has been minimized.

@nik9000

nik9000 May 26, 2015

Contributor

s/30m/30 minutes/

@nik9000

nik9000 May 26, 2015

Contributor

s/30m/30 minutes/

docs/reference/indices/flush.asciidoc
+
+1. Synced flush is a best effort operation. Any ongoing indexing operations will cause
+the synced flush to fail. This means that some shards may be synced flushed while others aren't. See below for more.
+2. The `sync_id` marker is removed as soon as the shard is flushed again. Uncommitted

This comment has been minimized.

@nik9000

nik9000 May 26, 2015

Contributor
  1. The sync_id marker is removed as soon as the shard is flushed again. That is because the marker is stored as part of a low level lucene commit which represents a point in time snapshot of the segments. In practice any indexing operation on an index removes the marker. Its not immediate because operations queue up in the transaction log before being flushed but that isn't usually a useful distinction.
@nik9000

nik9000 May 26, 2015

Contributor
  1. The sync_id marker is removed as soon as the shard is flushed again. That is because the marker is stored as part of a low level lucene commit which represents a point in time snapshot of the segments. In practice any indexing operation on an index removes the marker. Its not immediate because operations queue up in the transaction log before being flushed but that isn't usually a useful distinction.
+ "twitter": {
+ "total": 4,
+ "successful": 2,
+ "failed": 2,

This comment has been minimized.

@nik9000

nik9000 May 26, 2015

Contributor

"failed": 1 ?

I only see one failure message. :)

@nik9000

nik9000 May 26, 2015

Contributor

"failed": 1 ?

I only see one failure message. :)

This comment has been minimized.

@bleskes

bleskes May 27, 2015

Member

yeah, that's tricky - the failure is a failure of an entire shard group (for which we don't have a good name in ES), see text above. Thus it fails multiple shards. I agree it can be confusing, but I don't have a better way, which is consistent with other APIs and doesn't take lots of programming. suggestions welcome :)

@bleskes

bleskes May 27, 2015

Member

yeah, that's tricky - the failure is a failure of an entire shard group (for which we don't have a good name in ES), see text above. Thus it fails multiple shards. I agree it can be confusing, but I don't have a better way, which is consistent with other APIs and doesn't take lots of programming. suggestions welcome :)

This comment has been minimized.

@nik9000

nik9000 May 27, 2015

Contributor

Ah - I see - its the primary and all the replicas. Makes sense. I don't have a better way to write it though.

@nik9000

nik9000 May 27, 2015

Contributor

Ah - I see - its the primary and all the replicas. Makes sense. I don't have a better way to write it though.

+ "failed": 1,
+ "failures": [
+ {
+ "shard": 1,

This comment has been minimized.

@nik9000

nik9000 May 26, 2015

Contributor

Presumably the other copies succeeded in this case. Is it a good idea to output these in the same total/successful/failed/failures format? Also, it might be useful to mention what this means from a recovery standpoint. Something like "those copies that failed will not be eligible for fast recovery but those that succeeded still will be."

@nik9000

nik9000 May 26, 2015

Contributor

Presumably the other copies succeeded in this case. Is it a good idea to output these in the same total/successful/failed/failures format? Also, it might be useful to mention what this means from a recovery standpoint. Something like "those copies that failed will not be eligible for fast recovery but those that succeeded still will be."

This comment has been minimized.

@bleskes

bleskes May 27, 2015

Member

I added the extra clarification for recoveries- it's great. I don't follow the suggestion to change the output format?

@bleskes

bleskes May 27, 2015

Member

I added the extra clarification for recoveries- it's great. I don't follow the suggestion to change the output format?

This comment has been minimized.

@nik9000

nik9000 May 27, 2015

Contributor

I made that comment when I thought you had replica groups as the highest level so its not valid any more. Please ignore it.

@nik9000

nik9000 May 27, 2015

Contributor

I made that comment when I thought you had replica groups as the highest level so its not valid any more. Please ignore it.

docs/reference/setup/upgrade.asciidoc
+}'
+--------------------------------------------------
+
+* There is no problem to continue indexing while doing the upgrade. However, you can speed the process considerably

This comment has been minimized.

@nik9000

nik9000 May 26, 2015

Contributor

s/to continue indexing/continuing to index/ ?

@nik9000

nik9000 May 26, 2015

Contributor

s/to continue indexing/continuing to index/ ?

docs/reference/setup/upgrade.asciidoc
+--------------------------------------------------
+
+* There is no problem to continue indexing while doing the upgrade. However, you can speed the process considerably
+by stopping indexing temporarily to non-essential indices and issuing a manual <<indices-synced-flush, synced flush>>.

This comment has been minimized.

@nik9000

nik9000 May 26, 2015

Contributor

by temporarily stopping non-essential indexing

@nik9000

nik9000 May 26, 2015

Contributor

by temporarily stopping non-essential indexing

@nik9000

This comment has been minimized.

Show comment
Hide comment
@nik9000

nik9000 May 26, 2015

Contributor

I don't mind the "seal" name - I just stopped thinking of it as a hermetic seal and started thinking of it as a wax seal on an envelope. You break it when you stuff more documents in it. I forget the other half of the metaphor about having to break the seal to read the documents.

I take it back - after reviewing the documentation this way makes more sense to me. No fun metaphor though.

Contributor

nik9000 commented May 26, 2015

I don't mind the "seal" name - I just stopped thinking of it as a hermetic seal and started thinking of it as a wax seal on an envelope. You break it when you stuff more documents in it. I forget the other half of the metaphor about having to break the seal to read the documents.

I take it back - after reviewing the documentation this way makes more sense to me. No fun metaphor though.

@bleskes bleskes referenced this pull request May 26, 2015

Closed

Allow to seal an index #10032

@bleskes

This comment has been minimized.

Show comment
Hide comment
@bleskes

bleskes May 27, 2015

Member

@brwe @clintongormley @nik9000 thx for all the feedback. I pushed a new commit. Also assumed will end up with a 409 for #11251 ..

Member

bleskes commented May 27, 2015

@brwe @clintongormley @nik9000 thx for all the feedback. I pushed a new commit. Also assumed will end up with a 409 for #11251 ..

docs/reference/indices/flush.asciidoc
+[[indices-synced-flush]]
+=== Synced Flush
+
+Elasticsearch tracks the indexing activity of each shards. Shards that have not

This comment has been minimized.

@clintongormley

clintongormley May 27, 2015

Member

of each shards^H

@clintongormley

clintongormley May 27, 2015

Member

of each shards^H

docs/reference/indices/flush.asciidoc
+=== Synced Flush
+
+Elasticsearch tracks the indexing activity of each shards. Shards that have not
+received any indexing operations for 30 minutes (configurable) are automatically marked as inactive. This presents

This comment has been minimized.

@clintongormley

clintongormley May 27, 2015

Member

configurable, but where? do we document this anywhere? I couldn't find it. I'm not sure we should, but I'd either link "configurable" to the page with the setting, or remove it

@clintongormley

clintongormley May 27, 2015

Member

configurable, but where? do we document this anywhere? I couldn't find it. I'm not sure we should, but I'd either link "configurable" to the page with the setting, or remove it

This comment has been minimized.

@nik9000

nik9000 May 27, 2015

Contributor

Yeah - probably a good call. Just making it a link to an anchor tag would be great!

@nik9000

nik9000 May 27, 2015

Contributor

Yeah - probably a good call. Just making it a link to an anchor tag would be great!

docs/reference/indices/flush.asciidoc
+received any indexing operations for 30 minutes (configurable) are automatically marked as inactive. This presents
+an opportunity for Elasticsearch to reduce shard resources and also perform
+a special kind of flush, called `synced flush`. A synced flush performs normal
+flushing and adds a special uniquely generated marker (`sync_id`) to all shards.

This comment has been minimized.

@clintongormley

clintongormley May 27, 2015

Member

A synced flush performs a normal flush, then adds a generated unique marker (sync_id) to all shards.

@clintongormley

clintongormley May 27, 2015

Member

A synced flush performs a normal flush, then adds a generated unique marker (sync_id) to all shards.

docs/reference/indices/flush.asciidoc
+flushing and adds a special uniquely generated marker (`sync_id`) to all shards.
+
+Since the sync id marker was added when there were no ongoing indexing operations, it can
+be used as a quick way to check if two shards indices are identical. This quick sync id

This comment has been minimized.

@clintongormley

clintongormley May 27, 2015

Member

two shards indices -> two shard copies

@clintongormley

clintongormley May 27, 2015

Member

two shards indices -> two shard copies

docs/reference/indices/flush.asciidoc
+comparison (if present) is used during recovery or restarts to skip the first and
+most costly phase of the process. In that case, no segment files need to be copied and
+the transaction log replay phase of the recovery can start immediately. Note that since the sync id
+marker was applied together with a flush, it is highly likely that the transaction log will be empty,

This comment has been minimized.

@clintongormley

clintongormley May 27, 2015

Member

highly -> very

@clintongormley

clintongormley May 27, 2015

Member

highly -> very

docs/reference/indices/flush.asciidoc
+never or very rarely updated, such as time based data. This use case typically generates lots of indices whose
+recovery without the synced flush marker would take a long time.
+
+To check whether a shard has a marker or not, one can use the `commit` section of shard stats returned by

This comment has been minimized.

@clintongormley

clintongormley May 27, 2015

Member

"one can use" (v formal) -> "look for the sync_id in the commit section..."

@clintongormley

clintongormley May 27, 2015

Member

"one can use" (v formal) -> "look for the sync_id in the commit section..."

+GET /twitter/_stats/commit?level=shards
+--------------------------------------------------
+// AUTOSENSE
+

This comment has been minimized.

@clintongormley

clintongormley May 27, 2015

Member

Perhaps an example of the output?

@clintongormley

clintongormley May 27, 2015

Member

Perhaps an example of the output?

docs/reference/indices/flush.asciidoc
+=== Synced Flush API
+
+The Synced Flush API allows an administrator to initiate a synced flush manually. This can be particularly useful for
+a planned (rolling) cluster restart where one can stop indexing and doesn't want to wait for the default 30 minutes to pass

This comment has been minimized.

@clintongormley

clintongormley May 27, 2015

Member

"where ..." -> "you don't want to wait the default 30 minutes for idle indices to be sync-flushed automatically."

@clintongormley

clintongormley May 27, 2015

Member

"where ..." -> "you don't want to wait the default 30 minutes for idle indices to be sync-flushed automatically."

docs/reference/indices/flush.asciidoc
+While handy, there are a couple of caveats for this API:
+
+1. Synced flush is a best effort operation. Any ongoing indexing operations will cause
+the synced flush to fail. This means that some shards may be synced flushed while others aren't. See below for more.

This comment has been minimized.

@clintongormley

clintongormley May 27, 2015

Member

"to fail"+ "on that shard".

@clintongormley

clintongormley May 27, 2015

Member

"to fail"+ "on that shard".

+In practice, one should consider any indexing operation on an index as removing the marker as a flush can be triggered by Elasticsearch
+at any time.
+
+

This comment has been minimized.

@clintongormley

clintongormley May 27, 2015

Member

Perhaps add a note: "It is harmless to request a synced flush while there is ongoing indexing. Shards that are idle will succeed and shards that are not will fail. Any shards that succeeded will have faster recovery times."

@clintongormley

clintongormley May 27, 2015

Member

Perhaps add a note: "It is harmless to request a synced flush while there is ongoing indexing. Shards that are idle will succeed and shards that are not will fail. Any shards that succeeded will have faster recovery times."

+--------------------------------------------------
+
+
+Here is what it looks like when one shard group failed due to pending operations:

This comment has been minimized.

@clintongormley

clintongormley May 27, 2015

Member

add a callout with the returned HTTP status. In fact, should we include the status in the body, like we do for a number of other APIs?

@clintongormley

clintongormley May 27, 2015

Member

add a callout with the returned HTTP status. In fact, should we include the status in the body, like we do for a number of other APIs?

This comment has been minimized.

@bleskes

bleskes May 27, 2015

Member

it's not consistent - seems to be API dependent. Can do, but then I'll have to wrote the index section under indices and that makes _shards weird. I'll add the note.

@bleskes

bleskes May 27, 2015

Member

it's not consistent - seems to be API dependent. Can do, but then I'll have to wrote the index section under indices and that makes _shards weird. I'll add the note.

+
+[source,js]
+--------------------------------------------------
+{

This comment has been minimized.

@clintongormley

clintongormley May 27, 2015

Member

add a callout with the HTTP status

@clintongormley

clintongormley May 27, 2015

Member

add a callout with the HTTP status

@clintongormley

This comment has been minimized.

Show comment
Hide comment
@clintongormley

clintongormley May 27, 2015

Member

Minor doc comments, but looking good!

Member

clintongormley commented May 27, 2015

Minor doc comments, but looking good!

@brwe

This comment has been minimized.

Show comment
Hide comment
@brwe

brwe May 27, 2015

Contributor

this is tagged for 2.0 but should it not also go in 1.6?

Contributor

brwe commented May 27, 2015

this is tagged for 2.0 but should it not also go in 1.6?

@nik9000

This comment has been minimized.

Show comment
Hide comment
@nik9000

nik9000 May 27, 2015

Contributor

this is tagged for 2.0 but should it not also go in 1.6?

Yeah, I was just about to ask that.

Contributor

nik9000 commented May 27, 2015

this is tagged for 2.0 but should it not also go in 1.6?

Yeah, I was just about to ask that.

@bleskes

This comment has been minimized.

Show comment
Hide comment
@bleskes

bleskes May 27, 2015

Member

pushed another update. I was planning the back port PR (which is likely to happen) with 1.6., but can mark this one as well...

Member

bleskes commented May 27, 2015

pushed another update. I was planning the back port PR (which is likely to happen) with 1.6., but can mark this one as well...

@bleskes bleskes added the v1.6.0 label May 27, 2015

@brwe

This comment has been minimized.

Show comment
Hide comment
@brwe

brwe May 28, 2015

Contributor

LGTM but not sure if that counts as a go

Contributor

brwe commented May 28, 2015

LGTM but not sure if that counts as a go

@@ -140,26 +147,7 @@ public void testSyncedFlush() throws ExecutionException, InterruptedException, I
}
@TestLogging("indices:TRACE")

This comment has been minimized.

@s1monw

s1monw May 28, 2015

Contributor

remove the trace here

@s1monw

s1monw May 28, 2015

Contributor

remove the trace here

@s1monw

This comment has been minimized.

Show comment
Hide comment
@s1monw

s1monw May 28, 2015

Contributor

LGTM too

Contributor

s1monw commented May 28, 2015

LGTM too

@brwe brwe merged commit 37bdbe0 into elastic:master May 29, 2015

1 check passed

CLA Commit author is a member of Elasticsearch
Details

@kevinkluge kevinkluge removed the review label May 29, 2015

brwe added a commit that referenced this pull request May 29, 2015

brwe added a commit to brwe/elasticsearch that referenced this pull request May 29, 2015

bleskes added a commit to bleskes/elasticsearch that referenced this pull request Jun 3, 2015

Reduce shard inactivity timeout to 5m
To better distribute the memory allocating to indexing, the IndexingMemoryController periodically checks the different shard for their last indexing activity. If no activity has happened for a while, the controller marks the shards as in active and allocated it's memory buffer budget (but a small minimal budget)  to other active shards.  The recently added synced flush feature (#11179, #11336) uses this inactivity trigger to attempt as  a trigger to attempt adding a sync id marker (which will speed up future recoveries).

We wait for 30m before declaring a shard inactive. However, these days the operation just requires a refresh and is light. We can be stricter (and 5m) increase the chance a synced flush will be triggered.

bleskes added a commit that referenced this pull request Jun 3, 2015

Reduce shard inactivity timeout to 5m
To better distribute the memory allocating to indexing, the IndexingMemoryController periodically checks the different shard for their last indexing activity. If no activity has happened for a while, the controller marks the shards as in active and allocated it's memory buffer budget (but a small minimal budget)  to other active shards.  The recently added synced flush feature (#11179, #11336) uses this inactivity trigger to attempt as  a trigger to attempt adding a sync id marker (which will speed up future recoveries).

We wait for 30m before declaring a shard inactive. However, these days the operation just requires a refresh and is light. We can be stricter (and 5m) increase the chance a synced flush will be triggered.

Closes #11479

bleskes added a commit that referenced this pull request Jun 3, 2015

Reduce shard inactivity timeout to 5m
To better distribute the memory allocating to indexing, the IndexingMemoryController periodically checks the different shard for their last indexing activity. If no activity has happened for a while, the controller marks the shards as in active and allocated it's memory buffer budget (but a small minimal budget)  to other active shards.  The recently added synced flush feature (#11179, #11336) uses this inactivity trigger to attempt as  a trigger to attempt adding a sync id marker (which will speed up future recoveries).

We wait for 30m before declaring a shard inactive. However, these days the operation just requires a refresh and is light. We can be stricter (and 5m) increase the chance a synced flush will be triggered.

Closes #11479

bleskes added a commit that referenced this pull request Jun 3, 2015

Reduce shard inactivity timeout to 5m
To better distribute the memory allocating to indexing, the IndexingMemoryController periodically checks the different shard for their last indexing activity. If no activity has happened for a while, the controller marks the shards as in active and allocated it's memory buffer budget (but a small minimal budget) to other active shards. The recently added synced flush feature (#11179, #11336) uses this inactivity trigger to attempt as a trigger to attempt adding a sync id marker (which will speed up future recoveries).

We wait for 30m before declaring a shard inactive. However, these days the operation just requires a refresh and is light. We can be stricter (and 5m) increase the chance a synced flush will be triggered.

Closes #11479
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment