Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce shard inactivity timeout to 5m #11479

Closed
wants to merge 1 commit into from

Conversation

Projects
None yet
4 participants
@bleskes
Copy link
Member

bleskes commented Jun 3, 2015

To better distribute the memory allocating to indexing, the IndexingMemoryController periodically checks the different shard for their last indexing activity. If no activity has happened for a while, the controller marks the shards as in active and allocated it's memory buffer budget (but a small minimal budget) to other active shards. The recently added synced flush feature (#11179, #11336) uses this inactivity trigger to attempt as a trigger to attempt adding a sync id marker (which will speed up future recoveries).

We wait for 30m before declaring a shard inactive. However, these days the operation just requires a refresh and is light. We can be stricter (and 5m) increase the chance a synced flush will be triggered.

Reduce shard inactivity timeout to 5m
To better distribute the memory allocating to indexing, the IndexingMemoryController periodically checks the different shard for their last indexing activity. If no activity has happened for a while, the controller marks the shards as in active and allocated it's memory buffer budget (but a small minimal budget)  to other active shards.  The recently added synced flush feature (#11179, #11336) uses this inactivity trigger to attempt as  a trigger to attempt adding a sync id marker (which will speed up future recoveries).

We wait for 30m before declaring a shard inactive. However, these days the operation just requires a refresh and is light. We can be stricter (and 5m) increase the chance a synced flush will be triggered.
@bleskes

This comment has been minimized.

Copy link
Member Author

bleskes commented Jun 3, 2015

@s1monw can you have a look?

@s1monw

This comment has been minimized.

Copy link
Contributor

s1monw commented Jun 3, 2015

LGTM

bleskes added a commit that referenced this pull request Jun 3, 2015

Reduce shard inactivity timeout to 5m
To better distribute the memory allocating to indexing, the IndexingMemoryController periodically checks the different shard for their last indexing activity. If no activity has happened for a while, the controller marks the shards as in active and allocated it's memory buffer budget (but a small minimal budget)  to other active shards.  The recently added synced flush feature (#11179, #11336) uses this inactivity trigger to attempt as  a trigger to attempt adding a sync id marker (which will speed up future recoveries).

We wait for 30m before declaring a shard inactive. However, these days the operation just requires a refresh and is light. We can be stricter (and 5m) increase the chance a synced flush will be triggered.

Closes #11479

bleskes added a commit that referenced this pull request Jun 3, 2015

Reduce shard inactivity timeout to 5m
To better distribute the memory allocating to indexing, the IndexingMemoryController periodically checks the different shard for their last indexing activity. If no activity has happened for a while, the controller marks the shards as in active and allocated it's memory buffer budget (but a small minimal budget)  to other active shards.  The recently added synced flush feature (#11179, #11336) uses this inactivity trigger to attempt as  a trigger to attempt adding a sync id marker (which will speed up future recoveries).

We wait for 30m before declaring a shard inactive. However, these days the operation just requires a refresh and is light. We can be stricter (and 5m) increase the chance a synced flush will be triggered.

Closes #11479

@bleskes bleskes closed this in 26d71fe Jun 3, 2015

@kevinkluge kevinkluge removed the review label Jun 3, 2015

@bleskes bleskes deleted the bleskes:inactive_5m branch Jun 3, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.