Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introducing a new snapshot segments threadpool to uploads segments of shards in parallel #39657

Closed
wants to merge 1 commit into from
Closed

Introducing a new snapshot segments threadpool to uploads segments of shards in parallel #39657

wants to merge 1 commit into from

Commits on Mar 4, 2019

  1. Introducing a new snapshot segments threadpool to uploads segments of… 

    … shards in parallel.

Current Behavior:
The snapshot thread pool max size is determined by the number of processors by 2
capped at 5. Each snapshot thread works on snapshotting one shard. So, at any point
in time at max there will be 5 shards uploaded in parallel. This works good
for cases where the number of shards to be snapshotted is greater than 5. But,
for cases where there are lesser number of shards, the snapshot threads are
under utilized. This problem is more visible towards the end of a snapshot, where
snapshot is complete for most of the shards and in progress for 1 or 2 shards.

Proposed Behavior:
Introducing a new prioritized thread pool to upload segements of various shards in
parallel, this will prioritize uploading segments of one shard at a time. Thus the
snapshot processing capacity is fully utilized throughout the entire span of snapshot.
Prioritizing one shard also results in releasing the shard lock (held by snapshot) sooner.
    Prabhakar Sithanandam committed Mar 4, 2019
    Configuration menu
    Copy the full SHA
    2c3d4b7 View commit details
    Browse the repository at this point in the history