Permalink
Browse files

Schedule vbucket replications in small batches

Previously, xdcr manager would trigger couch replications for all
active vbuckets simultaneously which was causing performance to
suffer. The new approach, instead, schedules replications for only a
small number of vbuckets (32 by default and configurable through an
environment variable called MAX_CONCURRENT_REPS_PER_DOC) at any given
time.

This change set only supports xdcr continuous mode replication. We
start by scheduling the specified number of vbuckets (in couch mode
non-continuous) and periodicaly monitor them. As one or more of them
are done, new vbuckets are scheduled. After we've run through all of
them in this manner, we repeat the process.

An alternate approach is likely better where we can assign a time
slot for each vbucket instead of letting it run through completion
before scheduling another. Implementing this approach would be
greatly simplified if the Couch replicator API supported pause and
resume functions. Until then, we can stick to the current approach.

Also, I found it harded to break this change set into smaller pieces.
If reviewing seems hard or if there's a better way to organize stuff,
please let me know.

Change-Id: Ifc905becb41296e68ee66dbdc73b912bc055d15a
Reviewed-on: http://review.couchbase.org/13663
Tested-by: Srinivas Vadlamani <srinivas@couchbase.com>
Reviewed-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
Tested-by: Aliaksey Kandratsenka <alkondratenko@gmail.com>
  • Loading branch information...
1 parent 33cf577 commit 6de88fa79bb849216668034b93fa81c4505221da Srinivas Vadlamani committed with alk Feb 14, 2012
Showing with 97 additions and 192 deletions.
  1. +97 −192 src/xdc_rep_manager.erl
Oops, something went wrong.

0 comments on commit 6de88fa

Please sign in to comment.