Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fair Share Replication Scheduler Implementation (3.x) #3364

Merged
merged 4 commits into from Mar 11, 2021
Merged

Fair Share Replication Scheduler Implementation (3.x) #3364

merged 4 commits into from Mar 11, 2021

Commits on Mar 11, 2021

  1. Move replicator #job record to couch_replicator.hrl 

    This is needed to prepare for the Fair Share scheduler feature since
both the scheduler and the fair share module will end up referencing
the #job record.
    @nickva
    nickva committed Mar 11, 2021
    Copy the full SHA
    113614b View commit details
    Browse the repository at this point in the history

  2. Fair Share Replication Scheduler Implementation 

    Fair share replication scheduler allows configuring job priorities
per-replicator db.

Previously jobs from all the replication dbs would be added to the scheduler
and run in a round-robin order. This update makes it possible to specify the
relative priority of jobs from different databases. For example, there could be
low, high and default priority _replicator dbs.

The original algorithm comes from the [A Fair Share
Scheduler](https://proteusmaster.urcf.drexel.edu/urcfwiki/images/KayLauderFairShare.pdf
"Fair Share Scheduler") paper by Judy Kay and Piers Lauder. A summary of how
the algorithm works is included in the top level comment in the
couch_replicator_share module.

There is minimal modification to the main scheduler logic. Besides the
share accounting logic each cycle, the other changes are:

  * Running and stopping candidates are now picked based on the priority first,
    and then on their last_started timestamp.

  * When jobs finish executing mid-cycle, their charges are accounted for. That
    holds for jobs which terminate normally, are removed by the user, or crash.

Other interesting aspects are the interaction with the error back-off mechanism
and how one-shot replications are treated:

  * The exponential error back-off mechanism is unaltered and takes precedence
  over the priority values. That means unhealthy jobs are rejected and
  "penalized" before the priority value is even looked at.

  * One-shot replications, once started, are not stopped during each scheduling
  cycle unless the operator manually adjusts the `max_jobs` parameter. That
  behavior is necessary to preserve the "snapshot" semantics and is retained in
  this update.
    @nickva
    nickva committed Mar 11, 2021
    Copy the full SHA
    dcc4598 View commit details
    Browse the repository at this point in the history

  3. [fixup] Use =< when clearing 0 entries from priority and usage tables

    @nickva
    nickva committed Mar 11, 2021
    Copy the full SHA
    12f5fa0 View commit details
    Browse the repository at this point in the history

  4. [fixup] remove extra blank line

    @nickva
    nickva committed Mar 11, 2021
    Copy the full SHA
    7ad40ff View commit details
    Browse the repository at this point in the history