Replication extremely slow with millions of changes pending

[byrnemluke]

Description

Continuous replication of https://skimdb.npmjs.com/registry gets extremely slow over time - it worked fine for up to 400k documents, then became extremely slow. The entire dataset contains 1.6M documents.

Here is the active task

[
  {
    "node": "couchdb@localhost",
    "pid": "<0.1300.89>",
    "changes_pending": 13442245,
    "checkpoint_interval": 30000,
    "checkpointed_source_seq": 611203,
    "continuous": true,
    "database": "shards/80000000-ffffffff/_replicator.1622573219",
    "doc_id": "ad71dbf4189be812bfb41c30d6000da6",
    "doc_write_failures": 1,
    "docs_read": 533605,
    "docs_written": 533604,
    "missing_revisions_found": 533605,
    "replication_id": "a5df5f8f23b0d94f28a1ed02d4a2f103+continuous",
    "revisions_checked": 540275,
    "source": "https://skimdb.npmjs.com/registry/",
    "source_seq": 612872,
    "started_on": 1622581539,
    "target": "http://localhost:5984/npm_mirror/",
    "through_seq": 612872,
    "type": "replication",
    "updated_on": 1622581842,
    "user": "admin"
  },
  {
    "node": "couchdb@localhost",
    "pid": "<0.22157.91>",
    "changes_done": 6107,
    "database": "shards/00000000-7fffffff/npm_mirror.1622573154",
    "phase": "document_copy",
    "progress": 2,
    "started_on": 1622581839,
    "total_changes": 268814,
    "type": "database_compaction",
    "updated_on": 1622581841
  }
]

Steps to Reproduce

Using the Couchbase GUI:

1. Navigate to replication
2. Set the remote source as https://skimdb.npmjs.com/registry
3. Set the remote as a new local database

Expected Behaviour

For the replication to run more quickly than it does (3+ hours to get through ~30% of the replication).

Your Environment

We're running 1 central instance locally for now.

{
  "couchdb": "Welcome",
  "version": "3.1.1",
  "git_sha": "ce596c65d",
  "uuid": "5e70eea19ba38ca970bd9a8020677dbf",
  "features": [
    "access-ready",
    "partitioned",
    "pluggable-storage-engines",
    "reshard",
    "scheduler"
  ],
  "vendor": {
    "name": "The Apache Software Foundation"
  }
}

• CouchDB version used:
• Browser name and version:
• Operating system and version:

Additional Context

We have no attachments, just JSON documents.

[byrnemluke]

Issue resolved when I set retries_per_request from 20 to 5