Return detailed replication stats for running and pending jobs

[nickva]

Overview

Previously _scheduled/docs returned detailed replication statistics for
completed jobs only. To get the same level of details from a running or pending
jobs users had to use _active_tasks, which is not optimal and required jumping
between monitoring endpoints.

info field was originally meant to hold these statistics but they were not
implemented and it just returned null as a placeholder. With work for 3.0
finalizing, this might be a good time to add this improvement to avoid
disturbing the API afterwards.

Just updating the _scheduler/docs was not quite enough since, replications
started from the _replicate endpoint would not be visible there and users
would still have to access _active_tasks to get inspect them, so let's add
the info field to the _scheduler/jobs as well.

After this update, all states and status details from _active_tasks and
_replicator docs should be available under _scheduler/jobs and
_scheduler/docs endpoints.

Testing recommendations

1. Run couch_replicator eunit tests

(edit: unit test added for the test below as well)
2. Create a few replications and monitor _scheduler/jobs and _scheduler/docs
3. Make those jobs pending by lowering the max_jobs to 0
4. Inspect _scheduler/jobs and docs. Stats should still be showing even when the jobs are pending.
5. Delete the max_jobs setting and wait for jobs to start running against. They should keep their stats and update them when new docs are replicated over.

Related Issues or Pull Requests

Checklist

• Code is written and works correctly
• Changes are covered by tests
• A PR for documentation changes will be made after discussion

[nickva]

Marking it as as draft pull request in order to start the discussion on the mailing list since it involves API changes

[nickva]

Example of _scheduler/docs response:

{
  "total_rows": 1,
  "offset": 0,
  "docs": [
    {
      "database": "_replicator",
      "doc_id": "2e5df7bf99f1a9d06feef6301300055f",
      "id": "d99e532c1129e9cacbf7ed085deca509+continuous",
      "node": "node1@127.0.0.1",
      "source": "http://adm:*****@localhost:15984/src/",
      "target": "http://adm:*****@localhost:15984/tgt/",
      "state": "pending",
      "info": {
        "revisions_checked": 113,
        "missing_revisions_found": 113,
        "docs_read": 113,
        "docs_written": 113,
        "changes_pending": 0,
        "doc_write_failures": 0,
        "checkpointed_source_seq": "113-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTV0MDQy1zMAQsMckEQiQ1L9____szKYE01ygQLsZsYGqcamiZjKcRqRxwIkGRqA1H-oSbZgk1KMLCzTDE0wdWUBAF6HJIQ",
        "source_seq": "113-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTV0MDQy1zMAQsMckEQiQ1L9____szKYE01ygQLsZsYGqcamiZjKcRqRxwIkGRqA1H-oSbZgk1KMLCzTDE0wdWUBAF6HJIQ",
        "through_seq": "113-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTV0MDQy1zMAQsMckEQiQ1L9____szKYE01ygQLsZsYGqcamiZjKcRqRxwIkGRqA1H-oSbZgk1KMLCzTDE0wdWUBAF6HJIQ"
      },
      "error_count": 0,
      "last_updated": "2019-10-31T06:46:11Z",
      "start_time": "2019-10-31T06:42:16Z",
      "source_proxy": null,
      "target_proxy": null
    }
  ]
}

[nickva]

Example of _scheduler/jobs response:

{
    "jobs": [
        {
            "database": "_replicator",
            "doc_id": "2e5df7bf99f1a9d06feef6301300055f",
            "history": [
                {
                    "timestamp": "2019-10-31T06:46:11Z",
                    "type": "stopped"
                },
                {
                    "timestamp": "2019-10-31T06:42:16Z",
                    "type": "started"
                },
                {
                    "timestamp": "2019-10-31T06:42:16Z",
                    "type": "added"
                }
            ],
            "id": "d99e532c1129e9cacbf7ed085deca509+continuous",
            "info": {
                "changes_pending": 0,
                "checkpointed_source_seq": "113-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTV0MDQy1zMAQsMckEQiQ1L9____szKYE01ygQLsZsYGqcamiZjKcRqRxwIkGRqA1H-oSbZgk1KMLCzTDE0wdWUBAF6HJIQ",
                "doc_write_failures": 0,
                "docs_read": 113,
                "docs_written": 113,
                "missing_revisions_found": 113,
                "revisions_checked": 113,
                "source_seq": "113-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTV0MDQy1zMAQsMckEQiQ1L9____szKYE01ygQLsZsYGqcamiZjKcRqRxwIkGRqA1H-oSbZgk1KMLCzTDE0wdWUBAF6HJIQ",
                "through_seq": "113-g1AAAACTeJzLYWBgYMpgTmHgz8tPSTV0MDQy1zMAQsMckEQiQ1L9____szKYE01ygQLsZsYGqcamiZjKcRqRxwIkGRqA1H-oSbZgk1KMLCzTDE0wdWUBAF6HJIQ"
            },
            "node": "node1@127.0.0.1",
            "pid": null,
            "source": "http://adm:*****@localhost:15984/src/",
            "start_time": "2019-10-31T06:42:16Z",
            "target": "http://adm:*****@localhost:15984/tgt/",
            "user": null
        }
    ],
    "offset": 0,
    "total_rows": 1
}

[nickva]

Vote passed on the mailing list: https://lists.apache.org/thread.html/799772bc092c473f06c16c97330ce8abe6912d5afd9ac431ef34e666@%3Cdev.couchdb.apache.org%3E

Opening PR

[davisp]

Should we add your pending test you described as an actual test?

[nickva]

@davisp Good idea. Added the test.

There was a test that already check active_tasks details were retained so modified to check that scheduler also has required fields both when it is running and pending.

[davisp]

+1