Report shard transfer progress

[xzfc]

Closes #3433.

The shard progress state is tracked locally inside TransferTasksPool (aka Collection::transfer_tasks).
Since it's local, only the sender node shows the progress in the result of GET collections/{name}/cluster.

Output of GET collections/{name}/cluster:

{"result": {
  "shard_transfers": [
    {
      "shard_id": 1,
      "from": 3351940210608703,
      "to": 7522668664620757,
      "sync": false,
      "method": "stream_records",
      ↓ ↓ ↓   This field is added   ↓ ↓ ↓
      "comment": "Transferring records (136000/193725), started 15s ago, ETA: 6.83s"
      ↑ ↑ ↑   This field is added  ↑ ↑ ↑
    }
  ],
  "peer_id": 3351940210608703,
  "local_shards": [
    {
      "shard_id": 1,
      "points_count": 178839,
      "state": "Active"
    }, …
  ],
  "remote_shards": [
    {
      "shard_id": 1,
      "peer_id": 7522668664620757,
      "state": "Partial"
    }, …
  ], …
}, … }

---

All Submissions:

• Contributions should target the dev branch. Did you create your branch from dev?
• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

1. Does your submission pass tests?
2. Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
3. Have you checked your code using cargo clippy --all --all-features command?

Changes to Core Features:

• Have you added an explanation of what your changes do and why you'd like us to include them?
• Have you written new tests for your core changes, as applicable?
• Have you successfully ran tests with your changes locally?

[agourlay]

"eta_seconds": 73.496774076

I think truncating to two decimals should be enough precision :)

[xzfc]

I think truncating to two decimals should be enough precision :)

Done.

[agourlay]

Thanks for the tests and switching to the ringbuffer crate 👍

[xzfc]

After a discussion, switched to a human-readable text format, e.g.

"comment": "Transferring records (136000/193725), started 15s ago, ETA: 6.83s"

[agourlay]

After a discussion, switched to a human-readable text format, e.g.

Can you please share why this decision was taken? :)

[xzfc]

@agourlay
The text-based format is enough for debugging purposes, and it is flexible enough to cover other transfer types.
OTOH, a strict schema will be too complicated and we don't see any meaningful way to use it in automation.

Like, we can't just to put these values on Grafana dashboard: if we aggregate on points_transferred and points_total over all transfers, we'll get saw graphs which is not very informative.

We want to add total active transfers count metric later to put it on Grafana, but it's outside scope of this PR.

Also, while previous schema was fine for record streaming, the delta/snapshot transfer will complicate reporting by introducing steps (i.e. first download a blob, then transfer records), and we do not want to complicate the schema because of it.

We considered these alternatives:

• Just percentage.

  { "progress": 35.0 }
  

• Generic counter name.

  { "items_transferred": 123, "items_total": 12345 }
  

• Multiple counters.

  { "points_transferred": …, "points_total": …,
    "bytes_transferred”: …, “bytes_total”: … }
  

• Dynamic JSON object. e.g. progress: Map<String, ProgressItem>

  { "points": { "transferred": 30, "total": 12345 },
    "snapshot": { "transferred": 12345, "total": 12345 },
    "…": { … }
  }
  

• String just for users, not for machines.

  { "progress": "Step 2/4: Transferring records (123/12345)…" }
  

• Steps counter.

  { "step_current": 2,
    "step_total": 4,
    "step_progress_current": 123,
    "step_progress_total": 12345 }
  

[timvisee]

🙌