mgr/rbd_support: support scheduling long-running background operations

[dillaman]

Certain RBD operations might take a long (blocking) amount of time so it would be preferable if we offered a way to schedule these tasks to run in the background. We would expose the ability to manage these background tasks via the MGR's "rbd_support" module via "ceph" CLI commands similar to the following:

ceph rbd task add flatten <image-spec>
ceph rbd task add remove <image-spec>
ceph rbd task add trash remove <image-id-spec>
ceph rbd task add migration execute <image-spec>
ceph rbd task add migration commit <image-spec>
ceph rbd task add migration abort <image-spec>
ceph rbd task cancel <task-id>
ceph rbd task list

The ceph-csi will be one of the first to utilize these new background tasks to offload its workload and support proper re-starting of in-progress events.

[rjfd]

@dillaman FYI, I opened #29048 yesterday to integrate progress mgr module events in the dashboard.

[dillaman]

@rjfd No rush, but when you get a chance, can you review my changes to the progress? I just want to ensure we are on the same page for how to record the events to support the dashboard. Right now, here is an example:

$ ceph progress json
{
  "events": [],
  "completed": [
    {
      "id": "b586a7a8-3100-4819-9a1c-8dbe08bb7b75",
      "message": "Removing image rbd_tasks/50459f17-f529-4d2e-98de-4798d9c547b7",
      "refs": {
        "origin": "rbd_support",
        "action": "remove",
        "pool_name": "rbd_tasks",
        "pool_namespace": "",
        "image_name": "50459f17-f529-4d2e-98de-4798d9c547b7"
      },
      "started_at": 1563900987.7811384,
      "finished_at": 1563900989.6917968,
      "failed": true,
      "failure_message": "Operation canceled"
    },
    {
      "id": "8f144eaf-deb9-46d9-aad4-1f95746ab8d2",
      "message": "Removing image rbd/image",
      "refs": {
        "origin": "rbd_support",
        "action": "remove",
        "pool_name": "rbd",
        "pool_namespace": "",
        "image_name": "image"
      },
      "started_at": 1563901014.1282082,
      "finished_at": 1563901038.7095985
    }
  ]
}

[rjfd]

progress module changes look good, thanks!

[dillaman]

@rjfd ... and if the dashboard wants to translate the rbd_support progress events to something it knows about, the ev.refs["action"] field now includes the action type [1].

[1] 8774317#diff-0eb9731567425c7d4afb3332c908c4fcR67-R72

[rjfd]

@rjfd ... and if the dashboard wants to translate the rbd_support progress events to something it knows about, the ev.refs["action"] field now includes the action type [1].

[1] 8774317#diff-0eb9731567425c7d4afb3332c908c4fcR67-R72

That's great! This way the problem I had with knowing the "event type" is solved with the pair refs["origin"], refs["action"].

[trociny]

@dillaman There are some issues when testing on python2.7.

[trociny]

And yet another thing noticed. When the test failed for some reason while there were tasks in the mgr in-memory queue, and then the pool was deleted, the task remained int the mgr in-memory queue until it was restarted.

[dillaman]

The new rbd_tasks teuthology test is still failing [1]. Will investigate.

[1] http://pulpito.ceph.com/jdillaman-2019-07-25_10:51:34-rbd-wip-jd-testing-distro-basic-smithi/

[dillaman]

Teuthology test is now passing:

http://pulpito.ceph.com/jdillaman-2019-07-25_15:24:54-rbd-wip-jd-testing-distro-basic-smithi/
http://pulpito.ceph.com/jdillaman-2019-07-25_15:25:24-rbd-wip-jd-testing-distro-basic-smithi/

[trociny]

And just as note, I suppose this this a bug in progress module, but when displaying a task progress (either with ceph progress or ceph status) the output looks like below:

  progress:
    Removing image test_task/XXXMG (since 00h 00m 54s)
      [========================......]

And this since 00h 00m 54s looks confusing to me. Shouldn't it show the task start time? Or does it mean task duration? Then I would use e.g. for 00h 00m 54s , not since ....

[rjfd]

And this since 00h 00m 54s looks confusing to me. Shouldn't it show the task start time? Or does it mean task duration? Then I would use e.g. for 00h 00m 54s , not since ....

Yes, it's the duration. And I agree "since ..." is misleading.

[dillaman]

I added a tracker ticket [1] for the unrelated progress issue.

[1] http://tracker.ceph.com/issues/40976

[trociny]

LGTM

[trociny]

Just cosmetic. When remove is canceled, we have in the mgr log:

2019-07-26T14:23:17.338+0300 7f59a97d1700 -1 librbd::TrimRequest: trim encountered an error: (125) Operation canceled
2019-07-26T14:23:17.338+0300 7f59a8fd0700 -1 librbd::image::RemoveRequest: 0x56096bf6f1e0 handle_trim_image: failed to remove some object(s): (125) Operation canceled

It does not look very bad, as I suppose for a user from the error message it is clear that it is not actually an error but just cancel, but I can imagine it could be improved if you think it is worth doing.

[dillaman]

It does not look very bad, as I suppose for a user from the error message it is clear that it is not actually an error but just cancel, but I can imagine it could be improved if you think it is worth doing.

It could be improved in a later PR I think (throughout since it will affect more than just removal I'd suspect).