rgw: abort multipart if upload meta object doesn't exist

[fangyuxiangGL]

Hi @cbodley

I am not sure if the phenomenon mentioned below is an issue.

For testing, we configured a lc rule that will delete residual multipart uploaded a day ago, then we configured rgw_lc_debug_interval to 300, which meant that lc rule will delete residual multipart uploaded 300 seconds ago.

Then we started to upload a big object use s3 multipart and got 'invalid part' error finally, which was caused by lc process for the uploading time exceeded 300 seconds and it was reasonable.

But the result was that there were some residual parts from the upload and they can't be deleted by lc process any more, the reason was that upload meta object was deleted from the bucket index in last process by lc, so they will be forever residual parts.

So is it reasonable that we test if upload meta object is existing before set current part to its omap? The part will fail if upload meta object is not existing.

[cbodley]

So is it reasonable that we test if upload meta object is existing before set current part to its omap? The part will fail if upload meta object is not existing.

that does sound reasonable, yeah. i'm guessing we could end up in the same situation with a racing AbortMultipart too? my question is, after detecting EEXIST on the meta object with your approach here, do we still leak the part objects themselves? we might want some cleanup logic in that case that schedules the part for gc (similar to abort_multipart_upload())

[cbodley]

this could use a clarifying comment, ex:

  const bool must_exist = true; // detect races with abort
  r = store->omap_set(raw_meta_obj, p, bl, must_exist);

[fangyuxiangGL]

@cbodley
great idea!

[fangyuxiangGL]

yeah. i'm guessing we could end up in the same situation with a racing AbortMultipart too?

Did you mean that HTTP AbortMultipart race with HTTP PutObject?

my question is, after detecting EEXIST on the meta object with your approach here

yeah, my approach is to detect the meta object exist, in other words meta object related to the upload-id must exist when a part is being uploaded.

do we still leak the part objects themselves?

could you explain what part objects point to?

[fangyuxiangGL]

@cbodley

I think i got u, thanks!

There is one kind of race conditon.
t1 AbortMultipart-send_gc_chain
t2 PutObject-omap_set (successful !!!)
t3 AbortMultipart-delete_obj
then the part uploaded by PutObject will be leaked.

A perfect method maybe need iterate the rados pool upload meta object. Do you have better idea?

[cbodley]

i had something a bit different in mind:

• PutObj starts, read_obj_policy() finds the multipart meta obj, starts transfering data for the part
• AbortMultipart comes in, completes, and removes the meta obj
• PutObj finishes writing the data, and omap_set on the meta obj fails with ENOENT

my idea was to, when PutObj detects ENOENT there, schedule the part for gc

[fangyuxiangGL]

PutObj starts, read_obj_policy() finds the multipart meta obj, starts transfering data for the part
AbortMultipart comes in, completes, and removes the meta obj
PutObj finishes writing the data, and omap_set on the meta obj fails with ENOENT
my idea was to, when PutObj detects ENOENT there, schedule the part for gc

@cbodley If I am not mistaken, the process you mentioned above may not cause part leaked.
Codes in RGWPutObjProcessor_Aio::~RGWPutObjProcessor_Aio() will cleanup rados objects written in that part:

  for (iter = written_objs.begin(); iter != written_objs.end(); ++iter) {
    const rgw_raw_obj& obj = *iter;
    if (!head_obj.empty() && obj == raw_head) {
      ldout(store->ctx(), 5) << "NOTE: we should not process the head object (" << obj << ") here" << dendl;
      need_to_remove_head = true;
      continue;
    }

    int r = store->delete_raw_obj(obj);
    if (r < 0 && r != -ENOENT) {
      ldout(store->ctx(), 5) << "WARNING: failed to remove obj (" << obj << "), leaked" << dendl;
    }
  }

[fangyuxiangGL]

I think this is a kind of race conditon that will cause data leak, but it may be low probability.

There is one kind of race conditon.
t1 AbortMultipart-send_gc_chain
t2 PutObject-omap_set (successful !!!)
t3 AbortMultipart-delete_obj
then the part uploaded by PutObject will be leaked.

[yuriw]

wip-yuri2-testing-2018-01-16-1725