rgw: avoid stuck worker thread infinitely when there are PG stuck at peering

[guangyy]

At OSD side - if the op has a flag that it does not want to wait, and the PG it is hitting is at peering, reply back immediately with -EAGAIN, rather than queue it into waiting list

At librados side - Extend the API to support a timeout passed from caller, once it is set, it will switch to 'do not wait' mode and retry (polling) at client side until the timeout reaches.

At radosgw - leverage the new API and add a configuration to turn the flag to do timeout when communicating with OSD.

[guangyy]

TODO:

1. Testing - this is completely not tested yet.
2. At librados, instead of busy polling, we properly want some back-off wait.

[guangyy]

@liewegas , @yehudasa , @athanatos , would you please take a look if the general flow makes sense or not?

[liewegas]

@yehudasa is the rgw piece that simple?

[liewegas]

The only place I see that EAGAIN is currently returned by the OSD is when you are sending to replicas and they tell you to go back to the primary. We probably do want to use a different error code. EHOSTDOWN? Or we could define our own...

[loic-bot]

FAILURE: http://jenkins.ceph.dachary.org/job/ceph/6922/

• FAILURE http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/6922/

[yehudasa]

@liewegas @guangyy not sure if we want the timeout to apply to all operations or just very specific operations. If we want it to apply to all rados operations then maybe there are a few more places that need to be changed?

[guangyy]

Hi @liewegas ,
Could you take a look at this one? Thanks.

[loic-bot]

SUCCESS: http://jenkins.ceph.dachary.org/job/ceph/6983/

• SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/6983/
• SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=ubuntu-14.04&&x86_64/6983/

[guangyy]

Fixed according to @liewegas 's comments, add two commits:

1. Expose the option for rados tool
2. Add a test to cover the newly added option

[loic-bot]

SUCCESS: http://jenkins.ceph.dachary.org/job/ceph/7054/

• SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/7054/
• SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=ubuntu-14.04&&x86_64/7054/

[ghost]

@guangyy this needs rebasing

[guangyy]

@dachary , sorry for the delayed response. Rebased.

[ghost]

@guangyy thanks :-)

[loic-bot]

SUCCESS: http://jenkins.ceph.dachary.org/job/ceph/7522/

• SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=ubuntu-14.04&&x86_64/7522/

[athanatos]

I'm a bit worried about this, doesn't it break write ordering on a single object? You wouldn't be able to pipeline ops on a single object while using this feature. That seems awfully brittle.

[liewegas]

Yeah, this would reorder operations. Unless it was only used on reads? (It's ok to reorder those, as we do with RWORDERED vs !RWORDERED)

[liewegas]

Closing this pull request since it appears to be abandoned. If you it is not abandoned, and you are still interested in championing this change, please address any review comments so far and reopen the pull request.