LifeCycle feature

[chenji-kael]

As same as amazon S3 interface,"PUT Bucket lifecycle" and
"DELETE Bucket lifecycle" have been implemented,
"GET Bucket lifecycle" not realized yet as S3cmd has not
realize it also.
The feature`s main point is to remove expire file per day.
As ceph does not have a tier concept, so files transfer
from hot layer to cold layer is also not supported.

lifecycle config file format:


sample-rule

enable

1

Signed-off-by: Ji Chen insomnia@139.com

[loic-bot]

FAILURE: http://jenkins.ceph.dachary.org/job/ceph/8523/

• FAILURE http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/8523/

[loic-bot]

FAILURE: http://jenkins.ceph.dachary.org/job/ceph/8524/

• FAILURE http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/8524/

[loic-bot]

SUCCESS: http://jenkins.ceph.dachary.org/job/ceph/8527/

• SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/8527/

[yehudasa]

@chenji-kael this is very interesting, looks like a good start. I do see it needs some cleanups, and we'll need to take a closer look at it to make sure it conforms to what we had in mind when thinking about this feature. Assigining it to @dang to take a closer look.

[chenji-kael]

@yehudasa ping

[yehudasa]

@chenji-kael there are a few problems that I see. It still does it using the bucket name, not the bucket instance which is problematic. I also think that modifying the bucket metadata is problematic because it's racy (can race with another change to the bucket metadata). I still think we need to keep the list of bucket instances that need to be scanned in a separate sharded omap. Each gateway will try to grab a lease on a shard, then process it. This way multiple gateways can coordinate the work.

[chenji-kael]

@yehudasa OK,I will get it fix soon

[chenji-kael]

@yehudasa ping
Hi, I have changed a lot, Now save lc bucket list in shard object. and use "bucket_name:bucket_id" as the key.Please check it, thanks

[chenji-kael]

@yehudasa ping

[yehudasa]

@chenji-kael haven't forgotten you. Will try to get it reviewed soon. Thanks!

[dang]

@chenji-kael
Can you rebase on master? You can probably resolve conflicts faster than I can.

[chenji-kael]

@dang hi,rebase has been done

[loic-bot]

FAILURE: http://jenkins.ceph.dachary.org/job/ceph/9888/

• FAILURE http://jenkins.ceph.dachary.org/job/ceph/LABELS=centos-7&&x86_64/9888/

[ghost]

@chenji-kael please ignore the bot failure, it is a false negative (see http://tracker.ceph.com/issues/13997 for more information). You can re-schedule a job by rebasing your branch and repushing.

[dang]

@chenji-kael The call to get_bucket_info() needs to be updated for tenants from commit f7ca00a this is causing me build failures.

I suspect this will need a new argument, but I'm not sure.

Also, can you add your new files to src/CMakeList.txt?

[chenji-kael]

@dang sorry for have not noticed tenant has been merged, now it works fine, please check it, thanks

[loic-bot]

FAILURE: http://jenkins.ceph.dachary.org/job/ceph/9949/

• FAILURE http://jenkins.ceph.dachary.org/job/ceph/LABELS=ubuntu-14.04&&x86_64/9949/

[loic-bot]

SUCCESS: http://jenkins.ceph.dachary.org/job/ceph/9950/

• SUCCESS http://jenkins.ceph.dachary.org/job/ceph/LABELS=ubuntu-14.04&&x86_64/9950/

[yehudasa]

@chenji-kael @dang I reverted the merge off github for now. There still are some issues that I think will need to be fixed. Note that I pushed some fixes to wip-rgw-lifecycle, but we'll need to do some more work before we can merge it.

[chenji-kael]

@yehudasa @dang Glad to hear that, I will do more test, Thanks guys!