New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rgw: create a tool for orphaned objects cleanup #5717

Merged
22 commits merged into from Sep 3, 2015

Conversation

Projects
None yet
4 participants
@ghost

ghost commented Aug 30, 2015

@ghost ghost self-assigned this Aug 30, 2015

@ghost ghost added this to the hammer milestone Aug 30, 2015

@ghost ghost added bug fix rgw labels Aug 30, 2015

ghost pushed a commit that referenced this pull request Aug 30, 2015

Merge pull request #5717: rgw: create a tool for orphaned objects cle…
…anup

Reviewed-by: Loic Dachary <ldachary@redhat.com>
@ghost

This comment has been minimized.

ghost commented Aug 30, 2015

depends on the 7c43e06 commit from #5612 which is included in this pull request as 8da2fb1

prallabh and others added some commits Apr 8, 2015

RGW: Make RADOS handles in RGW to be a configurable option
Having a configurable number of RADOS handles is resulting
in significant performance boost for all types of workloads.
Each RGW worker thread would now get to pick a RADOS handle
for its lifetime, from the available bunch.

Signed-off-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>
(cherry picked from commit b3c3a24)
rgw: initial work of orphan detection tool implementation
So far doesn't do much, iterate through all objects in a specific pool
data, store it in a sharded index.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 767fc29)
rgw-admin: build index of bucket indexes
for the orphan search tool

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 16a2dbd)
Avoid an extra read on the atomic variable
Signed-off-by: Pavan Rallabhandi <pavan.rallabhandi@sandisk.com>
(cherry picked from commit 7476e94)
rgw: async object stat functionality
An async functionality that stats object.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 13adf3c)
rgw: iterate asynchronously over linked objects
Read objects manifest. So that we could keep the relevant info later.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 05a953d)
rgw: add rgw_obj::parse_raw_oid()
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 7c1aa83)
rgw: iterate over linked objects, store them
only keep part of the oid name if it is in a namespace.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 75902fd)
rgw: keep accurate state for linked objects orphan scan
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit f19b2f0)
rgw: compare oids and dump leaked objects
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 1bc63d9)
rgw: modify orphan search fingerprints
Now works with multipart uploads.

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit fa61ea2)
rgw: list_objects() sets namespace appropriately
list_objects() now uses parse_raw_oid(), so that it can set the correct
namespace. It only affects users of the function that want to get all
objects in bucket, regardless to the namespace associated with it. This
makes it so that the orphan code actually works now with namespaced
objects, and with special named objects (namely, start with underscore).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 20bd490)
rgw: stat_async() sets the object locator appropriately
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit d7d1171)
rgw: cannot re-init an orphan scan job
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit f899310)
radosgw-admin: orphans finish command
A command to remove orphans data

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 55d6f5e)
radosgw-admin: stat orphan objects before reporting leakage
We don't want to report new objects as leaked, because they might just
be written, and waiting to be linked to a new object. The number of
seconds prior to the job initialization can be cofigured through
--orphan-stale-secs (by default it's 24h).

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 66b0090)
radosgw-admin: simplify orphan command
No need for --init-search

Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 4980cbd)
rgw: orphan, fix truncated detection
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit d8ce04d)
rgw: change error output related to orphans
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit cac57ca)
rgw: orphans, change default number of shards
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 5528f21)
rgw: orphans, fix check on number of shards
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 47edec3)
rgw: api adjustment following a rebase
Signed-off-by: Yehuda Sadeh <yehuda@redhat.com>
(cherry picked from commit 7cf1f37)

ghost pushed a commit that referenced this pull request Aug 30, 2015

Merge pull request #5717: rgw: create a tool for orphaned objects cle…
…anup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ghost pushed a commit that referenced this pull request Aug 30, 2015

Merge pull request #5717: rgw: create a tool for orphaned objects cle…
…anup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ghost pushed a commit that referenced this pull request Aug 31, 2015

Merge pull request #5717: rgw: create a tool for orphaned objects cle…
…anup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ghost pushed a commit that referenced this pull request Aug 31, 2015

Merge pull request #5717: rgw: create a tool for orphaned objects cle…
…anup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ghost pushed a commit that referenced this pull request Aug 31, 2015

Merge pull request #5717: rgw: create a tool for orphaned objects cle…
…anup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ghost pushed a commit that referenced this pull request Sep 1, 2015

Merge pull request #5717: rgw: create a tool for orphaned objects cle…
…anup

Reviewed-by: Loic Dachary <ldachary@redhat.com>

ghost pushed a commit that referenced this pull request Sep 2, 2015

Merge pull request #5717: rgw: create a tool for orphaned objects cle…
…anup

Reviewed-by: Loic Dachary <ldachary@redhat.com>
@ghost

This comment has been minimized.

ghost commented Sep 2, 2015

@oritwas does this backport look good to merge ? It passed a run of the hammer rgw suite ( see http://pulpito.ceph.com/loic-2015-09-01_17:25:32-rgw-hammer-backports---basic-multi/ ).

@oritwas

This comment has been minimized.

Contributor

oritwas commented Sep 3, 2015

looks good to merge

ghost pushed a commit that referenced this pull request Sep 3, 2015

Loic Dachary
Merge pull request #5717 from dachary/wip-12591-hammer
rgw: create a tool for orphaned objects cleanup

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

@ghost ghost merged commit 6b02be7 into ceph:hammer Sep 3, 2015

This issue was closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment