Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

jewel: rgw: automated trimming of datalog and mdlog #20061

Merged
merged 37 commits into from Feb 23, 2018
Merged

jewel: rgw: automated trimming of datalog and mdlog #20061

merged 37 commits into from Feb 23, 2018

Conversation

cbodley
Copy link
Contributor

@cbodley cbodley commented Jan 22, 2018 •
edited

backports datalog trim (#10372 and #10546) and mdlog trim (#13111) for jewel

Fixes: http://tracker.ceph.com/issues/18277

@cbodley
rgw: use RGWShardCollectCR for RGWReadDataSyncStatusCoroutine
ce041f0 
this allows us to limit the number of outstanding requests for shard
markers

there also appeared to be issues with spawning the shard CRs
from RGWReadDataSyncStatusCoroutine::handle_data(), because
handle_data() was returning before the shard CRs completed

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 30823c5)
@cbodley
rgw: add dout_prefix for rgw_data_sync.cc
1f9eb57 
and took out the redundant 'rgw' from 'rgw meta sync:'

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 821c70d)
@cbodley
rgw: add json decoders for data sync status
9d2090b 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit ebbb70b)
@cbodley
rgw: expose sync managers through RGWRados
50971dc 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit ccef4b0)
@cbodley
rgw: add rest handlers to query sync status
3a46af7 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit a66b4cc)

Conflicts: src/rgw/rgw_rest_log.cc
   jewel/v10.2.10 already has "rgw: change metadata read_sync_status interface"
   so we need to use the read_sync_status() call in
   src/rgw/rgw_rest_log.cc.
@cbodley
rgw: enable async calls to time_log_trim
691bfec 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit d67436b)
@cbodley
rgw: add RGWRadosTimelogTrimCR
0fa2db4 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 20f31a4)
@cbodley
rgw: add RGWDataLogTrimCR
9b20b55 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 6a366f9)
@cbodley
rgw: add RGWSyncLogTrimThread to RGWRados
446d8b2 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 3b674bb)
@cbodley
rgw: pass cr registry to managers
608430f 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit ef4d7ea)

Conflicts:
	src/rgw/rgw_data_sync.cc (this is a re-cherry-pick that includes
                the RGWSyncLogTrimThread piece that was not available
                as of 87cb847)
@cbodley
rgw: use factory function to hide RGWDataLogTrimCR
67f81c4 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 5ded7ee)

Conflicts:
	src/rgw/rgw_data_sync.h (sync modules not backported)
@cbodley
rgw: add static function to generate lock cookie
3bea2a3 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 9a1b235)
@cbodley
rgw: hold lock for data log trimming
f885c7b 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 0777fff)
@cbodley
rgw: DataLogTrimCR renamed zone to zone_id
1ad8d2a 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit f8d9ac6)
@cbodley
rgw: skip sync thread if current period is empty
86d670e 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 5deabd3)
@cbodley
rgw: move timelog trim wrapper to header
c41bdb7 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit db79bfa)
@cbodley
rgw: move coroutines out of anonymous namespace
34f5f98 
anonymous namespaces do terrible things to name mangling, and this shows
up in our coroutine logging

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit fadfc16)
@cbodley
rgw: add objv_tracker arg to RGWRados::put_system_obj_data
f97015a 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2cee82f)

Conflicts: rgw_obj vs rgw_raw_obj
	src/rgw/rgw_cache.h
	src/rgw/rgw_rados.cc
	src/rgw/rgw_rados.h
@cbodley
rgw: add objv_tracker arg to RGWSimpleRadosRead/WriteCR
80b7b7e 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit ca462a3)

Conflicts: rgw_obj vs rgw_raw_obj
	src/rgw/rgw_cr_rados.cc
	src/rgw/rgw_cr_rados.h
@cbodley cbodley added this to the jewel milestone Jan 22, 2018
@cbodley cbodley force-pushed the wip-jewel-rgw-log-trim branch 2 times, most recently from 23bdb0a to 215aba6 Compare January 23, 2018 15:45
@cbodley
rgw: add RGWRadosRemoveCR
67addc2 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 57b9489)

Conflicts: rgw_obj vs rgr_raw_obj
        src/rgw/rgw_sync.cc
@cbodley
rgw: use objv_tracker for mdlog history
ea170d0 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 2c78973)
@cbodley
rgw: add CRs for async mdlog history operations
fbfa4c3 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 70bcf45)

Conflicts: rgw_raw_obj
        src/rgw/rgw_metadata.cc
@cbodley
rgw: add MetaMasterTrimCR to query sync status from peers
919f50b 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8bb1067)
@cbodley
rgw: master calculates minimum sync status of peers
cf261ac 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 00d7b7b)
@cbodley
rgw: add PurgePeriodLogsCR to purge entire mdlog periods
a724ceb 
Conflicts: rgw_raw_obj
        src/rgw/rgw_sync.cc

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 32d16a9)
@cbodley
rgw: master purges period mdlogs once all peers are done
119a3b2 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit eb140e3)
@cbodley
rgw: add MetaMasterTrimShardCollectCR
0d837a3 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1765f86)
@cbodley
rgw: master trims mdlogs as peers make progress on current period
90b9149 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1afabc1)
@cbodley
rgw: add MetaPeerTrimCR to query master mdlog info
5caf51e 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 6e9296d)
@cbodley
rgw: peer purges mdlog periods before master's oldest
dd458fa 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 6cde60e)
@cbodley
rgw: add MetaPeerTrimShardCR to trim mdlog shards
0f8156a 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 3376c4b)
@cbodley
rgw: peer trims mdlog shards up to master's oldest entry
e4ffe3a 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 0557a79)
@cbodley
rgw: add MetaTrimPollCR to coordinate polling and leases
8fc8004 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 8431f17)
@cbodley
rgw: RGWSyncLogTrimThread runs mdlog trim
befe59a 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit aaab1ec)
@cbodley
radosgw-admin: add 'mdlog autotrim' command
6dbaecf 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 1646264)
@cbodley
rgw: use RGWShardCollectCR in MetaMasterTrimCR
1dc97ae 
limit the number of concurrent sync status requests to peers

Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit 667ec9f)
@cbodley
rgw: add TODOs to split trim logic into separate source files
f374444 
Signed-off-by: Casey Bodley <cbodley@redhat.com>
(cherry picked from commit ec0a655)
@smithfarm
Copy link
Contributor

Integration testing: http://tracker.ceph.com/issues/21742#note-11

Will post a summary to the PR when the testing is complete.

@smithfarm
Copy link
Contributor

@yehudasa @oritwas This PR passed two RGW runs at http://tracker.ceph.com/issues/21742#note-11

Do you think it's ready to merge?

@smithfarm
Copy link
Contributor

@yehudasa @mattbenjamin @oritwas Pinging for review. Is this something that needs to go into 10.2.11 ?

@mattbenjamin
Copy link
Contributor

@smithfarm yes please

@smithfarm smithfarm merged commit 35d8924 into ceph:jewel Feb 23, 2018
@cbodley cbodley deleted the wip-jewel-rgw-log-trim branch February 23, 2018 16:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yehudasa yehudasa yehudasa approved these changes

@oritwas oritwas Awaiting requested review from oritwas

Assignees
No one assigned
Labels
feature jewel-batch-1 rgw
Projects
None yet
Milestone
jewel
4 participants
@cbodley @smithfarm @mattbenjamin @yehudasa