-
Notifications
You must be signed in to change notification settings - Fork 5.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
mon,osd,osdc: refactor snap trimming (phase 1) #18276
Commits on Dec 2, 2017
-
osd/osd_types: note about removed_snaps hack
We add in the new snap_seq just to try to keep the interval_set contiguous. Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for c536d4c - Browse repository at this point
Copy the full SHA c536d4cView commit details -
osd/PG: share_pg_info shares past_itnervals, not PastIntervals()
In reality we only call this when the PG is peered and thus past_intervals is empty, but this is more defensive in case that changes someday! Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for c8bfe3f - Browse repository at this point
Copy the full SHA c8bfe3fView commit details -
osd/OSDMap: improve osdmap flag dumping in json
Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 81d63f2 - Browse repository at this point
Copy the full SHA 81d63f2View commit details -
qa/suites/rados/singleton/all/thrash-eio: more whitelist
Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for df7523b - Browse repository at this point
Copy the full SHA df7523bView commit details -
include/interval_set: add get_end() to iterator
Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for ea308ad - Browse repository at this point
Copy the full SHA ea308adView commit details -
include/mempool: add flat_set alias
Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 3119cf5 - Browse repository at this point
Copy the full SHA 3119cf5View commit details -
include/types: flat_set operator<<
Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 1b1eec2 - Browse repository at this point
Copy the full SHA 1b1eec2View commit details -
osd/osd_types: SnapSet: remove get_first_snap_after()
No users. Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for b9c5a24 - Browse repository at this point
Copy the full SHA b9c5a24View commit details -
pg_pool_t's snap_seq is an upper bound on any allocated (or removed) snapids. Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for e89649d - Browse repository at this point
Copy the full SHA e89649dView commit details -
mon/OSDMonitor: reset OSDMap state before decode
This ensures we don't have any cruft left over in fields that decode() assumes are initialized from the ctor (and not a previous instance). Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 1f133a2 - Browse repository at this point
Copy the full SHA 1f133a2View commit details -
mon/OSDMonitor: clear pending_metadata* in create_pending
Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 37c4aff - Browse repository at this point
Copy the full SHA 37c4affView commit details -
osd/OSDMap: track newly removed and purged snaps in each epoch
Instead of maintaining a set of snapids that have been removed over all time, instead note just the newly removed and newly purged snaps in each OSDMap epoch. This is easier to consume for both the Objecter and OSD. Also keep the interval of snaps that have been removed but not perged in each OSDMap. This is extremely convenient because it frees the OSDs from having to maintain this information in parallel even when they may not have PGs belonging to those pools. These structures will be large right when the ugprade happens and the pg_pool_t::removed_snaps gets copied to the new fields, but in the steady state it will be relatively small, reflecting only the set of snaps that are currently being removed. This also provides convenient visibility into the "trimming snaps" set that the cluster is working on. Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 553048f - Browse repository at this point
Copy the full SHA 553048fView commit details -
mon/OSDMonitor: record removed_snaps by epoch outside of the osdmap
Index by snap and by epoch; separate out pools. Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 9d606c5 - Browse repository at this point
Copy the full SHA 9d606c5View commit details -
mon/OSDMonitor: share snaps removed during a map gap
If a client requests a map older than the mon's oldest, share with them snaps deleted during the gap too. Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 49833c3 - Browse repository at this point
Copy the full SHA 49833c3View commit details -
mon/MgrStatMonitor: dump PGMapDigest at debug level 20
Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 38e96ec - Browse repository at this point
Copy the full SHA 38e96ecView commit details -
osdc/Objecter: prune new_removed_snaps from active op snapc's
If an in-flight Op has a snapc referencing a deleted snap, remove it from the snapc. Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 32d7538 - Browse repository at this point
Copy the full SHA 32d7538View commit details -
osdc/Objecter: rename _scan_requests force_resend -> skipped_map
This is what the caller is passing. Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for b1b8fc6 - Browse repository at this point
Copy the full SHA b1b8fc6View commit details -
osdc/Objecter: apply removed_snaps from gap to in-flight requests
If we are so laggy that we aren't contiguous with the mon's latest map, the mon will provide a summary of removed_snaps for the gap. Apply those to our in-flight ops. Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 192a8dc - Browse repository at this point
Copy the full SHA 192a8dcView commit details -
osd,mon: add 'nosnaptrim' osd flag
Now ceph osd {set,unset} nosnaptrim will suspend or resume snap trimming. Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for a53ba73 - Browse repository at this point
Copy the full SHA a53ba73View commit details -
osd/osd_types: add purged_snaps to pg_stat_t
Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 345d3b6 - Browse repository at this point
Copy the full SHA 345d3b6View commit details -
osd/PG: share purged_snaps with mgr at mimic
Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 6df912b - Browse repository at this point
Copy the full SHA 6df912bView commit details -
mon/PGMap: add purged_snaps map to PGMapDigest
Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 86f0b81 - Browse repository at this point
Copy the full SHA 86f0b81View commit details -
osd/PG: move debug_verify_cached_snaps check into PGPool::update
Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for e5f62fb - Browse repository at this point
Copy the full SHA e5f62fbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 33c9907 - Browse repository at this point
Copy the full SHA 33c9907View commit details -
osd/PG: break out of Active AdvMap handler if interval change
If we are about to lose our primary status, we don't want to do *any* of this stuff... especially share_pg_info(), which would get tagged with the current epoch but confuse our peers! Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for f04729c - Browse repository at this point
Copy the full SHA f04729cView commit details -
osd/PG: simplify replica purged_snaps update
This dependency on the ondisk version dates back before argonaut, and no longer makes sense. Once the snap is trimmed by the primary, and purged_snaps is updated, the replica can (must!) blindly follow suit. Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 231ec67 - Browse repository at this point
Copy the full SHA 231ec67View commit details -
osd/PG: use new mimic osdmap structures for removed, pruned snaps
- update snap_trimq and purged_snaps based on new mimic OSDMap fields - improve debug output to include both trimq and purged Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 6e1b7c4 - Browse repository at this point
Copy the full SHA 6e1b7c4View commit details -
osd/osd_types: pg_pool_t: add FLAG_{SELFMANAGED,POOL}_SNAPS flags
Explicitly track whether we are a pool snaps pool or a selfmanaged snaps pool. This was inferred from removed_snaps.empty() before, but that was fragile and kludgey and removed_snaps is going away. The upgrade/compat behavior is a bit tricky: - on decode, we set the flags based on the legacy condition. This lets us use and rely on the flags in memory. - on encode, we exclude the flags if decoding an older pg_pool_t Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 16c5bcc - Browse repository at this point
Copy the full SHA 16c5bccView commit details -
mon/OSDMonitor: convert removed_snaps on first mimic map
On the first mimic map, consider previously removed_snaps to be removed in that epoch (since we don't easily know when it happened). Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for fd6a59e - Browse repository at this point
Copy the full SHA fd6a59eView commit details -
mon/OSDMonitor: prune purged snaps
Be a bit careful here because the mon has to do some bookkeeping to avoid pruning things twice. If the PGMapDigest set appears obviously stale, skip some work (looking at this particular interval) until it is not obviously stale--move onto the next interval instead. Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 9607a2d - Browse repository at this point
Copy the full SHA 9607a2dView commit details -
mon/OSDMonitor: propagate new_removed_snaps to other tiers
Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for f2d602a - Browse repository at this point
Copy the full SHA f2d602aView commit details
Commits on Dec 3, 2017
-
osd/PG: ignore purged_snaps inconsistencies for now
These are possible because we update purged_snaps, part of the pg_info_t, but we do not bump the pg version or match it with a log entry, which means that the change does not reliably propagate to new OSDs during peering etc. Signed-off-by: Sage Weil <sage@redhat.com>
Configuration menu - View commit details
-
Copy full SHA for 8c44dab - Browse repository at this point
Copy the full SHA 8c44dabView commit details