Allow objects to be reaped sooner than leeway interval. #470
Conversation
ghost
assigned 
cleaned up prior to the expiration of the leeway interval. Fixes #470 * Change the keys written to the GC bucket to be representative of the current time instead of a time in the future * Move enforcement of the leeway_interval to the GC daemon * Provide a way to override the default leeway_interval from the riak-cs-gc tool
cleaned up prior to the expiration of the leeway interval. Fixes #470 * Change the keys written to the GC bucket to be representative of the current time instead of a time in the future * Move enforcement of the leeway_interval to the GC daemon * Provide a way to override the default leeway_interval from the riak-cs-gc tool
|
I plan to squash all 4 commits and any that result from code review, but I have left them for now to show the breakdown of the work. |
| @@ -155,7 +155,7 @@ idle(_S) -> | |||
| {history, {call, ?GCD_MODULE, resume, []}}, | |||
| {history, {call, ?GCD_MODULE, set_interval, [infinity]}}, | |||
| {{paused, idle}, {call, ?GCD_MODULE, pause, []}}, | |||
| {fetching_next_batch, {call, ?GCD_MODULE, manual_batch, [[testing]]}} | |||
| {fetching_next_fileset, {call, ?GCD_MODULE, manual_batch, [[testing]]}} | |||
There was a problem hiding this comment.
The state transitions have changed a bit since this work began due to the addition of optional 2I pagination for GC.
|
This branch has been freshly rebased from There is still the issue of confusing and inconsistent terminology (e.g. batch, fileset), but I am deferring that to the forthcoming PR to add concurrency to the GC process. |
|
One thing I've been working through in my head is what happens as you upgrade to this change. I wanted to make sure that as you upgrade, no garbage will be collected early. As I've worked it out, it is safe. In fact, one 'leeway' of garbage will be collected late (2x leeway). Here's how I came up with this: Assume 24-hour leeway. Before upgrading, some garbage is created on day 0, and written to key 1 (lets assume keys with day resolution). Now, upgrade Riak CS. Keys written to day 1 won't be collected until day 2. Which means there will be a 48-hour leeway, but just for those keys written before the upgrade. Keys written after the upgrade will be written to |
Yes it should. Fixed. |
Yes, I see your point, but I think I'd rather do that as a specific refactor of that script instead of throwing it in here. |
This does seem correct. A possible workaround is to temporarily set a negative leeway time if that extra interval poses a real problem, but I suspect in most cases it will not be an issue. |
|
Great work: 👍 |
cleaned up prior to the expiration of the leeway interval. Fixes #470 * Change the keys written to the GC bucket to be representative of the current time instead of a time in the future * Move enforcement of the leeway_interval to the GC daemon * Provide a way to override the default leeway_interval from the riak-cs-gc tool
Change how manifest delete marking works so that objects may be cleaned up prior to the expiration of the leeway interval. The current scheme works by writing the deleted or overwritten manifest version to a garbage collection bucket with a key that is representative of a timestamp in the future when the blocks of the object represented by the manifest are eligible to actually be deleted. The garbage collection daemon periodically checks the garbage collection bucket for keys that represent a time that is in the past and judges any such key as eligible to be reaped.
The problem with this is approach is that there is not a way to override the leeway interval in the case of a disk space shortage, bloated manifests, or situations where the system may be under stress. To improve this situation the process should be upended so that the keys written to the GC bucket are representative of the current time and not a future point in time and the enforcement of the
leeway_intervalshould be done by the GC daemon. This would allow early cleanup of garbage to be accomplished even after a manifest is written to the GC bucket.The goals of this work are as follows:
leeway_intervalto the GC daemonleeway_intervalfrom theriak-cs-gctool