Adding KV tombstones to fix non-monotonic index on deletes #577

Merged
merged 30 commits into from Jan 5, 2015

Conversation

Projects
None yet
3 participants
@armon
Member

armon commented Jan 5, 2015

Fixes #195. The visible issue from the outside of Consul is that if you are watching a prefix for changes (say service/web/config) for changes, and a key is deleted from the folder then the X-Consul-Index value either doesn't change or goes backward. This means that long-polling for changes breaks during a deletion. The reason for this is that the index is calculated as the Max(ModifyTime) of all the entries. The Max() function however is only monotonic if the inputs are also monotonic. A delete breaks this by removing entries, causing the Max value to slide backward or to stagnate. Because of this, blocking queries get "stuck". This is what a user experiences.

To fix this, we need to ensure X-Consul-Index is monotonic, even with deletes. This is tricky because you need to detect the absence of data. One way to do this is to never actually delete, but only soft delete and filter the results. Unfortunately, this causes unbounded data usage. The trade off we make is to soft delete by creating a "tombstone" entry. These are kept around for a configurable amount of time (default 15 minutes) and then GC'd later. This prevents unbounded data growth. However, while the tombstones are around, we use them to calculate the X-Consul-Index value and ensure the inputs are monotonic. Because of the tradeoff with unbounded storage growth, this only papers over the issue until the GC happens, at which point the index may slide backwards again.

In practice, this fixes the issues with replication and change detection and so works well enough. The only clients that would be affected by the GC are ones that are doing blocking queries extremely infrequently (less often than the GC runs). This is an edge case, and the behavior will be no worse than it currently is for them.

armon added a commit that referenced this pull request Jan 5, 2015

Merge pull request #577 from hashicorp/f-tombstone
Adding KV tombstones to fix non-monotonic index on deletes

@armon armon merged commit e86ed70 into master Jan 5, 2015

1 check was pending

continuous-integration/travis-ci The Travis CI build is in progress
Details

@armon armon deleted the f-tombstone branch Jan 5, 2015

@ryanuber

This comment has been minimized.

Show comment
Hide comment
Member

ryanuber commented Jan 5, 2015

@mitchellh

This comment has been minimized.

Show comment
Hide comment
@mitchellh

mitchellh Jan 5, 2015

Member

Straight outta India (literally).

Member

mitchellh commented Jan 5, 2015

Straight outta India (literally).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment