Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure orhpaned series cleaned up with shard drop #10212

Merged
merged 4 commits into from
Aug 21, 2018
Merged

Conversation

e-dard
Copy link
Contributor

@e-dard e-dard commented Aug 21, 2018

This PR fixes an issue where series could be left in the inmem index, when they should have been removed.

It could be triggered as follows:

  • insert some series into a shard that only exist in that shard;
  • wait for the retention policy enforcement to remove the shard;
  • run a SHOW query such as SHOW TAG KEYS;
  • The series that were removed with the shard will be missing (as expected);
  • re-insert the same series;
  • run a SHOW query such as SHOW TAG KEYS;
  • The series will not be present (they should bed).

This manifests itself because the series were removed from the series file, but not from the inmem index. If the server was restarted the issue would resolve itself.

Commit f52de2d ensures that the series are removed from the inmem index if they're the last occurence of those series in the index.

TODO: whilst doing this work, it's become apparent that dropping shards with the inmem index is particularly racy, when one considers concurrent writes of the same series. I will add a ticket to track that.

This commit ensures that any orphaned series (series that are to be
removed and no longer are referenced anywhere in the database) are
removed from the `inmem` index when a shard is dropped.
@jacobmarble
Copy link
Member

Thanks for breaking this up into multiple commits.

Copy link
Contributor

@stuartcarnie stuartcarnie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants