Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Ingesters Occassionally Double Flushing #364

Merged
merged 14 commits into from Nov 24, 2020

Conversation

joe-elliott
Copy link
Member

@joe-elliott joe-elliott commented Nov 23, 2020

What this PR does:
Occassionally ingesters fail to flush due to a double queuing (#217) issue discovered by @calvernaz. This PR looks to fix that by:

  • Creating a shim in front of the priority queues ExclusiveQueues that prevents multiple items from being queued with the same key
  • Adjusting the flushLoop() to either call .Requeue() or .ClearKey() after flush has failed or succeeded
  • Consolidated helper methods to extract prometheus metric values to ./pkg/util/test
  • Actually wait for flush queues to clear in stopping() eep!

Which issue(s) this PR fixes:
Fixes #217

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott joe-elliott changed the title Fix Bad Flushes Fix Ingster Sometimes Failing to Flush Nov 23, 2020
@joe-elliott joe-elliott changed the title Fix Ingster Sometimes Failing to Flush Fix Ingester Sometimes Failing to Flush Nov 23, 2020
@joe-elliott joe-elliott changed the title Fix Ingester Sometimes Failing to Flush Fix Ingesters Occassionally Double Flushing Nov 23, 2020
Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Copy link
Contributor

@mdisibio mdisibio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm. A few minor comments mentioned, but more of just questions than recommendations.

Signed-off-by: Joe Elliott <number101010@gmail.com>
Signed-off-by: Joe Elliott <number101010@gmail.com>
Copy link
Contributor

@calvernaz calvernaz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I just have a question regarding the clear operation.

pkg/flushqueues/exclusivequeues.go Outdated Show resolved Hide resolved
pkg/flushqueues/exclusivequeues.go Show resolved Hide resolved
Signed-off-by: Joe Elliott <number101010@gmail.com>
@joe-elliott joe-elliott merged commit 62af44e into grafana:master Nov 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Ingesters sometimes fail to flush
4 participants