Add store gateway consistency check errors to errors catalog #2150

zenador · 2022-06-20T14:51:20Z

What this PR does

Following what we've done in #2066, this PR adds common store gateway consistency check errors to errors catalogue.

Which issue(s) this PR fixes or relates to

N/A

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

docs/sources/operators-guide/mimir-runbooks/_index.md

osg-grafana · 2022-06-20T16:43:38Z

docs/sources/operators-guide/mimir-runbooks/_index.md

+- At query time the querier and ruler determine how old a bucket index is based on the time it was last updated by the compactor.
+- If the age is older than the maximum stale period configured via `-blocks-storage.bucket-store.bucket-index.max-stale-period`, the query fails.
+- This circuit breaker ensures queriers and rulers do not return any partial query results due to a stale view over the long-term storage.


Suggested change

- At query time the querier and ruler determine how old a bucket index is based on the time it was last updated by the compactor.

- If the age is older than the maximum stale period configured via `-blocks-storage.bucket-store.bucket-index.max-stale-period`, the query fails.

- This circuit breaker ensures queriers and rulers do not return any partial query results due to a stale view over the long-term storage.

- At query time, the querier and the ruler determine how old a bucket index is based on the time that it was last updated by the compactor.

- If the age is older than the maximum stale period that is configured via `-blocks-storage.bucket-store.bucket-index.max-stale-period`, the query fails.

- This circuit breaker ensures that the queriers and rulers do not return any partial query results, due to a stale view that spans long-term storage.

Not 100% what "over" is trying to convey here. Do you mean, ", due to a stale overview of the entire long-term storage."?

Applied your suggestion except for the last part, please check that it makes sense.

osg-grafana · 2022-06-20T16:44:26Z

pkg/querier/blocks_finder_bucket_index.go

@@ -112,3 +114,7 @@ func (f *BucketIndexBlocksFinder) GetBlocks(ctx context.Context, userID string,

 	return blocks, matchingDeletionMarks, nil
 }
+
+func newBucketIndexTooOldError(updatedAt time.Time, maxStalePeriod time.Duration) error {
+	return errors.New(globalerror.BucketIndexTooOld.MessageWithLimitConfig(fmt.Sprintf("bucket index is too old. It was last updated at %s which exceeds the maximum allowed staleness period of %v", updatedAt.UTC().Format(time.RFC3339Nano), maxStalePeriod), tsdb.BucketIndexConfigPrefix+tsdb.MaxStalePeriodFlag))


Suggested change

return errors.New(globalerror.BucketIndexTooOld.MessageWithLimitConfig(fmt.Sprintf("bucket index is too old. It was last updated at %s which exceeds the maximum allowed staleness period of %v", updatedAt.UTC().Format(time.RFC3339Nano), maxStalePeriod), tsdb.BucketIndexConfigPrefix+tsdb.MaxStalePeriodFlag))

return errors.New(globalerror.BucketIndexTooOld.MessageWithLimitConfig(fmt.Sprintf("The bucket index is too old. It was last updated at %s, which exceeds the maximum allowed staleness period of %v", updatedAt.UTC().Format(time.RFC3339Nano), maxStalePeriod), tsdb.BucketIndexConfigPrefix+tsdb.MaxStalePeriodFlag))

Nit before applying this: golang errors have to start with lowercase.

If we're going to hack that rule (which may make sense since we're building end-user facing messages here, not just internal wrappable errors), we should make the hacking standard.

Applied accept for the initial capital. If we decide to change that, let's do it in a separate PR since it would affect many errors.

Thank you, @colega. Consistency wins in most cases. Thank you for the pushback. Good call, @zenador.

No separate PR needed. Let’s keep this as is per @colega’s rationale.

osg-grafana · 2022-06-20T16:45:28Z

pkg/querier/blocks_finder_bucket_index_test.go

+	}{
+		"newBucketIndexTooOldError": {
+			err: newBucketIndexTooOldError(time.Unix(1000000000, 0), time.Hour),
+			msg: `bucket index is too old. It was last updated at 2001-09-09T01:46:40Z which exceeds the maximum allowed staleness period of 1h0m0s (err-mimir-bucket-index-too-old). You can adjust the related per-tenant limit by configuring -blocks-storage.bucket-store.bucket-index.max-stale-period, or by contacting your service administrator.`,


Suggested change

msg: `bucket index is too old. It was last updated at 2001-09-09T01:46:40Z which exceeds the maximum allowed staleness period of 1h0m0s (err-mimir-bucket-index-too-old). You can adjust the related per-tenant limit by configuring -blocks-storage.bucket-store.bucket-index.max-stale-period, or by contacting your service administrator.`,

msg: "The bucket index is too old. It was last updated at 2001-09-09T01:46:40Z, which exceeds the maximum allowed staleness period of 1h0m0s (err-mimir-bucket-index-too-old). To adjust the related per-tenant limit, configure `-blocks-storage.bucket-store.bucket-index.max-stale-period`, or contact your service administrator.",

Change from single to double quotes; please make sure I did not break things.

Applied the first part. The second part of this message is part of a template, so if we decide to change that, let's do it in a separate PR since it would affect many errors.

osg-grafana · 2022-06-20T16:46:41Z

pkg/querier/blocks_store_queryable_test.go

+	}{
+		"newStoreConsistencyCheckFailedError": {
+			err: newStoreConsistencyCheckFailedError([]ulid.ULID{ulid.MustNew(1, nil)}),
+			msg: `consistency check failed because some blocks were not queried (err-mimir-store-consistency-check-failed). The blocks are: 00000000010000000000000000`,


Suggested change

msg: `consistency check failed because some blocks were not queried (err-mimir-store-consistency-check-failed). The blocks are: 00000000010000000000000000`,

msg: `The consistency check failed because some blocks were not queried (err-mimir-store-consistency-check-failed). The blocks are: 00000000010000000000000000`,

Applied except for the initial capital

pkg/storage/tsdb/config.go

osg-grafana

Unblocking with feedback; please have a look so things can be more clear.

jhesketh

The change(s) look good to me, but I agree with the other reviewer comments.

Co-authored-by: Ursula Kallio <ursula.kallio@grafana.com>

pkg/storage/tsdb/config.go

pkg/querier/blocks_store_queryable.go

pkg/querier/blocks_finder_bucket_index.go

docs/sources/operators-guide/mimir-runbooks/_index.md

Co-authored-by: Marco Pracucci <marco@pracucci.com> Co-authored-by: Ursula Kallio <ursula.kallio@grafana.com>

zenador · 2022-06-23T10:21:44Z

@osg-grafana Please check the latest changes when you're free, thank you!

pracucci

Great job, LGTM! I just left a couple of final nits.

docs/sources/operators-guide/mimir-runbooks/_index.md

osg-grafana · 2022-06-24T07:26:05Z

"Ship it!" :)

Co-authored-by: Marco Pracucci <marco@pracucci.com>

…#2150) * Add store gateway consistency check errors to errors catalogue * Apply suggestions from code review Co-authored-by: Ursula Kallio <ursula.kallio@grafana.com> * Manually apply code review changes * Apply suggestions from code review Co-authored-by: Marco Pracucci <marco@pracucci.com> Co-authored-by: Ursula Kallio <ursula.kallio@grafana.com> * Update based on changes from code review * Apply suggestions from code review Co-authored-by: Marco Pracucci <marco@pracucci.com> Co-authored-by: Ursula Kallio <ursula.kallio@grafana.com> Co-authored-by: Marco Pracucci <marco@pracucci.com>

zenador added the type/docs Improvements or additions to documentation label Jun 20, 2022

Add store gateway consistency check errors to errors catalogue

ab3b727

zenador force-pushed the zenador/err-cat-store-gateway-consis-check branch from 862457e to ab3b727 Compare June 20, 2022 16:05

zenador requested a review from pracucci June 20, 2022 16:12

osg-grafana reviewed Jun 20, 2022

View reviewed changes

docs/sources/operators-guide/mimir-runbooks/_index.md Outdated Show resolved Hide resolved

osg-grafana reviewed Jun 20, 2022

View reviewed changes

pkg/storage/tsdb/config.go Outdated Show resolved Hide resolved

osg-grafana approved these changes Jun 20, 2022

View reviewed changes

osg-grafana changed the title ~~Add store gateway consistency check errors to errors catalogue~~ Add store gateway consistency check errors to errors catalog Jun 20, 2022

jhesketh reviewed Jun 21, 2022

View reviewed changes

zenador and others added 2 commits June 22, 2022 13:30

Apply suggestions from code review

9610344

Co-authored-by: Ursula Kallio <ursula.kallio@grafana.com>

Manually apply code review changes

f01ab6b

pracucci reviewed Jun 22, 2022

View reviewed changes

zenador and others added 2 commits June 23, 2022 17:35

Apply suggestions from code review

1b164da

Co-authored-by: Marco Pracucci <marco@pracucci.com> Co-authored-by: Ursula Kallio <ursula.kallio@grafana.com>

Update based on changes from code review

7e39f62

zenador force-pushed the zenador/err-cat-store-gateway-consis-check branch from db27a98 to 7e39f62 Compare June 23, 2022 10:12

pracucci approved these changes Jun 24, 2022

View reviewed changes

docs/sources/operators-guide/mimir-runbooks/_index.md Outdated Show resolved Hide resolved

docs/sources/operators-guide/mimir-runbooks/_index.md Outdated Show resolved Hide resolved

Apply suggestions from code review

18e7203

Co-authored-by: Marco Pracucci <marco@pracucci.com>

pracucci enabled auto-merge (squash) June 24, 2022 08:52

pracucci merged commit 5a78ece into main Jun 24, 2022

pracucci deleted the zenador/err-cat-store-gateway-consis-check branch June 24, 2022 09:20

zenador mentioned this pull request Jun 24, 2022

Change the error message template for per-tenant limits #2234

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add store gateway consistency check errors to errors catalog #2150

Add store gateway consistency check errors to errors catalog #2150

zenador commented Jun 20, 2022

osg-grafana Jun 20, 2022

zenador Jun 22, 2022

osg-grafana Jun 20, 2022

colega Jun 21, 2022

zenador Jun 22, 2022

osg-grafana Jun 24, 2022

osg-grafana Jun 24, 2022

osg-grafana Jun 20, 2022 •

edited

Loading

zenador Jun 22, 2022

osg-grafana Jun 20, 2022

zenador Jun 22, 2022

osg-grafana left a comment

jhesketh left a comment

zenador commented Jun 23, 2022

pracucci left a comment

osg-grafana commented Jun 24, 2022

	return errors.New(globalerror.BucketIndexTooOld.MessageWithLimitConfig(fmt.Sprintf("bucket index is too old. It was last updated at %s which exceeds the maximum allowed staleness period of %v", updatedAt.UTC().Format(time.RFC3339Nano), maxStalePeriod), tsdb.BucketIndexConfigPrefix+tsdb.MaxStalePeriodFlag))
	return errors.New(globalerror.BucketIndexTooOld.MessageWithLimitConfig(fmt.Sprintf("The bucket index is too old. It was last updated at %s, which exceeds the maximum allowed staleness period of %v", updatedAt.UTC().Format(time.RFC3339Nano), maxStalePeriod), tsdb.BucketIndexConfigPrefix+tsdb.MaxStalePeriodFlag))

	msg: `bucket index is too old. It was last updated at 2001-09-09T01:46:40Z which exceeds the maximum allowed staleness period of 1h0m0s (err-mimir-bucket-index-too-old). You can adjust the related per-tenant limit by configuring -blocks-storage.bucket-store.bucket-index.max-stale-period, or by contacting your service administrator.`,
	msg: "The bucket index is too old. It was last updated at 2001-09-09T01:46:40Z, which exceeds the maximum allowed staleness period of 1h0m0s (err-mimir-bucket-index-too-old). To adjust the related per-tenant limit, configure `-blocks-storage.bucket-store.bucket-index.max-stale-period`, or contact your service administrator.",

	msg: `consistency check failed because some blocks were not queried (err-mimir-store-consistency-check-failed). The blocks are: 00000000010000000000000000`,
	msg: `The consistency check failed because some blocks were not queried (err-mimir-store-consistency-check-failed). The blocks are: 00000000010000000000000000`,

Add store gateway consistency check errors to errors catalog #2150

Add store gateway consistency check errors to errors catalog #2150

Conversation

zenador commented Jun 20, 2022

What this PR does

Which issue(s) this PR fixes or relates to

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

osg-grafana Jun 20, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

osg-grafana left a comment

Choose a reason for hiding this comment

jhesketh left a comment

Choose a reason for hiding this comment

zenador commented Jun 23, 2022

pracucci left a comment

Choose a reason for hiding this comment

osg-grafana commented Jun 24, 2022

osg-grafana Jun 20, 2022 •

edited

Loading