Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ingester: use consistent set of instances to avoid panic #4904

Merged
merged 1 commit into from
Dec 9, 2021

Conversation

bboreham
Copy link
Contributor

@bboreham bboreham commented Dec 9, 2021

Calling getInstances() twice runs the risk that the set has expanded or contracted between allocating the slice and starting to iterate.

Stack trace where it went wrong:

panic: runtime error: index out of range [1743] with length 1743
goroutine 514 [running]:
github.com/grafana/loki/pkg/ingester.newStreamsIterator({0x27cc280, 0xc000a73600})
	/src/enterprise-logs/vendor/github.com/grafana/loki/pkg/ingester/checkpoint.go:218 +0x2ba
github.com/grafana/loki/pkg/ingester.(*ingesterSeriesIter).Iter(0x17c33cf)
	/src/enterprise-logs/vendor/github.com/grafana/loki/pkg/ingester/checkpoint.go:189 +0x25
github.com/grafana/loki/pkg/ingester.(*Checkpointer).PerformCheckpoint(0xc127edff78)
	/src/enterprise-logs/vendor/github.com/grafana/loki/pkg/ingester/checkpoint.go:579 +0x25f
github.com/grafana/loki/pkg/ingester.(*Checkpointer).Run(0xc127edff78)
	/src/enterprise-logs/vendor/github.com/grafana/loki/pkg/ingester/checkpoint.go:618 +0x21c
github.com/grafana/loki/pkg/ingester.(*walWrapper).run(0xc0009a3810)
	/src/enterprise-logs/vendor/github.com/grafana/loki/pkg/ingester/wal.go:166 +0x21c
created by github.com/grafana/loki/pkg/ingester.(*walWrapper).Start
	/src/enterprise-logs/vendor/github.com/grafana/loki/pkg/ingester/wal.go:102 +0x6f

Checklist

  • NA Documentation added
  • Tests updated
  • Add an entry in the CHANGELOG.md about the changes.

(I didn't add a test as it would be complex to synthesise delays in fetching the set to recreate the race.)

Calling `getInstances()` twice runs the risk that the set has expanded
or contracted between allocating the slice and starting to iterate.
@bboreham bboreham requested a review from a team as a code owner December 9, 2021 18:32
Copy link
Contributor

@cyriltovena cyriltovena left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm

@cyriltovena cyriltovena merged commit b1ae906 into main Dec 9, 2021
@cyriltovena cyriltovena deleted the fix-streams-race branch December 9, 2021 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants