fix retry logic to avoid stuck in loop #16
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
|
Any chance this may get reviewed and merged? |
|
What is the reason for delay on this? It is a really awful bug. The loop counter for retries is being reset inside the loop creating an infinite loop which completely runs away with memory and CPU. This seems to happen on any logs that get rotated like istio logs for instance. I feel like "silly" is a pretty big understatement. |
|
@MasslessParticle is there a chance that you or a colleague of yours can have a look at this pr? Looks like a low hanging fruit that, at least in my case, causes major issues with the loki stack. |
|
Thanks for the contribution! |
|
@MasslessParticle Will it be propagated to grafana-agent? I'm facing this issue there. |
7 tasks
MichelHollands
pushed a commit
to grafana/loki
that referenced
this pull request
Jul 12, 2023
**What this PR does / why we need it**: This bumps the grafana/tail version used, mostly to address an issue with the retry logic (grafana/tail#16) **Which issue(s) this PR fixes**: Fixes #9864 #9877 #9869, maybe others **Special notes for your reviewer**: I did NOT add a note about this dependency bump in the changelog. Let me know if you want me to add that ... or kindly add that line yourself. **Checklist** - [x] Reviewed the [`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md) guide (**required**) - [ ] Documentation added - [ ] Tests updated - [ ] `CHANGELOG.md` updated - [ ] If the change is worth mentioning in the release notes, add `add-to-release-notes` label - [ ] Changes that require user attention or interaction to upgrade are documented in `docs/sources/upgrading/_index.md` - [ ] For Helm chart changes bump the Helm chart version in `production/helm/loki/Chart.yaml` and update `production/helm/loki/CHANGELOG.md` and `production/helm/loki/README.md`. [Example PR](d10549e)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix a silly mistake in tail.go
The retry countdown do help keep promtail from stuck in loop, causing promtail stop read logs.
Related issue: grafana/loki#3985