Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WAL support for Promtail #8197

Closed
7 of 11 tasks
thepalbi opened this issue Jan 18, 2023 · 1 comment
Closed
7 of 11 tasks

WAL support for Promtail #8197

thepalbi opened this issue Jan 18, 2023 · 1 comment
Assignees
@thepalbi

Comments

@thepalbi
Copy link
Contributor

thepalbi commented Jan 18, 2023 •
edited
Loading

This issue is a spin off the the following PR: #7993
Design is being discussed in the following doc.

Steps

After
@thepalbi thepalbi self-assigned this Jan 18, 2023
This was referenced Jan 24, 2023
Merged
Merged
Merged
Closed
@sandeepsukhani sandeepsukhani mentioned this issue Feb 9, 2023
Open
@thepalbi thepalbi mentioned this issue Feb 9, 2023
Merged
5 tasks
@chaudum chaudum mentioned this issue Feb 27, 2023
Closed
cstyan pushed a commit that referenced this issue Mar 29, 2023
@thepalbi
Promtail WAL support: Implement reader side (#8302)
89996c9 
**What this PR does / why we need it**:
This PR is the second in a series that will implement WAL support into
Promtail. The main objective of this PR is implement the reader side of
WAL. That is, using the previously instantiated `client.Manager` to
somehow read entries from the WAL instead of receiving them from a
channel.

**Pending work**:
- [x] Implement mechanism so that `WriteTo` targets of each
`wal.Watcher` can cleanup un-used series
- [x] Write a full fledged test-case that tests a WAL enabled Promtail
(maybe in a follow up)

**Which issue(s) this PR fixes**:
Part of #8197 

**Special notes for your reviewer**:
The line count might be big since this PR implements two fully fledged
test cases of Promtail with WAL enabled.

**Checklist**
- [ ] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [ ] Documentation added
- [ ] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/upgrading/_index.md`
@thepalbi thepalbi mentioned this issue Apr 14, 2023
Merged
5 tasks
cstyan pushed a commit that referenced this issue May 17, 2023
@thepalbi
Notification based Promtail WAL Watcher polling approach (#9135)
306c96c 
**What this PR does / why we need it**:

This PR implements a new mechanism for the wal Watcher in Promtail, to
know there are new records to be read. It uses a combination of:
- prometheus/prometheus#11950
- prometheus/prometheus#11949

The main idea is that the primary mechanism is a notification channel
between the `wal.Writer` and `wal.Watcher`. The Watcher subscribes to
write events the writer publishes, getting notified if the wal has been
written. The same subscriptions design is used for cleanup events.

As a backup, the watcher has a timer that implements an exponential
backoff strategy, which is constrained by a minimum and maximum that the
user can configure.

Below the cpu difference is shown of running both main and this branch
against the same scrape target.

<img width="2496" alt="image"
src="https://user-images.githubusercontent.com/2617411/232099483-7e5c36fa-9360-4eb9-8240-687adf46e330.png">

The yellow line is the latest main build from where this branch started,
and the green line is this branch. Both promtails tailing docker logs,
and using the following metrics to get cpu usage from cadvisor:
```
avg by (name) (rate(container_cpu_usage_seconds_total{job=~".+", instance=~".+", name=~"promtail-wal-test_promtail.+"}[$__rate_interval]))
```

**Which issue(s) this PR fixes**:
Part of #8197

**Special notes for your reviewer**:

**Checklist**
- [ ] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [ ] Documentation added
- [ ] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/upgrading/_index.md`
@thepalbi thepalbi mentioned this issue Jul 11, 2023
Merged
7 tasks
cstyan pushed a commit that referenced this issue Jul 21, 2023
@thepalbi
Refactor Promtail's client.Manager to do multi client instantiation (…
5f04eed 
…#9908)

**What this PR does / why we need it**:
This PR refactors Promtails `client.Manager` to include both WAL-enabled
client, and WAL disabled. This will make the transition easier towards
the agent codebase, only needing to instantiate one primitive to support
both WAL enabled/disabled clients under the `loki.write` component.
Also, re-uses a bunch of logic shared by both.

**Which issue(s) this PR fixes**:
Part of #8197

**Special notes for your reviewer**:

**Checklist**
- [ ] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [ ] Documentation added
- [x] Tests updated
- [ ] `CHANGELOG.md` updated
- [ ] If the change is worth mentioning in the release notes, add
`add-to-release-notes` label
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/upgrading/_index.md`
- [ ] For Helm chart changes bump the Helm chart version in
`production/helm/loki/Chart.yaml` and update
`production/helm/loki/CHANGELOG.md` and
`production/helm/loki/README.md`. [Example
PR](d10549e)
@thepalbi
Copy link
Contributor Author

Closing as this was implemented in https://grafana.com/docs/agent/next/flow/reference/components/loki.write/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@thepalbi thepalbi

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@thepalbi