Skip to content

Commit

Permalink
Promtail: Fix excludepath not evaluated on newly added files (#9831)
Browse files Browse the repository at this point in the history
**What this PR does / why we need it**:

In promtail, when a file matching the exclude path is created in a
directory that is being watched, this file should not be tailed by
promtail. However, previously this was the case. As described in #7115.
This change checks the filename to the excludePath and if it matches,
ignores the file.

**Which issue(s) this PR fixes**:
Fixes #7115

**Special notes for your reviewer**:
We tested manually this by running Promtail locally and creating a bunch
of files which should and should not be tailed by Promtail.

**Checklist**
- [x] Reviewed the
[`CONTRIBUTING.md`](https://github.com/grafana/loki/blob/main/CONTRIBUTING.md)
guide (**required**)
- [x] Tests updated
- [x] `CHANGELOG.md` updated
- [ ] If the change is worth mentioning in the release notes, add
`add-to-release-notes` label
- [ ] Changes that require user attention or interaction to upgrade are
documented in `docs/sources/upgrading/_index.md`
  • Loading branch information
SijmenHuizenga authored Jan 8, 2024
1 parent 20d1fa6 commit 5e50ea0
Show file tree
Hide file tree
Showing 3 changed files with 26 additions and 1 deletion.
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@
##### Fixes
* [11074](https://github.com/grafana/loki/pull/11074) **hainenber** Fix panic in lambda-promtail due to mishandling of empty DROP_LABELS env var.
* [11195](https://github.com/grafana/loki/pull/11195) **canuteson** Generate tsdb_shipper storage_config even if using_boltdb_shipper is false
* [9831](https://github.com/grafana/loki/pull/9831) **sijmenhuizenga**: Fix Promtail excludepath not evaluated on newly added files.
* [11551](https://github.com/grafana/loki/pull/11551) **dannykopping** Do not reflect label names in request metrics' "route" label.
* [11601](https://github.com/grafana/loki/pull/11601) **dannykopping** Ruler: Fixed a panic that can be caused by concurrent read-write access of tenant configs when there are a large amount of rules.
* [11606](https://github.com/grafana/loki/pull/11606) **dannykopping** Fixed regression adding newlines to HTTP error response bodies which may break client integrations.
Expand Down
14 changes: 14 additions & 0 deletions clients/pkg/promtail/targets/file/filetarget.go
Original file line number Diff line number Diff line change
Expand Up @@ -350,6 +350,20 @@ func (t *FileTarget) startTailing(ps []string) {
continue
}

if t.pathExclude != "" {
matched, err := doublestar.Match(t.pathExclude, p)
if err != nil {
level.Error(t.logger).Log("msg", "ignoring file, exclude pattern match failed", "error", err, "filename", p, "pathExclude", t.pathExclude)
t.metrics.totalBytes.DeleteLabelValues(p)
continue
}
if matched {
level.Info(t.logger).Log("msg", "ignoring file", "error", "file matches exclude pattern", "filename", p, "pathExclude", t.pathExclude)
t.metrics.totalBytes.DeleteLabelValues(p)
continue
}
}

var reader Reader
if t.decompressCfg != nil && t.decompressCfg.Enabled {
level.Debug(t.logger).Log("msg", "reading from compressed file", "filename", p)
Expand Down
12 changes: 11 additions & 1 deletion clients/pkg/promtail/targets/file/filetarget_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -475,6 +475,7 @@ func TestHandleFileCreationEvent(t *testing.T) {
positionsFileName := filepath.Join(dirName, "positions.yml")
logDir := filepath.Join(dirName, "log")
logFile := filepath.Join(logDir, "test1.log")
logFileIgnored := filepath.Join(logDir, "test.donot.log")

if err := os.MkdirAll(logDir, 0750); err != nil {
t.Fatal(err)
Expand Down Expand Up @@ -511,7 +512,8 @@ func TestHandleFileCreationEvent(t *testing.T) {
}
}()

target, err := NewFileTarget(metrics, logger, client, ps, path, "", nil, nil, &Config{
pathExclude := "**/*.donot.log"
target, err := NewFileTarget(metrics, logger, client, ps, path, pathExclude, nil, nil, &Config{
// To handle file creation event from channel, set enough long time as sync period
SyncPeriod: 10 * time.Minute,
}, DefaultWatchConig, fakeFileHandler, fakeTargetHandler, "", nil)
Expand All @@ -523,10 +525,18 @@ func TestHandleFileCreationEvent(t *testing.T) {
if err != nil {
t.Fatal(err)
}
_, err = os.Create(logFileIgnored)
if err != nil {
t.Fatal(err)
}
fakeFileHandler <- fsnotify.Event{
Name: logFile,
Op: fsnotify.Create,
}
fakeFileHandler <- fsnotify.Event{
Name: logFileIgnored,
Op: fsnotify.Create,
}
requireEventually(t, func() bool {
return len(target.readers) == 1
}, "Expected tails to be 1 at this point in the test...")
Expand Down

0 comments on commit 5e50ea0

Please sign in to comment.