Feature: Replace stage in pipeline #2060

adityacs · 2020-05-10T09:07:24Z

What this PR does / why we need it:
Adds Replace stage in pipeline. Supports, replacing log lines based on captured groups.

Which issue(s) this PR fixes:
Fixes #2033

Checklist

Documentation added
Tests updated

codecov-io · 2020-05-10T09:16:17Z

Codecov Report

Merging #2060 into master will decrease coverage by 0.00%.
The diff coverage is 63.77%.

@@            Coverage Diff             @@
##           master    #2060      +/-   ##
==========================================
- Coverage   61.31%   61.30%   -0.01%     
==========================================
  Files         146      147       +1     
  Lines       11189    11316     +127     
==========================================
+ Hits         6860     6937      +77     
- Misses       3784     3816      +32     
- Partials      545      563      +18

Impacted Files	Coverage Δ
pkg/logentry/stages/stage.go	`47.36% <50.00%> (+0.19%)`	⬆️
pkg/logentry/stages/replace.go	`64.22% <64.22%> (ø)`
pkg/promtail/targets/tailer.go	`76.13% <0.00%> (-2.28%)`	⬇️
pkg/logql/evaluator.go	`90.69% <0.00%> (-0.59%)`	⬇️

docs/clients/promtail/stages/regex.md

docs/clients/promtail/stages/replace.md

pkg/logentry/stages/replace.go

codecov-commenter · 2020-05-25T05:51:03Z

Codecov Report

Merging #2060 into master will decrease coverage by 0.05%.
The diff coverage is 62.26%.

@@            Coverage Diff             @@
##           master    #2060      +/-   ##
==========================================
- Coverage   61.31%   61.25%   -0.06%     
==========================================
  Files         146      147       +1     
  Lines       11189    11295     +106     
==========================================
+ Hits         6860     6919      +59     
- Misses       3784     3815      +31     
- Partials      545      561      +16

Impacted Files	Coverage Δ
pkg/logentry/stages/stage.go	`47.36% <50.00%> (+0.19%)`	⬆️
pkg/logentry/stages/replace.go	`62.74% <62.74%> (ø)`
pkg/promtail/targets/tailer.go	`73.86% <0.00%> (-4.55%)`	⬇️
pkg/promtail/targets/filetarget.go	`68.67% <0.00%> (-1.81%)`	⬇️

docs/clients/promtail/stages/regex.md

cyriltovena · 2020-05-27T15:36:08Z

pkg/logentry/stages/replace.go

+	}
+
+	// All the named captured group will be extracted
+	for i, name := range r.expression.SubexpNames() {


I'm not sure this is useful ?

A lot of you code would be much simpler.

Let's drop this for now. You can have a regex to do that.

Let's drop this for now. You can have a regex to do that

This is just for reducing using different stages in pipeline. If user wants to extract variable and then template the extracted variable he should use regex stage and then template stage. This would reduce that effort

A lot of you code would be much simpler.

Sure. Let me know where I have to improve.

Let's start with this.

cyriltovena

LGTM

pyr0hu · 2020-07-27T12:57:24Z

Is there any ETA when will this feature be released? Latest 1.5.0 seems to be released on May 20 and this was merged on Jun 1.

pyr0hu · 2020-07-27T14:18:10Z

pkg/logentry/stages/replace.go

+	}
+
+	if c.Replace == "" {
+		return nil, errors.New(ErrEmptyReplaceStageConfig)


also, I know this PR is merged, but why is this validated? If, for example, I want to remove part of the log, I cannot use replace pipeline as it won't accept an empty replace. I could use a whitespace, but then it looks funny

I think it's a fair request. You can open a PR or may be @adityacs is up for it.

I think I'll have some free time today or tomorrow to do a PR. Just wanted to ask your opinions about it :)

cyriltovena · 2020-07-28T00:24:41Z

Is there any ETA when will this feature be released? Latest 1.5.0 seems to be released on May 20 and this was merged on Jun 1.

@slim-bean probably has an idea for the ETA.

slim-bean · 2020-07-28T01:23:00Z

This week!

pull-request-size bot added the size/L label May 10, 2020

adityacs force-pushed the 2033_replace_regex branch 2 times, most recently from 1b55961 to d676c3d Compare May 10, 2020 09:11

cyriltovena reviewed May 11, 2020

View reviewed changes

docs/clients/promtail/stages/regex.md Outdated Show resolved Hide resolved

adityacs force-pushed the 2033_replace_regex branch from d676c3d to 8774c11 Compare May 12, 2020 14:58

adityacs marked this pull request as draft May 12, 2020 14:58

pull-request-size bot added size/XL and removed size/L labels May 14, 2020

adityacs force-pushed the 2033_replace_regex branch 4 times, most recently from bc11d7b to f37f68b Compare May 14, 2020 20:53

adityacs marked this pull request as ready for review May 14, 2020 20:58

adityacs requested a review from cyriltovena May 14, 2020 20:58

cyriltovena reviewed May 19, 2020

View reviewed changes

docs/clients/promtail/stages/replace.md Show resolved Hide resolved

cyriltovena reviewed May 19, 2020

View reviewed changes

docs/clients/promtail/stages/replace.md Show resolved Hide resolved

cyriltovena reviewed May 19, 2020

View reviewed changes

docs/clients/promtail/stages/replace.md Outdated Show resolved Hide resolved

cyriltovena reviewed May 19, 2020

View reviewed changes

pkg/logentry/stages/replace.go Outdated Show resolved Hide resolved

adityacs force-pushed the 2033_replace_regex branch from f37f68b to 4c3f806 Compare May 25, 2020 03:26

adityacs changed the title ~~Feature: Replace substage for regex~~ Feature: Replace stage in pipeline May 25, 2020

adityacs force-pushed the 2033_replace_regex branch from 4c3f806 to 08698c2 Compare May 25, 2020 05:39

adityacs requested a review from cyriltovena May 25, 2020 05:43

adityacs force-pushed the 2033_replace_regex branch 3 times, most recently from 7ee362e to 8678482 Compare May 26, 2020 17:28

cyriltovena reviewed May 26, 2020

View reviewed changes

docs/clients/promtail/stages/regex.md Outdated Show resolved Hide resolved

adityacs force-pushed the 2033_replace_regex branch 2 times, most recently from afc3fcf to c342759 Compare May 27, 2020 05:32

Feature: Replace stage in pipeline

4eee2b1

adityacs force-pushed the 2033_replace_regex branch from c342759 to 4eee2b1 Compare May 27, 2020 05:34

cyriltovena reviewed May 27, 2020

View reviewed changes

cyriltovena approved these changes Jun 1, 2020

View reviewed changes

cyriltovena merged commit 38ee57c into grafana:master Jun 1, 2020

pyr0hu reviewed Jul 27, 2020

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: Replace stage in pipeline #2060

Feature: Replace stage in pipeline #2060

adityacs commented May 10, 2020 •

edited

Loading

codecov-io commented May 10, 2020 •

edited

Loading

codecov-commenter commented May 25, 2020 •

edited

Loading

cyriltovena May 27, 2020

cyriltovena May 27, 2020

adityacs May 27, 2020

cyriltovena Jun 1, 2020

cyriltovena left a comment

pyr0hu commented Jul 27, 2020

pyr0hu Jul 27, 2020

cyriltovena Jul 28, 2020

adityacs Jul 28, 2020

pyr0hu Jul 28, 2020

cyriltovena commented Jul 28, 2020

slim-bean commented Jul 28, 2020

Feature: Replace stage in pipeline #2060

Feature: Replace stage in pipeline #2060

Conversation

adityacs commented May 10, 2020 • edited Loading

codecov-io commented May 10, 2020 • edited Loading

Codecov Report

codecov-commenter commented May 25, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cyriltovena left a comment

Choose a reason for hiding this comment

pyr0hu commented Jul 27, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cyriltovena commented Jul 28, 2020

slim-bean commented Jul 28, 2020

adityacs commented May 10, 2020 •

edited

Loading

codecov-io commented May 10, 2020 •

edited

Loading

codecov-commenter commented May 25, 2020 •

edited

Loading