[promtail] Add support for journal matches (only conjuntions) #6656

carlospeon · 2022-07-11T14:39:47Z

What this PR does / why we need it:
Do not process all journal logs. Add an option to add matches to the journal reader.

Checklist

Documentation added
Tests updated
Is this an important fix or new feature? Add an entry in the CHANGELOG.md.
Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

CLAassistant · 2022-07-11T14:39:53Z

All committers have signed the CLA.

grafanabot · 2022-07-11T15:16:33Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

DylanGuedes

hey thanks for your contribution, this is fantastic. Added a nit and a question, lmk if they are too confusing.

CHANGELOG.md

clients/pkg/promtail/targets/journal/journaltarget.go

grafanabot · 2022-07-13T11:08:36Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

dannykopping

Thank you for your contribution @carlospeon 👍
Added a few nits for clarity and robustness

clients/pkg/promtail/targets/journal/journaltarget.go

docs/sources/clients/promtail/scraping.md

dannykopping · 2022-08-08T16:31:56Z

clients/pkg/promtail/scrapeconfig/scrapeconfig.go

@@ -159,6 +159,9 @@ type JournalTargetConfig struct {
 	// Path to a directory to read journal entries from. Defaults to system path
 	// if empty.
 	Path string `yaml:"path"`
+
+	// Journal matches to filter. Disjunctions (+) not supported.
+	Matches string `yaml:"matches"`


This is pluralised but it only seems to take a single entry.
Instead of using strings.Fields can we just allow a list to be specified in YAML?

I though about it before submitted this PR, but if "github.com/coreos/go-systemd/sdjournal" support logical OR (some work can be seen in their source code) it looked a bit weird to me to add a single + character as a list item.

So finally I decided to go with a string and the idea to set here the same arguments that you can use with journalctl, for example "_TRANSPORT=kernel + PRIORITY=3", but I don't have a strong opinion which two options is better.

Thanks for the detail 👍
I think we shouldn't let the implementation detail of a library we use bubble up to the end-user, so perhaps we can specify the matches as a list of conditions (I think this is the most ergonomic / idiomatic "YAML way"), and then provide them to the library in the way it requires.

I think the problem is precisely the implementation detail of the library that promtail uses. It uses and array of matches and this decision limits the way you could set AND/OR matches. Maybe that's the cause that this library doesn't implement it at all taking into account that Journal supports AND/OR matches, and the library is just a binding of journald code.

Journal support filters with logical AND and/or logical OR, so I just though how to support that expression of filters in promtail without bringing a no ideal (IMO) decision of the library to promtail.

Anyway, the change proposed is quite simple and could be easily coded although I think it should not be done.

OK, i've taken a look at the manpage to re-familiarise myself with the specifics of filtering.

Here's one example they cite:

To show all fields emitted by a unit and about the unit, option -u/--unit= should be used. journalctl -u name expands to a complex filter similar to _SYSTEMD_UNIT=name.service + UNIT=name.service _PID=1 + OBJECT_SYSTEMD_UNIT=name.service _UID=0 + COREDUMP_UNIT=name.service _UID=0 MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1

If a user were to try write a filter expression like the above in a promtail config, might it look like this?

... matches: - _SYSTEMD_UNIT=name.service - UNIT=name.service _PID=1 - OBJECT_SYSTEMD_UNIT=name.service _UID=0 - COREDUMP_UNIT=name.service _UID=0 MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1 ...

In the documentation we could specify that each matches item is a disjunction (logical OR), and the logical AND is combining multiple filters in a single entry separated by space.

It seems the library supports adding disjunctions, but I'm not familiar enough with the code to know how much effort this will be.

I understand that you've put in a lot of effort into this already, so I'm happy to merge this as-is, but in a perfect world I think we should follow the Principle of Least Surprise here to make the filtering work as a user would expect. If journal filtering allows for disjunctions, then we should support it, too.

If you don't have the time to add disjunction support, that's ok - we can always add it as an enhancement for later; I don't want to block this useful feature.

In the documentation we could specify that each matches item is a disjunction (logical OR), and the logical AND is combining multiple filters in a single entry separated by space.

Absolutely. Logical AND have preference and are evaluated first, so IMO that's the way to express the filters. Or going further in the "yaml way" a two levels nested list:

... matches: - _SYSTEMD_UNIT=name.service - - UNIT=name.service - _PID=1 - - OBJECT_SYSTEMD_UNIT=name.service - _UID=0 - - COREDUMP_UNIT=name.service - _UID=0 - MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1 ...

In the short time, in order to user current go-systemd functionality there should be only one list item:

... matches: - COREDUMP_UNIT=name.service _UID=0 MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1 ...

or

... matches: - - COREDUMP_UNIT=name.service - _UID=0 - MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1 ...

which is a bit weird and we must explain it in the docs. So we have a trade off between being ready to support future functionality and a weird (IMO) configuration or have a more simple configuration and no-compatibility with future functionality.

It seems the library supports adding disjunctions, but I'm not familiar enough with the code to know how much effort this will be.

Yes, it is ready to use disjuntions, but the reader does not uses them at all. IMO the library should mimic journalctl add_matches.

My main concern here is that proposing a modification in coreos/go-systemd reader to support disjunctions will need a different JournalReaderConfig struct, maybe replacing the match list with a two levels nested list of matches, and this will break compatibility of anyone using it, like promtail.

I will reach you in your public slack so we can talk about next steps.

@carlospeon heads up, I'm on PTO from tomorrow and back on the 22nd Aug.
I'm happy for this to be merged as-is once all the comments are addressed (except for this one)

For now, I think given the complexity of this we can stick with what we have,
Let's leave this comment unresolved so we can reference it later.
Thank you for the great discussion!

dannykopping · 2022-08-09T13:10:55Z

clients/pkg/promtail/scrapeconfig/scrapeconfig.go

@@ -159,6 +159,9 @@ type JournalTargetConfig struct {
 	// Path to a directory to read journal entries from. Defaults to system path
 	// if empty.
 	Path string `yaml:"path"`
+
+	// Journal matches to filter. Disjunctions (+) not supported.
+	Matches string `yaml:"matches"`


Thanks for the detail 👍
I think we shouldn't let the implementation detail of a library we use bubble up to the end-user, so perhaps we can specify the matches as a list of conditions (I think this is the most ergonomic / idiomatic "YAML way"), and then provide them to the library in the way it requires.

clients/pkg/promtail/targets/journal/journaltarget_test.go

docs/sources/clients/promtail/scraping.md

clients/pkg/promtail/targets/journal/journaltarget.go

dannykopping

@carlospeon feel free to engage me in our public Slack if you want to discuss this in a higher-bandwidth way.

I'm happy to proceed with this as-is, but I've shared my thoughts below.

dannykopping · 2022-08-10T08:39:21Z

clients/pkg/promtail/scrapeconfig/scrapeconfig.go

@@ -159,6 +159,9 @@ type JournalTargetConfig struct {
 	// Path to a directory to read journal entries from. Defaults to system path
 	// if empty.
 	Path string `yaml:"path"`
+
+	// Journal matches to filter. Disjunctions (+) not supported.
+	Matches string `yaml:"matches"`


OK, i've taken a look at the manpage to re-familiarise myself with the specifics of filtering.

Here's one example they cite:

To show all fields emitted by a unit and about the unit, option -u/--unit= should be used. journalctl -u name expands to a complex filter similar to _SYSTEMD_UNIT=name.service + UNIT=name.service _PID=1 + OBJECT_SYSTEMD_UNIT=name.service _UID=0 + COREDUMP_UNIT=name.service _UID=0 MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1

If a user were to try write a filter expression like the above in a promtail config, might it look like this?

... matches: - _SYSTEMD_UNIT=name.service - UNIT=name.service _PID=1 - OBJECT_SYSTEMD_UNIT=name.service _UID=0 - COREDUMP_UNIT=name.service _UID=0 MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1 ...

In the documentation we could specify that each matches item is a disjunction (logical OR), and the logical AND is combining multiple filters in a single entry separated by space.

It seems the library supports adding disjunctions, but I'm not familiar enough with the code to know how much effort this will be.

I understand that you've put in a lot of effort into this already, so I'm happy to merge this as-is, but in a perfect world I think we should follow the Principle of Least Surprise here to make the filtering work as a user would expect. If journal filtering allows for disjunctions, then we should support it, too.

If you don't have the time to add disjunction support, that's ok - we can always add it as an enhancement for later; I don't want to block this useful feature.

dannykopping

LGTM, thank you @carlospeon!
Please fix the conflicts and then we can merge this

grafanabot · 2022-08-12T08:28:42Z

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

kavirajk

LGTM 👍

carlospeon requested review from KMiller-Grafana and a team as code owners July 11, 2022 14:39

pull-request-size bot added the size/S label Jul 11, 2022

github-actions bot added the area/docs label Jul 11, 2022

carlospeon force-pushed the main branch 3 times, most recently from 6e3f1c7 to 70545ca Compare July 11, 2022 15:11

DylanGuedes reviewed Jul 11, 2022

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

clients/pkg/promtail/targets/journal/journaltarget.go Show resolved Hide resolved

carlospeon force-pushed the main branch from 70545ca to 842236c Compare July 12, 2022 12:28

carlospeon requested a review from DylanGuedes July 12, 2022 13:30

carlospeon force-pushed the main branch from 842236c to 30e999b Compare July 13, 2022 11:02

KMiller-Grafana added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Jul 18, 2022

dannykopping reviewed Aug 8, 2022

View reviewed changes

KMiller-Grafana removed the area/docs label Aug 8, 2022

pull-request-size bot added size/M and removed size/S labels Aug 9, 2022

dannykopping reviewed Aug 9, 2022

View reviewed changes

dannykopping reviewed Aug 10, 2022

View reviewed changes

dannykopping approved these changes Aug 11, 2022

View reviewed changes

dannykopping mentioned this pull request Aug 11, 2022

Add full journal filtering support #6881

Open

carlospeon force-pushed the main branch from 05d334c to 0640c67 Compare August 12, 2022 07:27

Add support for journal matches (only conjuntions)

32b82f8

carlospeon force-pushed the main branch from 0640c67 to 32b82f8 Compare August 12, 2022 08:21

kavirajk approved these changes Aug 12, 2022

View reviewed changes

kavirajk merged commit 171a09b into grafana:main Aug 12, 2022

lxwzy pushed a commit to lxwzy/loki that referenced this pull request Nov 7, 2022

Add support for journal matches (only conjuntions) (grafana#6656)

719b4b1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[promtail] Add support for journal matches (only conjuntions) #6656

[promtail] Add support for journal matches (only conjuntions) #6656

carlospeon commented Jul 11, 2022 •

edited

CLAassistant commented Jul 11, 2022 •

edited

grafanabot commented Jul 11, 2022

DylanGuedes left a comment

grafanabot commented Jul 13, 2022

dannykopping left a comment

dannykopping Aug 8, 2022 •

edited

carlospeon Aug 9, 2022 •

edited

dannykopping Aug 9, 2022

carlospeon Aug 9, 2022

dannykopping Aug 10, 2022

carlospeon Aug 10, 2022

dannykopping Aug 11, 2022

dannykopping Aug 11, 2022

dannykopping Aug 9, 2022

dannykopping left a comment

dannykopping Aug 10, 2022

dannykopping left a comment

grafanabot commented Aug 12, 2022

kavirajk left a comment

[promtail] Add support for journal matches (only conjuntions) #6656

[promtail] Add support for journal matches (only conjuntions) #6656

Conversation

carlospeon commented Jul 11, 2022 • edited

CLAassistant commented Jul 11, 2022 • edited

grafanabot commented Jul 11, 2022

DylanGuedes left a comment

Choose a reason for hiding this comment

grafanabot commented Jul 13, 2022

dannykopping left a comment

Choose a reason for hiding this comment

dannykopping Aug 8, 2022 • edited

Choose a reason for hiding this comment

carlospeon Aug 9, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dannykopping left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dannykopping left a comment

Choose a reason for hiding this comment

grafanabot commented Aug 12, 2022

kavirajk left a comment

Choose a reason for hiding this comment

carlospeon commented Jul 11, 2022 •

edited

CLAassistant commented Jul 11, 2022 •

edited

dannykopping Aug 8, 2022 •

edited

carlospeon Aug 9, 2022 •

edited