Promtail: not all pipeline stages are observed by -inspect

[dannykopping]

Describe the bug
The pipeline inspection feature hooks into the stageProcessor.Run function to observe the state before and after a stage is processed. However, if a stage is not wrapped in a stageProcessor, the inspection will not occur.

This seems to be the case for the following stages:

• docker
• cri
• multiline
• match
• pack

Even though docker and cri are collections of stages (pipelines), we should treat them as a single stage.

Expected behavior
All pipeline stages should be inspected.

See #4331 for initial report.

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

[gautamvij94]

Is there a workaround to make multiline work ?

[chennin]

This is making it impossible to debug why my match isn't working, when the stages work fine without using a match

[cstyan]

@dannykopping do you know if this is still an issue?

[dannykopping]

It is, unfortunately.
I've added the good first issue label since it won't be that difficult to fix, but it's very low down on my list of priorities right now so I'm unlikely to get to it soon.

[Kylea650]

It is, unfortunately.
I've added the good first issue label since it won't be that difficult to fix, but it's very low down on my list of priorities right now so I'm unlikely to get to it soon.

Hi, I’m happy to have a go at this if you can give me an idea of where to start looking?

[cstyan]

@Kylea650 I think Danny's opening comment is a good starting point, The pipeline inspection feature hooks into the stageProcessor.Run function to observe the state before and after a stage is processed. However, if a stage is not wrapped in a stageProcessor, the inspection will not occur. and then he mentions which stages are currently not wrapped. We either need to wrap those stages without breaking other functionality, or find another way to provide the inspection functionality such as introducing a new interface.

[Kylea650]

@Kylea650 I think Danny's opening comment is a good starting point, The pipeline inspection feature hooks into the stageProcessor.Run function to observe the state before and after a stage is processed. However, if a stage is not wrapped in a stageProcessor, the inspection will not occur. and then he mentions which stages are currently not wrapped. We either need to wrap those stages without breaking other functionality, or find another way to provide the inspection functionality such as introducing a new interface.

Started looking at this today! Could this be as easy as implementing the missing Process method for the Processor interface and calling toStage() on each of the missing pipeline stages? Or something a bit more refined? Also any tips on generating logs for testing changes? What do others usually do? Thank you!

[cstyan]

Unfortunately I can't comment on the implementation, I haven't worked in that part of the promtail code enough to have an opinion. Can take a look some more later this week maybe.

As far as generating logs for testing, if you mean manual testing as in running promtail and gathering logs with it you could try https://github.com/mingrammer/flog I know we've used that in the past for basic testing.

[Kylea650]

It is, unfortunately. I've added the good first issue label since it won't be that difficult to fix, but it's very low down on my list of priorities right now so I'm unlikely to get to it soon.

Hey @dannykopping. Perhaps I’m missing something, but not sure I understand how inspecting the json stage ‘doesn’t’ give a none result. It doesn’t seem to hook into the StageProcessor and doesn’t implement the processor interface (that I can see anyway…). Again, maybe I’m misunderstanding the code base.