New component: receiver/promtailreceiver

[fbuetler]

The purpose and use-cases of the new component

The Promtail receiver implements the Loki push api as specified here. It allows Promtail instances to specify the open telemetry collector as their lokiAddress.

The implementation may be done analogously to fluentforwardreceiver

Example configuration for the component

The configuration of the receiver may be adapted from the loki_push_api configuration

Example

receivers:
  promtail:
    endpoint: 0.0.0.0:8006
    # Label map to add to every log line sent to the push API
    labels:
      [ <labelname>: <labelvalue> ... ]    

Telemetry data types supported

Logs only.

Sponsor (Optional)

[djaglowski]

I'll sponsor this.

[mar4uk]

Hi! I also has started working on promtailreceiver component. I was thinking to implement this component using opentelemetry-log-collection repo. The config would look like:

receivers:
  promtail:
    config:
      positions:
        filename: /tmp/positions.yaml
        sync_period: 10s
      scrape_configs:
        - job_name: system
          static_configs:
            - labels:
                job: varlogs
                __path__: /var/log/*.log
      target_config:
        sync_period: 10s

And the new input operator would be added to opentelemetry-log-collection repo. Here what I have done so far mar4uk/opentelemetry-log-collection#1 (draft). And in openelemetry-contrib repo it would look like filelogreceiver (I also have a draft)

[djaglowski]

It looks as though we have designs for both push and pull implementations. Both seem like valid use cases. Any thoughts on whether these should be the same receiver or separate?

[mar4uk]

I think it should be the same receiver, we can just add loki_push_api section to config:

receivers:
  promtail:
    config:
      scrape_configs:
        loki_push_api:
          server:
            http_listen_port: 3500
            grpc_listen_port: 3600
          labels:
            pushserver: push1 

took this example from Example Push Config

[djaglowski]

@mar4uk, I'm still learning promtail, but I see now that your proposal is a generalized integration of promtail, whereas the push use case is just a specific case. Is that right? One receiver makes sense to me then.

A couple other questions:

Am I understanding correctly that promtail can be run in process and does not require an additional executable?

What do you think about removing the config level in the receiver config? It seems redundant to me:

receivers:
  promtail:
    scrape_configs:
      ...

What are your thoughts on all of this @fbuetler?

[Mario-Hofstaetter]

Would lokireceiver be a better name instead of promtailreceiver ? The push api is a loki specification, promtail is just one client out of many.
Afterall there is already a lokiexporter, not a promtailexporter.

Features of promtail other than pushapi (tailing files, windows eventlog, ...) are handled by (or will be handled by?) filelogreceiver and other receivers.

Also, the link in OP points to GET /loki/api/v1/tail, it should be POST /loki/api/v1/push imho?

[mar4uk]

What do you think about removing the config level in the receiver config? It seems redundant to me:

My idea was to keep under config level exact promtail config format, so user can paste promtail config as it is under config level. On the same level as config could be added additional settings, like postions_directory or something elsse

Am I understanding correctly that promtail can be run in process and does not require an additional executable?

Yes, that's correct

Features of promtail other than pushapi (tailing files, windows eventlog, ...) are handled by (or will be handled by?) filelogreceiver and other receivers.

Agree with this point. Actually promtail has features that are already implemented by filelog, journard, syslog, kafka receivers.. Maybe adding promtail as a separate receiver is not the best way to go? The more clean way is to add new receivers to handle features that are currently not supported by existing receivers and supported by promtail like cloudflare receiver, gelf, windows_events (this one already has and issue created)

I'm not sure if the implementation of loki push api in the receiver is necessary as well. Why don't just use otlp receiver and loki exporter to push logs to loki instead of adding new promtail receiver?

[Mario-Hofstaetter]

I'm not sure if the implementation of loki push api in the receiver is necessary as well. Why don't just use otlp receiver and loki exporter to push logs to loki instead of adding new promtail receiver?

Currently we are using promtail loki_push_api as target for our .NET apps using serilog-sinks-grafana-loki, which is working very nice. (Loki devs said it is discouraged to directly push to loki because promtail does buffering & retry)

OpenTelemetry Logs are not yet stable (?), but I am not entirely sure about the state of Logging Export in OLTP in otel-dotnet. I have not found a Serilog OLTP Sink yet, if that even makes sense.

*While the OpenTelemetryLoggerProvider (i.e integration with ILogger) is stable, the OTLP Exporter for Logs is still non-stable.

So an OpenTelemetry Collector lokireceiver would could be another piece to further help use transition to only using otelcol for everything.

[mar4uk]

@Mario-Hofstaetter

Currently we are using promtail loki_push_api as target for our .NET apps using serilog-sinks-grafana-loki,

I was thinking that you can use collector as a target instead of promtail loki_push_api.
For example, let's imagine you have OpenTelemetry Collector only, and don't have promtail.
Collector has otlp receiver it can receive requests (http or grpc) on configured port. So instead of sending logs to loki_push_api target you can send logs to collector via otlp.

(There is also thread in CNCF otel-collector channel about promtail receiver https://cloud-native.slack.com/archives/C01N6P7KR6W/p1649822054450679 about possible promtail receiver usage).

So an OpenTelemetry Collector lokireceiver would could be another piece to further help use transition to only using otelcol for everything.

Did I understand the use case correct? You would like to have lokireceiver with config like

receivers:
  lokireceiver:
    endpoint: 0.0.0.0:8006
    labels:  [ <labelname>: <labelvalue> ... ]

from your app you would send logs to that endpoint (which one would you use POST /loki/api/v1/push or POST /promtail/api/v1/raw?)
Collector will have loki receiver and loki exporter enabled in pipeline. Is this correct?

My suggestion is to use otlp receiver instead of lokireceiver for that case:

receivers:
  otlp:
    protocols:
      http:
        endpoint: "0.0.0.0:8006"

from your app you would send logs to that endpoint. Collector will have otlp receiver and loki exporter enabled in pipeline.
The difference with previous example is the format of the body of the POST request.

The reason to use lokireceiver (promtailreceiver) instead of otlp receiver is to avoid changing the part of application responsible for sending logs. So you will be able to switch from promtail to otelcollector right away. Is this what you are looking for from lokireceiver (promtailreceiver)?

[Mario-Hofstaetter]

@mar4uk

Collector has otlp receiver it can receive requests (http or grpc) on configured port. So instead of sending logs to loki_push_api target you can send logs to collector via otlp.

Correct, but until recently logs was considered "alpha" in OTEL. Also I am only seeing the .NET side of things with otel-dotnet , where everything is a bit weird. Logging uses the existing ILogger interface of .NET.
On top of that we use serilog, so not sure if we can just turn on exporting logs to otelcol. Whether interoperability between serilog and otel (via ILogger) is possible is not yet clear to me.

Here it still says:

This package currently only supports exporting traces and metrics. Once the OTLP log data model is deemed stable, the OTLP log exporter will be folded into this package.

Which is not encouraging to use OLTP for logs. To quote from: open-telemetry/opentelemetry-dotnet#1955 (comment)

Once the Open Telemetry Logging is fully baked, then I think there will be no need to use Serilog anymore. Instead, you'd simply configure an Open Telemetry exporter for your Traces, Metrics and Logs (You have to do each one separately in Open Telemetry .NET).

That would be ideal, using only otelcol for everything, but we would still have to keep serilog for local file logging (as backup, if otel-collector goes down, we'd have no logs anymore which is not acceptable).

I will have to spend some time trying out logging in .NET with OLTP to OTELcol and subsequently pushing to loki using the lokiexporter to see if that is stable enough.
We could also drop the "pushing logs" completely for now since we log to JSON files anywe, and use the filelogreceiver to tail the files, but that component is also still in ALPHA.

EDIT: The OTLP Logs Exporter for OpenTelemetry .NET is still pre-release on nuget, so not usable for serious work from the looks of it.

Since we also need to tail logs of third party apps (without otel support) we will also have to rebuild our log pipelines (currently promtail) to use Filelog Receiver, but not sure if it already has all the necessary features (timestamp parsing, regex, json)
There is lots of stuff, but I don't have the impression this is production ready.

from your app you would send logs to that endpoint

Correct.

(which one would you use POST /loki/api/v1/push or POST /promtail/api/v1/raw?)

The package we currently use uses /loki/api/v1/push

Collector will have loki receiver and loki exporter enabled in pipeline. Is this correct?

Correct.

The reason to use lokireceiver (promtailreceiver) instead of otlp receiver is to avoid changing the part of application responsible for sending logs. So you will be able to switch from promtail to otelcollector right away.

Correct again. Other users who have been using promtail and push api longer than us may be after this too.

We only recently rolled out pushing to loki, before that we only used promtail to tail log files of legacy apps.
So we certainly can switch to OLTP logs in the midterm, depending on stability and availabilty of features in otel-dotnet and otelcol.

[mar4uk]

Created draft PR just to not lose what I have developed so far #10109. Also it could be the place for further discussions

[mar4uk]

I will be on a long leave for a few months, and will not be able to continue working on this component. Feel free to use my draft PR if you continue working on this component

[djaglowski]

Thank you for the discussion and draft @mar4uk.

I'd like to see this component move forward in some form, but given the rather different approaches that appear to be on the table, I think it's worth discussing in the Collector SIG next week. I'll add it to the agenda.

[Mario-Hofstaetter]

@djaglowski I was thinking about this and other components (like lokiexporter).
Could it be possible to get the grafana developers on board to maintain these? The best scenario for users would be if otelcol supports all features of promtail and replaces / superseeds it completely. Since promtail is free to use otecol does not hurt their profits, rather otelcol could drive more users to using their products / cloud services.

There is also grafana-agent which similarily to otelcol collects logs, metrics and traces, but is tailored to be used with grafana cloud.
But I guess the AGPL-3.0 license makes source sharing impossible?

I don't know how much grafana is already involved in OpenTelemetry (probably a lot) so please forgive my ignorence.

[djaglowski]

@Mario-Hofstaetter, I'm glad you pointed out the AGPL license. Unfortunately, we cannot accept any dependency on code licensed under AGPL. In theory, we can consider an implementation that does not depend on copyleft-licensed source. However, given the need for clarity on this issue, I suggest that in the case such an implementation is possible, a fresh proposal should be drafted.

[Blackclaws]

@Mario-Hofstaetter, I'm glad you pointed out the AGPL license. Unfortunately, we cannot accept any dependency on code licensed under AGPL. In theory, we can consider an implementation that does not depend on copyleft-licensed source. However, given the need for clarity on this issue, I suggest that in the case such an implementation is possible, a fresh proposal should be drafted.

Just an FYI not everything about Loki is AGPL. The logprotocol is Apache 2: https://github.com/grafana/loki/tree/main/pkg/logproto

So a general: "Close this because AGPL" is a bit weird.

I think having such a receiver component would indeed be nice to have for the reasons stated above. You'd have to re-implement the receiver services and can't just copy paste them from Loki for the license reasons but aside from that it shouldn't be a license issue.

@jpkrohling Is there any interest from the Grafana side of things for a promtail receiver component in the opentelemetry-collector?

[jpkrohling]

I'll ask around and provide feedback here. If I don't get back in a couple of days, feel free to ping me again.

[jpkrohling]

I'm reopening, as:

1. @mar4uk is from Grafana Labs, she's just on a temporary leave
2. I'm not sure which AGPL code is being referenced here. From what I understand, all we would depend on here is Apache License v2

[djaglowski]

I may have misunderstood, but when I looked into this I concluded that the proposal was dependent on AGPL licensed code.

Taking another look at @mar4uk's draft PR, I still reach the same conclusion, as it depends on packages from github.com/grafana/loki which appears to be licensed under AGPL-3.0.

Apologies if I'm misinterpreting this somehow. I'd like to understand what I'm missing if that's the case.

[jpkrohling]

It's not very intuitive, but the two packages that PR depends on are licensed under Apache Licence, despite the repository being licensed under AGPL-3.

https://github.com/grafana/loki/blob/main/pkg/logproto/LICENSE_APACHE2
https://github.com/grafana/loki/blob/main/clients/LICENSE_APACHE2

[djaglowski]

Got it. I missed the carve outs. I see they are listed here as well.

[jpkrohling]

@djaglowski, apparently, I forgot to set this to "accepted" component. Could you please confirm your understanding is also that this component has been accepted?

[djaglowski]

@jpkrohling, I am still will to sponsor this so we can call it accepted.

[gfonseca-tc]

Hey @jpkrohling and @djaglowski , just found this issue here. What is the status for this? I would much appreciate that feature in the collector! Is there still a license problem?

[jpkrohling]

No, @mar4uk is coming back from her license, and we should have news soon.

[mar4uk]

New PR #14632 was created. The license is not a blocker anymore

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.