-
Notifications
You must be signed in to change notification settings - Fork 2.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New component: receiver/promtailreceiver #9800
Comments
I'll sponsor this. |
Hi! I also has started working on promtailreceiver component. I was thinking to implement this component using opentelemetry-log-collection repo. The config would look like:
And the new input operator would be added to opentelemetry-log-collection repo. Here what I have done so far mar4uk/opentelemetry-log-collection#1 (draft). And in openelemetry-contrib repo it would look like filelogreceiver (I also have a draft) |
It looks as though we have designs for both push and pull implementations. Both seem like valid use cases. Any thoughts on whether these should be the same receiver or separate? |
I think it should be the same receiver, we can just add loki_push_api section to config:
took this example from Example Push Config |
@mar4uk, I'm still learning promtail, but I see now that your proposal is a generalized integration of promtail, whereas the push use case is just a specific case. Is that right? One receiver makes sense to me then. A couple other questions: Am I understanding correctly that promtail can be run in process and does not require an additional executable? What do you think about removing the receivers:
promtail:
scrape_configs:
... What are your thoughts on all of this @fbuetler? |
Would Features of promtail other than pushapi (tailing files, windows eventlog, ...) are handled by (or will be handled by?) filelogreceiver and other receivers. Also, the link in OP points to GET /loki/api/v1/tail, it should be POST /loki/api/v1/push imho? |
My idea was to keep under config level exact promtail config format, so user can paste promtail config as it is under config level. On the same level as config could be added additional settings, like postions_directory or something elsse
Yes, that's correct
Agree with this point. Actually promtail has features that are already implemented by filelog, journard, syslog, kafka receivers.. Maybe adding promtail as a separate receiver is not the best way to go? The more clean way is to add new receivers to handle features that are currently not supported by existing receivers and supported by promtail like cloudflare receiver, gelf, windows_events (this one already has and issue created) I'm not sure if the implementation of loki push api in the receiver is necessary as well. Why don't just use otlp receiver and loki exporter to push logs to loki instead of adding new promtail receiver? |
Currently we are using promtail OpenTelemetry Logs are not yet stable (?), but I am not entirely sure about the state of Logging Export in OLTP in otel-dotnet. I have not found a Serilog OLTP Sink yet, if that even makes sense.
So an OpenTelemetry Collector |
I was thinking that you can use collector as a target instead of promtail loki_push_api. (There is also thread in CNCF otel-collector channel about promtail receiver https://cloud-native.slack.com/archives/C01N6P7KR6W/p1649822054450679 about possible promtail receiver usage).
Did I understand the use case correct? You would like to have lokireceiver with config like
from your app you would send logs to that endpoint (which one would you use POST /loki/api/v1/push or POST /promtail/api/v1/raw?) My suggestion is to use otlp receiver instead of lokireceiver for that case:
from your app you would send logs to that endpoint. Collector will have otlp receiver and loki exporter enabled in pipeline. The reason to use lokireceiver (promtailreceiver) instead of otlp receiver is to avoid changing the part of application responsible for sending logs. So you will be able to switch from promtail to otelcollector right away. Is this what you are looking for from lokireceiver (promtailreceiver)? |
Correct, but until recently
Which is not encouraging to use OLTP for logs. To quote from: open-telemetry/opentelemetry-dotnet#1955 (comment)
That would be ideal, using only otelcol for everything, but we would still have to keep serilog for local file logging (as backup, if otel-collector goes down, we'd have no logs anymore which is not acceptable). I will have to spend some time trying out logging in .NET with OLTP to OTELcol and subsequently pushing to loki using the lokiexporter to see if that is stable enough. EDIT: The OTLP Logs Exporter for OpenTelemetry .NET is still pre-release on nuget, so not usable for serious work from the looks of it. Since we also need to tail logs of third party apps (without otel support) we will also have to rebuild our log pipelines (currently promtail) to use Filelog Receiver, but not sure if it already has all the necessary features (timestamp parsing, regex, json)
Correct.
The package we currently use uses
Correct.
Correct again. Other users who have been using promtail and push api longer than us may be after this too. We only recently rolled out pushing to loki, before that we only used promtail to tail log files of legacy apps. |
Created draft PR just to not lose what I have developed so far #10109. Also it could be the place for further discussions |
I will be on a long leave for a few months, and will not be able to continue working on this component. Feel free to use my draft PR if you continue working on this component |
Thank you for the discussion and draft @mar4uk. I'd like to see this component move forward in some form, but given the rather different approaches that appear to be on the table, I think it's worth discussing in the Collector SIG next week. I'll add it to the agenda. |
@djaglowski I was thinking about this and other components (like lokiexporter). There is also grafana-agent which similarily to otelcol collects logs, metrics and traces, but is tailored to be used with grafana cloud. I don't know how much grafana is already involved in OpenTelemetry (probably a lot) so please forgive my ignorence. |
@Mario-Hofstaetter, I'm glad you pointed out the AGPL license. Unfortunately, we cannot accept any dependency on code licensed under AGPL. In theory, we can consider an implementation that does not depend on copyleft-licensed source. However, given the need for clarity on this issue, I suggest that in the case such an implementation is possible, a fresh proposal should be drafted. |
Just an FYI not everything about Loki is AGPL. The logprotocol is Apache 2: https://github.com/grafana/loki/tree/main/pkg/logproto So a general: "Close this because AGPL" is a bit weird. I think having such a receiver component would indeed be nice to have for the reasons stated above. You'd have to re-implement the receiver services and can't just copy paste them from Loki for the license reasons but aside from that it shouldn't be a license issue. @jpkrohling Is there any interest from the Grafana side of things for a promtail receiver component in the opentelemetry-collector? |
I'll ask around and provide feedback here. If I don't get back in a couple of days, feel free to ping me again. |
I'm reopening, as:
|
I may have misunderstood, but when I looked into this I concluded that the proposal was dependent on AGPL licensed code. Taking another look at @mar4uk's draft PR, I still reach the same conclusion, as it depends on packages from Apologies if I'm misinterpreting this somehow. I'd like to understand what I'm missing if that's the case. |
It's not very intuitive, but the two packages that PR depends on are licensed under Apache Licence, despite the repository being licensed under AGPL-3. https://github.com/grafana/loki/blob/main/pkg/logproto/LICENSE_APACHE2 |
Got it. I missed the carve outs. I see they are listed here as well. |
@djaglowski, apparently, I forgot to set this to "accepted" component. Could you please confirm your understanding is also that this component has been accepted? |
@jpkrohling, I am still will to sponsor this so we can call it accepted. |
Hey @jpkrohling and @djaglowski , just found this issue here. What is the status for this? I would much appreciate that feature in the collector! Is there still a license problem? |
No, @mar4uk is coming back from her license, and we should have news soon. |
New PR #14632 was created. The license is not a blocker anymore |
This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping |
The purpose and use-cases of the new component
The Promtail receiver implements the Loki push api as specified here. It allows Promtail instances to specify the open telemetry collector as their
lokiAddress
.The implementation may be done analogously to fluentforwardreceiver
Example configuration for the component
The configuration of the receiver may be adapted from the loki_push_api configuration
Example
Telemetry data types supported
Logs only.
Sponsor (Optional)
The text was updated successfully, but these errors were encountered: