feat(new sink): Initial loki sink
#1783
Conversation
Signed-off-by: Lucio Franco <luciofranco14@gmail.com>
Signed-off-by: Lucio Franco <luciofranco14@gmail.com>
Signed-off-by: Lucio Franco <luciofranco14@gmail.com>
| let events = events | ||
| .into_iter() | ||
| // The final json output should be: `[ts, line]` | ||
| .map(|e| json!([format!("{}", e.0), e.1])) |
There was a problem hiding this comment.
Would it be better to explicitly format the timestamp, e.g. to_rfc3339()?
There was a problem hiding this comment.
This is already an i64 but Loki wants it as a string instead of a json number.
src/sinks/loki.rs
Outdated
| })) | ||
| .unwrap(); | ||
|
|
||
| let uri = format!("{}loki/api/v1/push", self.host); |
There was a problem hiding this comment.
Are we checking that the configured value has a trailing slash anywhere?
There was a problem hiding this comment.
A valid Uri should always provide the / is what I've found in my experience. Since, our config uses UriSerde then this should be safe to assume.
src/sinks/loki.rs
Outdated
| } | ||
|
|
||
| async fn healthcheck(config: LokiConfig, resolver: Resolver) -> Result<(), crate::Error> { | ||
| let uri = format!("{}ready", config.host); |
There was a problem hiding this comment.
Same here with the trailing slash.
There was a problem hiding this comment.
Same as #1783 (comment)
|
And obviously once CI is happy as well 😄 |
|
@tomwilkie @slim-bean @cyriltovena @rfratto good news! We're coming in for a landing with our Loki integration 🎉 . We'd love it if someone at Grafana could review/QA this. For context, Vector is an observability data router, that is oh-so-perfect for the Prometheus/Loki/Grafana stack. Check it out and let me know what you think. I'd love to chat more about this integration. Feel free to reach out to me at ben@timber.io . |
binarylogic
changed the title
loki sinkloki sink
| required = true | ||
| examples = [{label = "value"}] | ||
| description = """\ | ||
| A set of labels that will be attached to each batch of events. These values\ |
There was a problem hiding this comment.
Maybe we should remind users to avoid to use high cardinality labels like they would in Prometheus. I mean may be this is obvious.
Do you guys have some sort of label/tag discovery from k8s ?
There was a problem hiding this comment.
We're about to. @ktff is working on a kubernetes_metadata transform that should be completed in the next week or 2, and we currently have these enrichment transforms.
There was a problem hiding this comment.
Cool this should make it to Loki sink !
There was a problem hiding this comment.
👍 I will add a note about high cardinality labels.
So we don't really have a concept of a schema for Kubernetes labels but we are about to provide a transform that can enrich your events with kube labels. It may make sense to add this to our docs to explain how you can combine that transform with this sink.
The idea is that you can configure loki like so:
[sinks.my_loki]
# ~~ snip other config items ~~
labels = {pod_id = "{{ labels.pod_id }}"}Related k8 issue #1072
cc @binarylogic this is kinda in the realm of guide vs sink doc page, I think it makes sense to add a note about kube possibly within the loki sink docs? What do you think?
There was a problem hiding this comment.
Yep, it is. I've got it and thanks for the heads up.
| })); | ||
| } | ||
|
|
||
| let body = serde_json::to_vec(&json!({ |
There was a problem hiding this comment.
FYI: We also support snappy compressed proto.
There was a problem hiding this comment.
👍 that is good to know, do you know if any of the current clients support this? I didn't see anything for this in fluentd/fluentbit but I did not dive super deep into promtail.
We currently don't support snappy within any of our sinks but it should be fairly easy to support.
There was a problem hiding this comment.
Not saying you should use it, although some clients can't and that could be an advantage.
There was a problem hiding this comment.
Only promtail/docker and fluentbit are using it, it's part of our go client.
|
|
||
| ### Decentralized deployments | ||
|
|
||
| Loki currently does not support out-of-order inserts. If Vector is deployed in a decentralized setup |
There was a problem hiding this comment.
FYI: we're trying to improve this. but it's hard cause it's the foundation of the db behind. One option we're looking into is being able to let loki rework/create timestamp server side.
There was a problem hiding this comment.
A bit off topic, but I am a huge fan of how facebooks logdevice handles multiple sequencers (ingestors) by giving up on a global ordering of timestamps but using a combination of (seq_id, ts) which would help with this problem.
https://logdevice.io/docs/Writepath.html#sequencer
Excited to see what you all come up with! :) For now I think the best is to just run things through a central vector instance, there might be some funky hacky solutions around this but I think centralized is good enough for now.
Signed-off-by: Lucio Franco <luciofranco14@gmail.com>
Signed-off-by: binarylogic <bjohnson@binarylogic.com>
Signed-off-by: binarylogic <bjohnson@binarylogic.com>
Signed-off-by: binarylogic <bjohnson@binarylogic.com>
Signed-off-by: Lucio Franco <luciofranco14@gmail.com>
This adds our initial implementation of the
lokisink. This sink is a bit unique in that we need to do some sorting before we send events to ensure that loki will accept them. We do this in thebuild_requestphase. This sink also doesn't use our partition batching techniques but instead internally multiplexes events since a single request can contain multiple "streams". Stream labels are also generated via aHashMapthat allows templatable values to allow users to route logs to specific streams based on event attributes.The docs are mostly complete but they could use a good review. Mostly, I am not sure if I did the labels section correctly.
To test this sink ensure that you have started the
lokidocker container and then runcargo test --features docker loki.Closes #557