Proposal: Create VictoriaLogs

[zbindenren]

Is your feature request related to a problem? Please describe.
Grafana Loki is a log aggregation system inspired by Prometheus. It does not index the contents of the logs, but rather a set of labels for each log stream.

Unfortunately it is hard to run it on premises because it has dependencies to Cassandra, S3 etc...

Describe the solution you'd like
It would be greate to extend vmstorage, vminsert and vmselect to support loki protocol. I think vminsert and vmselect would not be a problem, but I am not sure about vmstorage.

[faceair]

I've finished a very early version of this. https://github.com/faceair/VictoriaLogs

[syepes]

OMG! This would be just amazing and a great addition to the Victoria Family!

[faceair]

These are the changes I made:

• vmstorage:
  • change value format from float64 to []byte, use zstd compression
  • added option to fetch only timestamps, which can improve performance when query logging metrics
• vmselect:
  • extend support logql filter expression
  • support loki http api & response format
• vminsert
  • support loki push api
  • support prometheus-style data writing via tcp

I changed some code in vminsert & vmselect that was necessary, and I'm not sure how I'm going to keep it in sync with the VictoriaMetrics repository code in the future.
The other bigger problem is that I have modified the storage data type of vmstorage so that I have no way to directly import the VictoriaMetrics repository and it is difficult to merge VictoriaLog into VictoriaMetrics.

Here's my new proposal VictoriaMetrics/VictoriaMetrics#819

[zbindenren]

Cool, I wonder if this gets accepted...

I see that you @faceair already started with this 5 days ago. So we had basically the same idea 😄

[syepes]

@valyala What is your take on this functionality? Or are you guys going to focus just on the metrics?

[valyala]

What is your take on this functionality? Or are you guys going to focus just on the metrics?

We are focusing on metrics right now, but VictoriaLogs is in our plans. It can be based on https://github.com/faceair/VictoriaLogs .

[faceair]

It looks like I can maintain VictoriaLogs on my own for a while, and later I will improve the compatibility with Loki and test the real performance.
I look forward to merging VictoriaLogs into VictoriaMetrics in the future.

[te4336]

@zbindenren If you happen to have the capability you can provide an S3 compatible service using Minio and then just use the Bolt-DB shipper function in the latest version of Loki. While having this as part of the Victoria Metrics family would be awesome not sure it's worth people re-inventing the wheel when there are solutions to get around some of the extraneous requirements for Loki.

[misiek08]

@te4336 almost 2 years passed and Loki is still unusable for scenarios with real logging data. Trying to query anything hitting more than 50k logs is exploding servers :)

I'll try to contribute to VictoriaLogs and I'm glad I found this issue and VictoriaLogs repo ❤️

[mxssl]

Do want!

[v1gnesh]

Hello,

Post the recent community day and expressed interest to create VictoriaLogs, please see if there are patterns & methods from this that can be rebuilt in Go, for VL.

https://www.youtube.com/watch?v=23iHoDeO7M4
https://www.uber.com/en-IN/blog/reducing-logging-cost-by-two-orders-of-magnitude-using-clp/

[tenmozes]

We started working on Victoria Logs, you can find some information here https://www.youtube.com/live/Gu96Fj2l7ls?feature=share&t=1952

[zbindenren]

We started working on Victoria Logs, you can find some information here https://www.youtube.com/live/Gu96Fj2l7ls?feature=share&t=1952

Hope that there will be a grafana datasource. Looks good.