Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loki and alerts #340

Open
soum1234 opened this issue Feb 18, 2019 · 21 comments
Open

Loki and alerts #340

soum1234 opened this issue Feb 18, 2019 · 21 comments

Comments

@soum1234
Copy link

@soum1234 soum1234 commented Feb 18, 2019

Suppose I have to alert when there is an error log.let me know how to so it

@tomwilkie

This comment has been minimized.

Copy link
Member

@tomwilkie tomwilkie commented Feb 18, 2019

Hi @soum1234! We don't have a good answer for this yet, but it is something we plan on supporting. The current idea is we're going to first add a Prometheus-style query language for Loki, then use Prometheus-style rules and alerts on top of this.

@soum1234

This comment has been minimized.

Copy link
Author

@soum1234 soum1234 commented Feb 19, 2019

Thanks a lot for the response

@soum1234

This comment has been minimized.

Copy link
Author

@soum1234 soum1234 commented Mar 8, 2019

could we expect it in the next release?also could you let me know if there are some documents on querying in boltdb pls send the link
thanks

@BushnevYuri

This comment has been minimized.

Copy link

@BushnevYuri BushnevYuri commented Mar 19, 2019

Prometheus style alerting for logs sounds truly amazing. Hopefully, this feature will be added soon.

@mcamou

This comment has been minimized.

Copy link

@mcamou mcamou commented Jun 10, 2019

Has there been any work done on this? Any idea on when we can expect to be able to generate alerts in Grafana from Loki?

@cyriltovena

This comment has been minimized.

Copy link
Contributor

@cyriltovena cyriltovena commented Jun 12, 2019

I will start investigating this since LogQL has moved forward.

@JnMik

This comment has been minimized.

Copy link

@JnMik JnMik commented Jul 15, 2019

yeah that would be nice to be able to send alerts if a specific query return results.
I would build queries looking up for "error" keywords and stuff like that.

@JnMik

This comment has been minimized.

Copy link

@JnMik JnMik commented Jul 29, 2019

By the way, have you made any progress @cyriltovena ?

@cyriltovena

This comment has been minimized.

Copy link
Contributor

@cyriltovena cyriltovena commented Jul 29, 2019

Not yet merged, but you can use promtail to create those alerts in the meantime.

You can create metrics in promtail based on log retrieved from files, then scrape those promtail using a prometheus and create an alert.

(https://github.com/grafana/loki/blob/master/docs/logentry/processing-log-lines.md)

@yutakatay yutakatay mentioned this issue Aug 13, 2019
2 of 5 tasks complete
@stale

This comment has been minimized.

Copy link

@stale stale bot commented Sep 3, 2019

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Sep 3, 2019
@mcamou

This comment has been minimized.

Copy link

@mcamou mcamou commented Sep 3, 2019

How about if we're not using promtail? We're streaming logs directly into the Loki endpoint.

@stale stale bot removed the stale label Sep 3, 2019
@cyriltovena

This comment has been minimized.

Copy link
Contributor

@cyriltovena cyriltovena commented Sep 3, 2019

We have plan to support Prometheus alert style.

@kalingals

This comment has been minimized.

Copy link

@kalingals kalingals commented Oct 10, 2019

Hi @cyriltovena can you please tell me when do you plan to release Loki alerting feature?

@tomwilkie

This comment has been minimized.

Copy link
Member

@tomwilkie tomwilkie commented Oct 10, 2019

Loki already accepts PromQL like expressions (LogQL) on a Prometheus-compatible API, so the rough plan is to use the Cortex Ruler to evaluate Prometheus-style alert rules against logs. The Cortex Rules already exists, is horizontally scalable & highly available, so the integration should be relatively straight forward we hope.

This plan is in its super early stages mind you; input is more than welcome!

@bukowa

This comment has been minimized.

Copy link

@bukowa bukowa commented Jan 3, 2020

Not yet merged, but you can use promtail to create those alerts in the meantime.

You can create metrics in promtail based on log retrieved from files, then scrape those promtail using a prometheus and create an alert.

(https://github.com/grafana/loki/blob/master/docs/logentry/processing-log-lines.md)

Link is dead :(

@migueleliasweb

This comment has been minimized.

Copy link

@migueleliasweb migueleliasweb commented Jan 3, 2020

https://github.com/grafana/loki/tree/master/docs/clients/promtail/stages

You are looking for the stages section. With that, you can define metrics based on matches against the log lines. Then you use Prometheus & Alertmanager as you normally would to create the alert itself ;)

I reckon this issue can be closed.

@bukowa

This comment has been minimized.

Copy link

@bukowa bukowa commented Jan 3, 2020

@migueleliasweb thanks a lot, ill do that but i guess the issue should not be closed to provide built-in solution to this.

@migueleliasweb

This comment has been minimized.

Copy link

@migueleliasweb migueleliasweb commented Jan 3, 2020

Hey @bukowa, I'm glad I helped 😉 .

Regarding the built-in solution for alerting, I would be surprised if this will ever be a feature. The integration with Prometheus/Alertmanager is already there and in every release it gets improved. Replicating the funcionality of creating and managing alerts would be highly deprioritized or even unwanted in this context. But that's just my view.

@bukowa

This comment has been minimized.

Copy link

@bukowa bukowa commented Jan 3, 2020

@migueleliasweb My use case is loki for docker logs:

services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log
    command: -config.file=/etc/promtail/docker-config.yaml
    depends_on:
      - loki
      - grafana

  grafana:
    image: grafana/grafana:master
    ports:
      - "3000:3000"

  web:
    image: nginx:alpine
    ports:
      - 85:80
    logging:
      driver: loki
      options:
        loki-url: "http://localhost:3100/loki/api/v1/push"
        loki-external-labels: "container_name={{.Name}}.{{.ID}}"
        loki-batch-size: "5000"
    depends_on:
      - grafana
      - loki
      - promtail
    labels:
      test: 1

firefox_W1jDvQ1Feu

Everything works great (great project thank you!) but now i cannot setup alerts for these logs (i really love simple solutions). So here goes my idea that this could and should be supported.

@peimanja

This comment has been minimized.

Copy link

@peimanja peimanja commented Feb 4, 2020

Really interested in Prometheus-style rules and alerts and possibly ability to send the alerts to Alertmanager so same receivers and alert routes get applied there.

@cyriltovena

This comment has been minimized.

Copy link
Contributor

@cyriltovena cyriltovena commented Feb 6, 2020

This is great idea we want to implement.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants
You can’t perform that action at this time.