HTTP API for getting alerts

[hryamzik]

I didn't find anything about API for the /alerts in docs. I can see that /alerts returns html:

router.Get("/alerts", instrf("alerts", h.alerts))

But is it possible to get them with json?

[brian-brazil]

There is no HTTP API, however you can access the ALERTS variable via the usual query API which contains most of the same information.

[hryamzik]

Looks like only firing alerts are available there and annotations are missing.

[fabxc]

There was a PR for that a while back. We should reconsider having an API
endpoint.

I raised before that we generally want to have an API that allows to build
a fully custom UI on top. If we ever get to a point where we can build a
more dynamic default API we will need it anyway.

On Thu, Apr 21, 2016, 9:32 PM Roman Belyakovsky notifications@github.com
wrote:

Looks like only firing alerts are available there and annotations are
missing.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#1570 (comment)

[hryamzik]

I'm thinking of a telegram bot to get more details on alerts when internet connection is limited and web UI is difficult to access. By the way it's not mobile friendly for some reason, pretty unusual for a bootstrap based one.

[brian-brazil]

For that sort of thing, a link to a dashboard generally works best.

[hryamzik]

That's much more to load than just an annotation string. So looking forward for an API. =)

[mtanda]

Personally I want this to embed Alert status in Grafana.
Grafana scripted dashboard (or panel plugin) can make alert dashboard.
(I make this by parsing HTML, mtanda/grafana@6b357c3 )

If Prometheus provide API, it's very helpful.

[iksaif]

ALERTS is fine but doesn't seem to show inactive alerts. An API would be great.

Looked quickly at the code, looks like simply converting to json the result of h.ruleManager.AlertingRules() would be a good start.

[fabxc]

Such an API would get accepted as part of our effort to make all UI features represented in our HTTP API. Contributions are gladly accepted.

[copernico]

+1

[mg03]

@fabxc : i have some code written to expose endpoint /api/v1/alerts which puts out a json like so

{
  "status": "success",
  "data": {
    "alerts": [
      {
        "name": "promup",
        "query": "up{job=\"prometheus\"} != 1",
        "duration": "1m0s",
        "labels": {
          "severity": "critical"
        },
        "annotations": {
          "dashboard": "http://grafana.com/dashboard/somedashbaord",
          "description": "lol hahahahha",
          "summary": "Prom Up"
        },
        "status": "inactive"
      },
      {
        "name": "promx",
        "query": "up{job=\"prom\"} < 1",
        "duration": "1m5s",
        "labels": {
          "severity": "critical"
        },
        "annotations": {
          "dashboard": "http://grafana.com/dashboard/somedashbaord",
          "description": "sad sad",
          "summary": "Promxxxxx Up"
        },
        "status": "pending",
        "activesince": "2017-03-31T19:09:07.808-07:00"
      },
      {
        "name": "promnoann",
        "query": "up{job=\"prom\"} < 1",
        "duration": "10s",
        "labels": {
          "severity": "critical"
        },
        "status": "firing",
        "activesince": "2017-03-31T19:09:07.808-07:00"
      }
    ]
  }
}

Is there any other info that I can expose via the api?
I have written a test case for the above api.
I can make a PR to the repo, would that be acceptible or is there some process (sorry noob to contributing)

[moolitayer]

I don't understand, there already seems to be an API endpoint exposed from alert manager.
That one seems to show also alerts that no longer firing:

GET http://alerts-vm-48-125.myserver.com/api/v1/alerts

{"status":"success","data":[{"labels":{"alertname":"Testing","instance":"vm-48-124.eng.lab.tlv.redhat.com","job":"kubernetes-nodes","monitor":"codelab-monitor"},"annotations":{"description":"a description","summary":"a summary"},"startsAt":"2017-05-17T23:24:37.804Z","endsAt":"2017-05-17T23:27:37.804Z","generatorURL":"http://prometheus-1161204464-e5li2:9090/graph?g0.expr=process_cpu_seconds_total%7Binstance%3D%22vm-48-124.eng.lab.tlv.redhat.com%22%2Cjob%3D%22kubernetes-nodes%22%7D+%3C+4615\u0026g0.tab=0"}]}

Where does that fit in this discussion?

[jsuchenia]

@moolitayer In a project name alertmanager != prometheus. However with federation you can scrap alerts like metrics:
http://prometheus.instance/federate?match[]=ALERTS

[moolitayer]

So there is an API for alerts in alert manager but not in Prometheus?

[juliusv]

@moolitayer Correct.

[moolitayer]

Thanks @juliusv. I'm interested in collecting alerts from Prometheus and integrating them in an external system. Here is another related question;

For effective collection I would like like to collect only new alerts using pull every X seconds.
Is there a mechanism I can use to get only new alerts? (happened or were resolved since last collection)

(possible solutions: using a time field OR using labels for collected alerts OR a streaming api similar to the kubernetes events endpoint)

[jsuchenia]

@moolitayer It doesn't make sense. From prometheus point of view it's collecting data every X time and evaluates rules every Y time. If you are scrapping alerts every Z seconds and it's more than Y then you expect that prometheus will keep historical alerts till your next check. Additionally how to handle restarts and your problems (like you lost updates about some resolve)

So it's quite good that at one shot you can get a list of active alarms - you have a clear picture of a status

[moolitayer]

@jsuchenia It seems alert manager already has that data and I'm wondering if there is a mechanism there to get what I need? If an alert has fired and was immediately resolved for example I would still like to know about it so only resolved alerts are not enough for my use case.

[jsuchenia]

@moolitayer According to Alert Manager configuration you can specify a webhook that will be triggered whenever an alert state was changed. Is this something that works for you?

[moolitayer]

@moolitayer According to Alert Manager configuration you can specify a webhook that will be triggered whenever an alert state was changed. Is this something that works for you?

Thanks @jsuchenia. I have to initiate all the connections due to security concerns. Is that possible somehow? (kubernetes watch api is a good example of what I need)

[brian-brazil]

@brancz Did you implement this already?

[brancz]

@brian-brazil I implemented the alertmanagers endpoint, not alerts. That's still tbd. However as suggested before I would recommend to implement a system as described above as a webhook that is called by the Alertmanager instead of scraping Prometheus alerts. Nonetheless there are usecases for the alerts API.

[moolitayer]

In case this is of useful to someone, to collect alerts we web hooked alerts into another component (running on the same k8s pod as alert manager) and we are collecting from that component:
https://github.com/openshift/prometheus-alert-buffer

[midnightconman]

I have a use case where I would like to see triggered and non-triggered alerts, the current alertmanager alerts API currently only exposes triggered alerts. Would you guys welcome a PR to add functionality to be able to glean all alerts? I don't need an API, just the information exposed... I would be happy with adding a label to the ALERTS metric and exposing all alerts there. Thoughts?

[brian-brazil]

It's not possible to list non-firing alerts, as we don't know their labels.

[midnightconman]

That's a good point... I really don't need an alerts api, but a rules api. I don't mind if it is read-only, just something better than html. In Prometheus 2.0 using /api/v1/rules I am getting: HTTP/1.1 405 Method Not Allowed

[iksaif]

Maybe a good extension for /api/v1/status/config ?

[midnightconman]

#3656

[omadawn]

I also have an existing need to be able to retrieve all rules via an API (mainly non-firing alerts.) Either #3656 or #3391 should address my need. Is there a plan to implement it yet and can I vote for this functionality if not?

[burnettk]

In case anyone comes in here looking for an firing alert dashboard in grafana, here's one: https://grafana.com/dashboards/4181

[amaury-d]

Hello,

For anyone else looking at the answer, you can call /api/v1/query?query=ALERTS on Prometheus

[simonpasquier]

See #4318

[ealexhaywood]

@amaury-d yes, but that only returns firing alerts. People here want inactive and pending alerts as well

[mxinden]

@ealexhaywood #4318 exposes pending alerts via api/v1/alerts as well. Inactive alerts are not exposed.

@juliusv as the initial issue is about getting alerts in general, should we close here with #4318 being merged?

[juliusv]

@mxinden Yes, thank you! Closing.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.