Skip to content

Add alerts endpoint#99

Merged
bzurkowski merged 3 commits intoopenrca:masterfrom
aleksandra-galara:Alerts_API_endpoint
Aug 17, 2020
Merged

Add alerts endpoint#99
bzurkowski merged 3 commits intoopenrca:masterfrom
aleksandra-galara:Alerts_API_endpoint

Conversation

@aleksandra-galara
Copy link
Copy Markdown
Member

It implements (#98).

Signed-off-by: Aleksandra Galara a.galara@samsung.com

It implements (openrca#98).

Signed-off-by: Aleksandra Galara <a.galara@samsung.com>
@aleksandra-galara
Copy link
Copy Markdown
Member Author

Payload example:

{
  "alerts": [
    {
      "id": "prometheus-alert-nodeclocknotsynchronising-node-192.168.122.250:9100",
      "origin": "prometheus",
      "properties": {
        "name": "NodeClockNotSynchronising",
        "kind": "node",
        "instance": "192.168.122.250:9100",
        "namespace": "{}",
        "message": "Clock on 192.168.122.250:9100 is not synchronising. Ensure NTP is configured on this host.",
        "severity": "warning"
      }
    },
    {
      "id": "prometheus-alert-kubecontainerwaiting-pod-test-1595997840-l848n-default",
      "origin": "prometheus",
      "properties": {
        "name": "KubeContainerWaiting",
        "kind": "pod",
        "instance": "test-1595997840-l848n",
        "namespace": "default",
        "message": "Pod default/test-1595997840-l848n container hello has been in waiting state for longer than 1 hour.",
        "severity": "warning"
      }
    }
  ]
}

Copy link
Copy Markdown
Member

@bzurkowski bzurkowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aleksandra-galara Thanks for the PR! 👍 I added some remarks to consider. Most of them are related to the payload structure. We must change it a little bit based on the discussion that we had yesterday with the UI team (some fields were missing). Also, the payload structure should be more flat to make it convenient for processing in the UI:

{
    "id": "xyz",
    "origin": "prometheus",
    "name": "SomethingHorribleHappenedWithAPod",
    "message": "It's bad :(",
    "severity": "critical",
    "source": {
        "origin": "kubernetes",
        "kind": "pod",
        "properties": {
            "name": "elasticsearch",
            "namespace": "logging"
        }
    },
    "created_at": 123456789,
    "updated_at": 123456789
}

Comment thread orca/api/__init__.py Outdated
from orca.api.resources.v1 import graph as graph_ns
from orca.api.resources.v1 import ingestor as ingestor_ns

from orca.api.resources.v1 import alert as alerts_ns
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's follow the convention and use the singular form: alert_ns.

Comment thread orca/api/resources/v1/alert.py Outdated
return marshal(data, alerts_fields)


def initialize(alerts):
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The variable name alerts is a little bit misleading. Let's rename to graph since the initialize method takes a graph instance as its argument.

Comment thread orca/api/resources/v1/alert.py Outdated
Comment on lines +22 to +24
'kind': fields.String(attribute='properties.source_mapping.kind'),
'instance': fields.String(attribute='properties.source_mapping.properties.name'),
'namespace': fields.String(attribute='properties.source_mapping.properties.namespace', default="n/a"),
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's put the source_mapping fields into a nested source field in the alert payload:

{
    "source": {
        "origin": "kubernetes",
        "kind": "pod",
        "properties": {
            "name": "elasticsearch",
            "namespace": "logging"
        }
    }
}

Also note that the source origin field is missing.

Comment thread orca/api/resources/v1/alert.py Outdated
node_fields = Model('Alert Node', {
'id': fields.String,
'origin': fields.String,
'properties': {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove the nesting via the properties field and treat the alerts payload flat instead.

'message': fields.String(attribute='properties.message'),
'severity': fields.String(attribute='properties.severity')
}
})
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's add timestamp fields: created_at, updated_at.

Comment thread orca/api/resources/v1/alert.py Outdated

class Alerts(Resource):

def __init__(self, api, alerts):
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The variable name alerts is a little bit misleading. Let's rename to graph since the API constructor takes a graph instance as its argument.

Comment thread orca/api/resources/v1/alert.py Outdated

from flask_restx import Model, Namespace, Resource, fields, marshal

node_fields = Model('Alert Node', {
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

node_fields is a left-over after the Graph API. Let's rename to alert_fields. We could also rename the alerts_fields (below) to alert_list_fields.

Comment thread orca/api/resources/v1/alert.py Outdated

def get(self):
data = {
'nodes': self._alerts.get_nodes(kind='alert'),
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we don't have to use the dict at this point. Instead, we should be able to directly pass the list of alert nodes to the marshal method. Try it out 😉

It introduces changes due
to review. (openrca#98)

Signed-off-by: Aleksandra Galara <a.galara@samsung.com>
@aleksandra-galara
Copy link
Copy Markdown
Member Author

As for now, the payload looks like that:

[
  {
    "id": "prometheus-alert-nodeclocknotsynchronising-node-192.168.122.250:9100",
    "origin": "prometheus",
    "name": "NodeClockNotSynchronising",
    "message": "Clock on 192.168.122.250:9100 is not synchronising. Ensure NTP is configured on this host.",
    "severity": "warning",
    "source": {
      "origin": "kubernetes",
      "kind": "node",
      "properties": {
        "name": "192.168.122.250:9100",
        "namespace": "{}"
      }
    },
    "created_at": "1595996122",
    "updated_at": "1595996122"
  },
  {
    "id": "prometheus-alert-kubepodcrashlooping-pod-zabbix-server-rkw4p-zabbix",
    "origin": "prometheus",
    "name": "KubePodCrashLooping",
    "message": "Pod zabbix/zabbix-server-rkw4p (zabbix-server) is restarting 0.43 times / 5 minutes.",
    "severity": "critical",
    "source": {
      "origin": "kubernetes",
      "kind": "pod",
      "properties": {
        "name": "zabbix-server-rkw4p",
        "namespace": "zabbix"
      }
    },
    "created_at": "1595996124",
    "updated_at": "1596021884"
  }
]

Comment thread orca/api/resources/v1/alert.py Outdated

def __init__(self, api, graph):
super().__init__()
self._alerts = graph
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's call things by name. Graph is not an alert list. Why not just: self._graph = graph? 😃

Comment thread orca/api/resources/v1/alert.py Outdated
Comment on lines +23 to +29
'source': {
'origin': fields.String(attribute='properties.source_mapping.origin'),
'kind': fields.String(attribute='properties.source_mapping.kind'),
'properties': {
'name': fields.String(attribute='properties.source_mapping.properties.name'),
'namespace': fields.String(attribute='properties.source_mapping.properties.namespace', default="n/a")
}
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering if we could refactor this a little bit by using the Nested field.

It introduces changes due
to review. (openrca#98)

Signed-off-by: Aleksandra Galara <a.galara@samsung.com>
Copy link
Copy Markdown
Member

@bzurkowski bzurkowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aleksandra-galara Thanks for the fixes. I wonder how the new endpoint operates with the new alerts dashboard in the UI 😉 Good work! 👍

@bzurkowski bzurkowski merged commit 4cd7744 into openrca:master Aug 17, 2020
@bzurkowski bzurkowski mentioned this pull request Sep 6, 2020
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants