AWX Grafana notification is not working

[mdobrzyn]

ISSUE TYPE

• AWX Grafana notification is not working

SUMMARY

The system is not able to send a notification to Grafana. Every time I receiving "Failed to start". Logs of awx_task container there are logs likes this:

• test notification

2020-03-03 08:57:05,279 ERROR    awx.main.tasks Send Notification Failed 'started'
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/tasks.py", line 324, in send_notifications
    sent = notification.notification_template.send(notification.subject, notification.body)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/models/notifications.py", line 196, in send
    return backend_obj.send_messages([notification_obj])
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/notifications/grafana_backend.py", line 46, in send_messages
    grafana_data['time'] = int((dp.parse(m.body['started']).replace(tzinfo=None) - epoch).total_seconds() * 1000)
KeyError: 'started'

• notification from job

2020-03-03 09:47:36,134 ERROR    awx.main.tasks Send Notification Failed string indices must be integers
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/tasks.py", line 324, in send_notifications
    sent = notification.notification_template.send(notification.subject, notification.body)
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/models/notifications.py", line 196, in send
    return backend_obj.send_messages([notification_obj])
  File "/var/lib/awx/venv/awx/lib/python3.6/site-packages/awx/main/notifications/grafana_backend.py", line 46, in send_messages
    grafana_data['time'] = int((dp.parse(m.body['started']).replace(tzinfo=None) - epoch).total_seconds() * 1000)
TypeError: string indices must be integers

ENVIRONMENT

• AWX version: 9.2.0
• AWX install method: docker on linux
• Ansible version: 2.9.5
• Operating System: Centos 7.5
• Web Browser: Chrome 79

STEPS TO REPRODUCE

1. Add notification type: Grafana
2. Add URL, API key

EXPECTED RESULTS

Notification will be send to Grafana

ACTUAL RESULTS

TEST Grafana: Notification failed.
'started'

ADDITIONAL INFORMATION

as above

[ryanpetrello]

@mdobrzyn thanks for reporting this - I've got a PR up that should address the issue:

#6145

[kdelee]

Set up grafana locally and pointed my awx to it, reproduced error on old version, and then updated to newest code that contains fixes and now able to test notifications and confirmed events arrive. Closing.

[kdelee]

I do want to mention there is the remote possibility that we could deploy grafana to k8s like we are doing for some services, and then we would need to integrate w/ grafana API to generate an api key and set up ability to verify the arrival of events. If that were ever to be desired, I estimate it would take several days to figure it all out.