Grafana is an open source metric analytics & visualization suite. It is most commonly used for visualizing time series data for infrastructure and application analytics but many use it in other domains including industrial sensors, home automation, weather, and process control. This integration extends the ability to alert the person on call to a group and pause or unpause an alert coming out of Grafana using out closed-loop integration. Check out the sweet video!
- GrafanaAlert.zip - Communication Plan containing the Inbound and Outbound integration with form templates
An alert fires a webhook to the xMatters integration builder. The integration builder parses the incoming JSON and builds the event, then fires the event to notify the default recipients. Alternatively subscriptions can be set up to notify the desired parties. From the alert, you can pause the alert in Grafana to investigate the issue. When the issue is fixed, you can resume the alerting rule.
- Login as the integration user in xMatters and navigate to the Developer Tab. (NOTE: Make sure the integration user has REST Web Services User Role. Reference General Notes section on how to create API user in xMatters.
- Import the GrafanaAlert.zip Steps to import a communication plan: Here
- Enable the plan, click Edit and select "Layout"
- Insert your targeted reciepients into the recipient field
- Save changes and return to the previous page. Click on Edit, then "Integration Builder" and copy the URL at the bottom to have it handy later.
- Close the "Update Inbound Integration" and return to the main page of the Integration Builder. On this main page, at the bottom are shared libraries. Select or edit the Grafana Library.
- Insert the API Key you generated in Grafana into the code. Reference "General Notes" section on how to generate the API key
- Add the Grafana Endpoint by clicking on "Edit Endpoints" in the Integration Builder. Add an endpoint and name it "Grafana". Add the URL to your Grafana instance. Make sure you save changes.
- You first need to set up a notification channel in Grafana which can be found under the Alerting section
- Click on "New Channel" and create a new webhook found under "type". Name it, set Http Method to "POST", set URL as the URL you saved. Reference step 6 in the xMatters set up for the URL. Your channel should look something like this. **NOTE: If you want to test out and see if the webhook works, you can click "Send Test" at the bottom.
- Go to the chart you want to be alerted on in Grafana, click "edit", click "Alert", then click "Notifications". Click the plus icon and add the Channel you just added in the previous step.
- How to set up the Grafana API Key to plug into xMatters. PLEASE NOTE: Grafana will only show the API you created ONCE. Suggestion is to copy and paste it in a notepad somewhere so you can get to it if you ever need it again
- How to create API User in xMatters Click "Users", click "Invite Users" or "Add Users". Information should be something like this.
If it requires an email, the email can be something like email@xmatters.com
Do an action in your Gafana application or infrastructure that will trigger the alert conditions set up in the Alert. This will fire the webhook into the inbound integration builder script and an event will be created, targeting the default recipients. After it targets the right person, you can use our 2-way integration by responding with "Pause Alert" to pause the alerting process in Grafana. Once you fix the issue in Grafana, you can come back to the original xMatters alert and use the response option "Resume Alert" to resume the alerting rule.
If it doesn't work, you can reference the "Activity Stream" of the integration. Please reference both screen shots to find the Activity Stream and what it should look like. You can look through the logs to see what the error is.
