-
Notifications
You must be signed in to change notification settings - Fork 11
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add alerts for graph-node subgraphs (#821)
Part of [Deploy v2 and updated v3 sushiswap subgraphs](https://www.notion.so/Deploy-v2-and-updated-v3-sushiswap-subgraphs-e331945fdeea487c890706fc22c6cc94) Reviewed-on: https://git.vdb.to/cerc-io/stack-orchestrator/pulls/821 Co-authored-by: Prathamesh Musale <prathamesh.musale0@gmail.com> Co-committed-by: Prathamesh Musale <prathamesh.musale0@gmail.com>
- Loading branch information
1 parent
254f95e
commit 8f2da38
Showing
4 changed files
with
93 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
64 changes: 64 additions & 0 deletions
64
stack_orchestrator/data/config/monitoring/subgraph-alert-rules.yml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,64 @@ | ||
apiVersion: 1 | ||
groups: | ||
- orgId: 1 | ||
name: subgraph | ||
folder: SubgraphAlerts | ||
interval: 30s | ||
rules: | ||
- uid: b2a9144b-6104-46fc-92b5-352f4e643c4c | ||
title: subgraph_head_tracking | ||
condition: condition | ||
data: | ||
- refId: diff | ||
relativeTimeRange: | ||
from: 600 | ||
to: 0 | ||
datasourceUid: PBFA97CFB590B2093 | ||
model: | ||
datasource: | ||
type: prometheus | ||
uid: PBFA97CFB590B2093 | ||
editorMode: code | ||
expr: ethereum_chain_head_number - on(network) group_right deployment_head{deployment=~"REPLACE_WITH_SUBGRAPH_IDS"} | ||
instant: true | ||
intervalMs: 1000 | ||
legendFormat: __auto | ||
maxDataPoints: 43200 | ||
range: false | ||
refId: diff | ||
- refId: condition | ||
relativeTimeRange: | ||
from: 600 | ||
to: 0 | ||
datasourceUid: __expr__ | ||
model: | ||
conditions: | ||
- evaluator: | ||
params: | ||
- 15 | ||
- 0 | ||
type: gt | ||
operator: | ||
type: and | ||
query: | ||
params: [] | ||
reducer: | ||
params: [] | ||
type: avg | ||
type: query | ||
datasource: | ||
name: Expression | ||
type: __expr__ | ||
uid: __expr__ | ||
expression: diff | ||
intervalMs: 1000 | ||
maxDataPoints: 43200 | ||
refId: condition | ||
type: threshold | ||
noDataState: OK | ||
execErrState: Alerting | ||
for: 5m | ||
annotations: | ||
summary: Subgraph deployment {{ index $labels "deployment" }} is falling behind head by {{ index $values "diff" }} | ||
labels: {} | ||
isPaused: false |
7 changes: 7 additions & 0 deletions
7
stack_orchestrator/data/config/monitoring/update-grafana-alerts-config.sh
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
#!/bin/bash | ||
|
||
echo Using CERC_GRAFANA_ALERTS_SUBGRAPH_IDS ${CERC_GRAFANA_ALERTS_SUBGRAPH_IDS} | ||
|
||
# Replace subgraph ids in subgraph alerting config | ||
# Note: Requires the grafana container to be run with user root | ||
sed -i "s/REPLACE_WITH_SUBGRAPH_IDS/$CERC_GRAFANA_ALERTS_SUBGRAPH_IDS/g" /etc/grafana/provisioning/alerting/subgraph-alert-rules.yml |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters