Clone this wiki locally
New to netdata? Check its demo: http://my-netdata.io/
table of contents
- Health Configuration
- Alarm Actions
- API Calls
netdata got health monitoring!
Dear dev-ops and sys-admins, netdata got alarms!
A few months ago, when I decided to let the netdata users decide the features they need us to develop, I was somewhat surprised that most users wanted health monitoring.
I think I get it now.
Health monitoring is problematic for most people. I have not seen a single sys-adm or dev-op totaly happy with the tools he/she has.
So, I decided to build a health monitoring system in netdata that will overcome most of the problems other systems have:
Of course an alarm is just a threshold, like
A > 90.
But netdata goes a lot beyond that...
netdata allows you to correlate different metrics. It is not just
A > 90. It can be:
(A > 90 AND (B > 80 OR C < 40)) OR (D > 50 AND E < 30)). This means for example that you can raise an alarm on the number of database requests only when the disk is congested, or when cpu utilization is too high too.
netdata allows you to take into account the values of the same metric some time in the past. So you can say:
A(now) > 90 AND A(30 mins ago) < 30.
You can even calculate rates:
A(now) - A(30 mins ago) / (30 * 60) = the rate A changes over the last 30 minutes. Then you can raise an alarm when:
A(rate of the last 30 mins) > 10. This means you can detect if your web server is facing an abnormal request flood, or if your web server although operational is getting way too low requests.
You can calculate a percentage of change over the last period:
(A(now) - A(1 min ago)) * 100 / (A(1 min ago) - A(2 min ago)) = the percentage of the volume of the last minute compared to the volume of its previous minute. This means you can track your servers minute-by-minute and trigger alarms based on their changes.
netdata also allows you to evaluate simple expressions on any timeframe of a metric. For example, you can use in expressions the
sum of any metric for any timeframe.
Then we come to configuration. All of you that use netdata already, know I hate configuration. I find absolutely no joy in configuring applications. Although netdata provides tons of configuration options, I always do my best so that most installations will need to configure nothing.
So, netdata comes with pre-defined alarms for detecting the most common problems. Out of the box:
- it will trigger alarms when the applications it monitors stop
- it will detect network interfaces errors
- it will detect disks not catching up with the load they are offered
- it will detect low disk space on any disk
- it will even predict in how many hours your system is going to be out of disk space and notify you if it is less than 48 hours.
- it will alert you if your system is running low on entropy (random numbers pool)
Even when you need to configure alarms by hand, netdata offers Alarm Templates. Once you have configured an alarm, netdata can apply this alarm on all similar charts/metrics. So, if for example, you build an alarm to detect a web server request flood, netdata can apply this alarm to all your web servers automatically.
netdata also offers Context Based Variables. When you configure an alarm to a chart, netdata automatically brings you as variables all the chart dimensions and all the dimensions and alarms of the charts that belong to the same family (e.g. family =
mysql server 1). This creates a context where similar things are available to be used with their "first" name.
netdata alarms are based on expressions. These expressions can use data from any chart, any dimension, any metric. If you want, you can correlate the backlog of the disk, to the number of database queries, to the packets rate of a network interface, to number of requests to a web server. This together with poweful database lookup and reduce functions allow you to create alarms for everything imaginable.
Check the health.d directory for all alarms shipped with netdata.
Here are a few examples:
A simple check if an apache server is alive:
template: apache_last_collected_secs on: apache.requests calc: $now - $last_collected_t every: 10s warn: $this > ( 5 * $update_every) crit: $this > (10 * $update_every)
The above checks that netdata is able to collect data from apache. In detail:
The above defines a template named
apache_last_collected_secs. The name is important since
$apache_last_collected_secs resolves to the
calc line. So, try to give something descriptive.
The above applies the template to all charts that have
context = apache.requests (i.e. all your apache servers).
calc: $now - $last_collected_t
$now is a standard variable that resolves to the current timestamp.
$last_collected_t is the last data collection timestamp of the chart. So this calculation gives the number of seconds passed since the last data collection.
The alarm will be evaluated every 10 seconds.
warn: $this > ( 5 * $update_every) crit: $this > (10 * $update_every)
If these result in non-zero or true, they trigger the alarm.
$this refers to the value of this alarm (i.e. the result of the
calc line. We could also use
$update_every is the update frequency of the chart, in seconds.
So, the warning condition checks if we have not collected data from apache for 5 iterations and the critical condition checks for 10 iterations.
Check if any of the disks is critically low on disk space:
template: disk_full_percent on: disk.space calc: $used * 100 / ($avail + $used) every: 1m warn: $this > 80 crit: $this > 95
$avail are the
avail chart dimensions as shown on the dashboard.
calc line finds the percentage of used space.
$this resolves to this percentage.
Predict if any disk will run out of space in the near future.
We do this in 2 steps:
Calculate the disk fill rate
template: disk_fill_rate on: disk.space lookup: max -1s at -30m unaligned of avail calc: ($this - $avail) / (30 * 60) every: 15s
$thisis the result of the
lookupline (i.e. the free space 30 minutes ago) and
$availis the current disk free space. So the
calcline will either have a positive number of GB/second if the disk if filling up, or a negative number of GB/second if the disk is freeing up space.
There is no
critlines here. So, this template will just do the calculation and nothing more.
Predict the hours after which the disk will run out of space
template: disk_full_after_hours on: disk.space calc: $avail / $disk_fill_rate / 3600 every: 10s warn: $this > 0 and $this < 48 crit: $this > 0 and $this < 24
calcline estimates the time in hours, we will run out of disk space. Of course, only positive values are interesting for this check, so the warning and critical conditions check for positive values and that we have enough free space for 48 and 24 hours respectively.
Once this alarm triggers we will receive an email like this:
Check if any network interface is dropping packets:
template: 30min_packet_drops on: net.drops lookup: sum -30m unaligned absolute every: 10s crit: $this > 0
lookup line will calculate the sum of the all dropped packets in the last 30 minutes.
crit line will issue a critical alarm if even a single packet has been dropped.
Note that the drops chart does not exist if a network interface has never dropped a single packet. When netdata detects a dropped packet, it will add the chart and it will automatically attach this alarm to it.
Everything related to health configuration is in
/etc/netdata/health.d. In this directory you can put any number of files (in any number of sub-directories) with a suffix
.conf to have them processed by netdata.
Health configuration can be reloaded at any time, without restarting netdata. Just send netdata the SIGUSR2 signal, like this:
killall -USR2 netdata
Reloading health configuration will re-do the actions of the alarms (i.e. will re-send emails for warning and critical alarms).
Entities in the health files
There are 2 entities:
alarms, which are attached to specific charts, and
templates, which define rules that should be applied to all charts having a specific
context. You can use this feature to apply alarms to all disks, all network interfaces, all mysql databases, all nginx web servers, etc.
Both of these entities have exactly the same format and feature set. The only difference is the label
netdata supports overriding templates with alarms. For example, when a template is defined for a set of charts, an alarm with exactly the same name attached to the same chart the template matches, will have higher precedence (i.e. netdata will use the alarm on this chart and prevent the template from being applied to it).
The following lines are parsed:
This line has to be first on each alarm or template.
NAMEis anything you would like to name it (the only symbols allowed are
on: CHART(for alarms) or
on: CONTEXT(for templates)
This line defines the data the alarm should be attached to.
CHARTyou can use a chart
name, as shown on the dashboard.
CONTEXTis the template of a chart. For example the charts
mysql_server2.nethave the same context:
mysql.net. So, you can use this to apply alarms to all
To find the
CONTEXTof a chart download the latest
netdata.conf, find the chart you are interested and you will see its context in its configuration section. You can also find it if you fetch
http://your.netdata:19999/api/v1/charts(again search for the chart and you will find its context).
lookup: METHOD AFTER [at BEFORE] [every DURATION] [OPTIONS] [of DIMENSIONS]
This makes a database lookup to find a value. Everything is the same with badges. In short:
METHODis one of
incremental-sum. This is required.
AFTERis a relative number of seconds, but it also accepts a single letter for changing the units, like
-1s= 1 second in the past,
-1m= 1 minute in the past,
-1h= 1 hour in the past,
-1d= 1 day in the past. You need a negative number (i.e. how far in the past to look for the value). This is required.
at BEFOREis by default 0 and is not required. Using this you can define the end of the lookup. So data will be evaluated between
every DURATIONsets the updated frequency of the lookup (supports single letter units as above too).
unaligned. Check also the the badges documentation for more info.
of DIMENSIONSis optional and has to be the last parameter. Dimensions have to be separated by
|. The space characters found in dimensions will be kept as-is (a few dimensions have spaces in their names).
The result of the lookup will be available as
$NAMEin expressions. The timestamps of the timeframe evaluated by the database lookup is available as variables
$before(both are unix timestamps).
Set the green and red thresholds of a chart. Both are available as
Sets the update frequency of this alarm. This is the same to the
every DURATIONgiven in the
This expression is evaluated just after the lookup. Its purpose is to apply some calculation before using the value looked up from the db. You can also have an expression without a lookup, using other variables that are available (so you can create new variables).
The result of the calculation will be available as
$NAMEin warning and critical expressions (overwriting the
Check expressions below for more information.
These expressions should evaluate to true or false (alternatively non-zero or zero). These trigger the alarm.
The script that will be executed when the alarm changes status.
This will be the first parameter of the script to be executed. Its meaning is left up to the
execscript. The default
alarm-notify.sh, uses this field as a space separated list of roles, which are then consulted to find the exact recipients per notification method.
delay: [[[up U] [down D] multiplier M] max X]
This is used to provide optional hysteresis settings for the notifications, to defend against notification flood. These settings do not affect the actual alarm - only the time the
execscript is executed.
up Udefines the delay to be applied to a notification for an alarm that raised its status (i.e. CLEAR to WARNING, CLEAR to CRITICAL, WARNING to CRITICAL). For example,
up 10s, the notification for this event will be send 10 seconds after the actual event. This is used in hope the alarm will get back to its previous state within the duration given. The default
down Ddefines the delay to be applied to a notification for an alarm that moves to lower state (i.e. CRITICAL to WARNING, CRITICAL to CLEAR, WARNING to CLEAR). For example,
down 1mwill delay the notification by 1 minute. This is used to prevent notifications for flapping alarms. The default
Dwhen an alarm changes state, while a notification is delayed. The default multiplier is
max Xdefines the maximum absolute notification delay an alarm may get. The default
max(U * M, D * M)(i.e. the max duration of
Dmultiplied once with
delay: up 10s down 15m multiplier 2 max 1h
The time is
00:00:00and the status of the alarm is CLEAR.
time of event new status delay notification will be sent why 00:00:01 WARNING
00:00:11 first state switch 00:00:05 CLEAR
down 15m x2
00:30:05 the alarm changes state while a notification is delayed, so it was multiplied 00:00:06 WARNING
up 10s x2 x2
00:00:26 multiplied twice 00:00:07 CLEAR
down 15m x2 x2 x2
00:45:07 multiplied 3 times.
Dare multiplied by
Mevery time the alarm changes state (any state, not just their matching one) and a delay is in place.
- All are reset to their defaults when the alarm switches state without a delay in place.
netdata has an internal infix expression parser. This parses expressions and creates an internal structure that allows fast execution of them.
These operators are supported
NOT. Boolean operators result in either
1 (true) or
The conditional evaluation operator
? is supported too. Using this operator IF-THEN-ELSE conditional statements can be specified. The format is:
(condition) ? (true expression) : (false expression). So, netdata will first evaluate the
condition and based on the result will either evaluate
true expression or
false expression. Example:
($this > 0) ? ($avail * 2) : ($used / 2). Nested such expressions are also supported (i.e.
true expression and
false expression can contain conditional evaluations).
Expressions also support the
Expressions can have variables. Variables start with
$. Check below for more information.
There are two special values you can use:
nan, for example
$this != nanwill check if the variable
thisis available. A variable can be
nanif the database lookup failed. All calculations (i.e. addition, multiplication, etc) with a
nanresult in a
nan. netdata will log an error if any
critexpressions result in
inf, for example
$this != infwill check if
thisis not infinite. A value or variable can be infinite if divided by zero. All calculations (i.e. addition, multiplication, etc) with a
infresult in a
inf. netdata will log an error if any
critexpressions result in
netdata supports 3 new internal indexes for variables that will be used in health monitoring:
chart local variables. All the dimensions of the chart are exposed as local variables. All chart alarms are exposed too.
charts define a few special variables:
last_collected_tis the unix timestamp of the last data collection
collected_total_rawis the sum of all the dimensions (their last collected values)
update_everyis the update frequency of the chart
redthe threshold defined in alarms (these are per chart - the charts inherits them from the the first alarm that defined them)
dimensions define their last calculated (i.e. interpolated) value, exactly as shown on the charts, but also a variable with their name and suffix
_rawthat resolves to the last collected value - as collected and another with suffix
_last_collected_tthat resolves to unix timestamp the dimension was last collected (there may be dimensions that fail to be collected while others continue normaly).
family variables. Families are used to group charts together. For example all
family = eth0. This index includes all local variables, but if there are overlaping variables, only the first are exposed.
host variables. All the dimensions of all charts, including all alarms, in fullname. Fullname is
CHARTis either the chart id or the chart name (both are supported).
There are also a few special variables:
this, which is resolved to the value of the current alarm.
status, which is resolved to the current status of the alarm (the current = the last status, i.e. before the current database lookup and the evaluation of the
calcline). This values can be compared with
$CRITICAL. These values are incremental, ie.
$status > $CLEALworks as expected.
now, which is resolved to current unix timestamp.
You can find all the variables that can be used for a given chart, using
http://your.netdata.ip:19999/api/v1/alarm_variables?chart=NAME. This will dump all the indexes from the chart's perspective. Example: variables for the
system.cpu chart of the registry.
exec line in health configuration defines an external script that will be called once the alarm is triggered. The default script is alarm-notify.sh. You can change the default script by editing
/etc/netdata/netdata.conf (if you installation is old, you may need to download a fresh copy of it from netdata using the url `http://your.netdata:19999/netdata.conf).
alarm-notify.sh is capable of:
- sending emails
- sending pushover.net notifications (push notification to your mobile)
- sending pushbullet.com notifications (push notifications to all devices linked within pushbullet account)
- sending messages to slack.com channels
It uses roles. For example
Each alarm is assigned to one or more roles, using the
to line of the alarm configuration. Then
alarm-notify.sh uses its own configuration (
/etc/netdata/health_alarm_notify.conf the default is here) to find the destination address of the notification for each method. Each role may have one or more destinations.
So, for example the
sysadmin role may send:
- emails to firstname.lastname@example.org and email@example.com
- pushover.net notifications to USERTOKENS
- pushbullet.com push notifications to firstname.lastname@example.org and email@example.com
- messages to slack.com channel
You can run the following command by hard, to test alarms configuration:
# become user netdata su -s /bin/bash netdata # send test alarms to sysadmin /usr/libexec/netdata/plugins.d/alarm-notify.sh test # send test alarms to any role /usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
Web Browser Notifications
The netdata dashboard shows HTML notifications, when it is open. Such web notifications look like this:
You need a working
sendmail command for email alerts to work. Almost all MTAs provide a
alarm-notify.sh will send the emails from user
netdata to the email address of the recipient.
/etc/netdata/health_alarm_notify.conf to configure recipients per role.
email notifications look like this:
Pushbullet.com push notifications
Will look like this on your browser:
And like this on your Android device:
You will need:
- Signup and Login to pushbullet.com
- Get your Access Token, go to https://www.pushbullet.com/#settings/account and create a new one
- Fill in the PUSHBULLET_ACCESS_TOKEN with that value
- Add the recipient emails to DEFAULT_RECIPIENT_PUSHBULLET !!PLEASE NOTE THAT IF THE RECIPIENT DOES NOT HAVE A PUSHBULLET ACCOUNT, PUSHBULLET SERVICE WILL SEND AN EMAIL!!
Set them in
/etc/netdata/health_alarm_notify.conf, like this:
############################################################################### # pushbullet (pushbullet.com) push notification options # multiple recipients can be given like this: # "firstname.lastname@example.org email@example.com" # enable/disable sending pushbullet notifications SEND_PUSHBULLET="YES" # Signup and Login to pushbullet.com # To get your Access Token, go to https://www.pushbullet.com/#settings/account # And create a new access token # Then just set the recipients emails # Please note that the if the email in the DEFAULT_RECIPIENT_PUSHBULLET does # not have a pushbullet account, the pushbullet service will send an email # to that address instead # Without an access token, netdata cannot send pushbullet notifications. PUSBULLET_ACCESS_TOKEN="o.Sometokenhere" DEFAULT_RECIPIENT_PUSHBULLET="firstname.lastname@example.org email@example.com"
This is what you will get:
- The incoming webhook URL as given by slack.com. You can use the same on all your netdata servers (or you can have multiple if you like - your decision).
- One or more channels to post the messages to.
Get them here: https://api.slack.com/incoming-webhooks
Set them in
/etc/netdata/health_alarm_notify.conf, like this:
############################################################################### # sending slack notifications # note: multiple recipients can be given like this: # "CHANNEL1 CHANNEL2 ..." # enable/disable sending pushover notifications SEND_SLACK="YES" # Login to slack.com and create an incoming webhook. # You need only one for all your netdata servers. # Without it, netdata cannot send slack notifications. SLACK_WEBHOOK_URL="https://hooks.slack.com/services/XXXXXXXX/XXXXXXXX/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" # if a role recipient is not configured, a notification will be send to # this slack channel: DEFAULT_RECIPIENT_SLACK="alarms"
You can define multiple channels like this:
You can give different channels per role using these (at the same file):
role_recipients_slack[sysadmin]="systems" role_recipients_slack[dba]="databases systems" role_recipients_slack[webmaster]="marketing development"
development are slack.com channels (they should already exist in slack).
pushover.net push notifications
pushover.net allows you to receive push notifications on your mobile phone. The service seems free for up to 7.500 messages per month.
netdata will send warning messages with priority
0 and critical messages with priority
1. pushover.net allows you to select do-not-disturb hours. The way this is configured, critical notifications will ring and vibrate your phone, even during the do-not-disturb-hours. All other notifications will be delivered silently.
- APP TOKEN. You can use the same on all your netdata servers.
- USER TOKEN for each user you are going to send notifications to. This is the actual recipient of the notification.
The configuration is like above (slack messages).
pushover.net notifications look like this:
Telegram is a messaging app with a focus on speed and security, it’s super-fast, simple and free. You can use Telegram on all your devices at the same time — your messages sync seamlessly across any number of your phones, tablets or computers.
With Telegram, you can send messages, photos, videos and files of any type (doc, zip, mp3, etc), as well as create groups for up to 5000 people or channels for broadcasting to unlimited audiences. You can write to your phone contacts and find people by their usernames. As a result, Telegram is like SMS and email combined — and can take care of all your personal or business messaging needs.
netdata will send warning messages without vibration.
- A bot token. To get one, contact the @BotFather bot and send the command
/newbot. Follow the instructions.
- A chat id for every chat you want to send messages to. Contact the @myidbot bot and send the command
/getidto get your personal chat id or invite him into a group and issue the same command to get the group chat id.
- Start a conversation with your bot or invite him into a group you want to sent messages to.
See slack for configuration.
Telegram messages look like this:
Twilio SMS notifications
Will look like this on your Android device:
You will need:
- Signup and Login to twilio.com
- Pick an SMS capable number during sign up.
- Get your SID, and Token from https://www.twilio.com/console
- Fill in TWILIO_ACCOUNT_SID="XXXXXXXX" TWILIO_ACCOUNT_TOKEN="XXXXXXXXX" TWILIO_NUMBER="+XXXXXXXXXXX"
- Add the recipient phone numbers to DEFAULT_RECIPIENT_TWILIO="+XXXXXXXXXXX"
!!PLEASE NOTE THAT IF YOUR ACCOUNT IS A TRIAL ACCOUNT YOU WILL ONLY BE ABLE TO SEND NOTIFICATIONS TO THE NUMBER YOU SIGNED UP WITH
Set them in
/etc/netdata/health_alarm_notify.conf, like this:
############################################################################### # Twilio (twilio.com) SMS options # multiple recipients can be given like this: # "+15555555555 +17777777777" # enable/disable sending twilio SMS SEND_TWILIO="YES" # Signup for free trial and select a SMS capable Twilio Number # To get your Account SID and Token, go to https://www.twilio.com/console # Place your sid, token and number below. # Then just set the recipients' phone numbers. # The trial account is only allowed to use the number specified when set up. # Without an account sid and token, netdata cannot send Twilio text messages. TWILIO_ACCOUNT_SID="xxxxxxxxx" TWILIO_ACCOUNT_TOKEN="xxxxxxxxxx" TWILIO_NUMBER="xxxxxxxxxxx" DEFAULT_RECIPIENT_TWILIO="+15555555555"
Messagebird SMS notifications
Will look like this on your Android device:
You will need:
- Signup and Login to messagebird.com
- Pick an SMS capable number after sign up to get some free credits
- Go to https://www.messagebird.com/app/settings/developers/access
- Create a new access key under 'API ACCESS (REST)' (you will want a live key)
- Fill in MESSAGEBIRD_ACCESS_KEY="XXXXXXXX" MESSAGEBIRD_NUMBER="+XXXXXXXXXXX"
- Add the recipient phone numbers to DEFAULT_RECIPIENT_MESSAGEBIRD="+XXXXXXXXXXX"
Set them in
/etc/netdata/health_alarm_notify.conf, like this:
#------------------------------------------------------------------------------ # Messagebird (messagebird.com) SMS options # multiple recipients can be given like this: # "+15555555555 +17777777777" # enable/disable sending messagebird SMS SEND_MESSAGEBIRD="YES" # to get an access key, create a free account at https://www.messagebird.com # verify and activate the account (no CC info needed) # login to your account and enter your phonenumber to get some free credits # to get the API key, click on 'API' in the sidebar, then 'API Access (REST)' # click 'Add access key' and fill in data (you want a live key to send SMS) # Without an access key, netdata cannot send Messagebird text messages. MESSAGEBIRD_ACCESS_KEY="XXXXXXXX" MESSAGEBIRD_NUMBER="XXXXXXX" DEFAULT_RECIPIENT_MESSAGEBIRD="XXXXXXX"
PagerDuty is the enterprise incident resolution service that integrates with ITOps and DevOps monitoring stacks to improve operational reliability and agility. From enriching and aggregating events to correlating them into incidents, PagerDuty streamlines the incident management process by reducing alert noise and resolution times.
Here is an example of a PagerDuty dashboard with netdata notifications:
To have netdata send notifications to PagerDuty, you'll first need to set up a PagerDuty
Generic API service and install the PagerDuty agent on the host running netdata. See the following guide for details:
During the setup of the
Generic API PagerDuty service, you'll obtain a
pagerduty service key. Keep this service key handy.
Once the PagerDuty agent is installed on your host and can send notifications from your host to your
Generic API service on PagerDuty, add the service key to
#------------------------------------------------------------------------------ # pagerduty.com notification options # # pagerduty.com notifications require the pagerduty agent to be installed and # a "Generic API" pagerduty service. # https://www.pagerduty.com/docs/guides/agent-install-guide/ # multiple recipients can be given like this: # "<pd_service_key_1> <pd_service_key_2> ..." # enable/disable sending pagerduty notifications SEND_PD="YES" # if a role's recipients are not configured, a notification will be sent to # the "General API" pagerduty.com service that uses this service key. # (empty = do not send a notification for unconfigured roles): DEFAULT_RECIPIENT_PD="<service key>"
Alarms can have the following statuses:
REMOVED- the alarm has been deleted (this happens when a SIGUSR2 is sent to netdata to reload health configuration)
UNINITIALIZED- the alarm is not initialized yet
UNDEFINED- the alarm failed to be calculated (i.e. the database lookup failed, a division by zero occurred, etc)
CLEAR- the alarm is not armed / raised (i.e. is OK)
WARNING- the warning expression resulted in true or non-zero
CRITICAL- the critical expression resulted in true or non-zero
The external script will be called for all status changes.
http://your.netdata.ip:19999/api/v1/alarms?allreturns all the running/active alarms.
http://your.netdata.ip:19999/api/v1/alarmsreturns all the currently raised alarms.
http://your.netdata.ip:19999/api/v1/alarm_logreturns all the events in the alarm log.
http://your.netdata.ip:19999/api/v1/alarm_log?after=UNIQUEIDreturns all the events in the alarm log that occurred after UNIQUEID (you poll it once without
after=, remember the last UNIQUEID of the returned set, which you give back to get incrementally the next events).
http://your.netdata.ip:19999/api/v1/badge.svg?alarm=NAMEreturns an SVG (XML) of the given alarm NAME.
Edit your netdata.conf and set
debug flags = 0x00800000. Then check your
/var/log/netdata/debug.log. It will show you how it works.
You can find the context of charts by looking up the chart in either
You can find how netdata interpreted the expressions by examining the alarm at
http://your.netdata:19999/api/v1/alarms?all. For each expression, netdata will return the expression as given in its config file, and the same expression with additional parentheses added to indicate the evaluation flow of the expression.