health_alarm_notify Being Overwritten

[Salad360]

Bug report summary

OS / Environment

Ubuntu Server 18.04 / Raspbian 9.8

Netdata version (ouput of netdata -V)

1.13.0-8-nightly

Component Name

Steps To Reproduce

Install Netdata, edit notification file and let is sit two weeks

Expected behavior

Not having the file revert to default settings.

[Salad360]

I run netdata on two servers running Ubuntu Server 18.04 and one Raspberry Pi running raspbian 9.8. After not receiving one notification from any of these devices for over two weeks I ran the alarm-notify test script to see what was going on only to discover that notifications weren't getting out because the titular file got reverted to default, presumably due to an update.

This is really annoying, how can I avoid having this happen?

[cakrit]

You should use /etc/netdata/edit-config, or copy the stock config file to /etc/netdata first and then edit it.

We keep seeing these types of issues and I'm sure we can do better here. @ktsaou, @paulkatsoulakis and @Ferroin, how about making the stock config files read only? Since the installer, updater and uninstaller all run as root, we shouldn't have a problem replacing them anyway.

[nyetwurk]

Is there a way to only include a subset of the items in an /etc/netdata config that overrides default, or does the entire config file have to be copied?

e.g. create a file /etc/netdata/health.d/local.conf
that contains only

   alarm:  1m_tcp_syn_queue_drops
   warn: $this > 1

or even

   alarm:  1m_tcp_syn_queue_drops
   warn: 

or

   alarm:  1m_tcp_syn_queue_drops
   to: silent

to disable the warning entirely?

This way, if upstream changes the defaults, copying the file around and modifying it AGAIN isn't necessary, and all upstream defaults follow whatever happens upstream, and only the local customizations get applied.

This problem has been solved in many other software packages; namely, they have defaults built in, and a place to put files that only override behavior/settings, without having to completely copy/paste the original settings. In a normal system daemon (e.g. not containerized), the configuration IS read only, both in /etc and in /usr/lib or what have you. Defaults are in /usr/lib (which can be safely replaced by an installer) and root adds customizations in /etc, which DO NOT GET TOUCHED on install/upgrade/etc.

I don't pretend to understand why netdata takes the approach it does; it seems naive and unaware of normal best practices about keeping upstream defaults distinct from local customizations, unless I'm missing something.

[cakrit]

You're right @nyetwurk and it is something that we'd like to change, but it won't be done with small patches. We have discussed the needs of proper, modern configuration management processes, so it's one of the big initiatives on our roadmap. We just haven't prioritized it yet, based on user feedback (our survey).

[cakrit]

Closing in favor of #6432