modularize c source

[ktsaou]

netdata daemon is modular. However the structure of the source code does not promote this modularity.

To modularize the source code, we need to make significant changes to automake and probably move code around. So, I suggest to make these changes once we move to cmake.

The C source currently has these plugins:

module	O/S	runtime	type	description
apps.plugin	linux freebsd	process	data collection	collects metrics from the process tree. It is an independent process because it needs to run with specific capabilities (or as root).
freeipmi.plugin	linux	process	data collection	collects enterprise hardware sensors. It is an independent process because it needs to run as root.
proc.plugin	linux	thread	data collection	collects system metrics on linux (/proc, /sys , like ipc). This is also modular by itself.
diskspace.plugin	linux	thread	data collection	collects disk space metrics on linux (it is a separate plugin because it requires to be run on its own thread due to the latency of collecting disk space utilization on linux)
freebsd.plugin	freebsd	thread	data collection	collects system metrics on freebsd. This is also modular by itself.
macos.plugin	macos	thread	data collection	collects system metrics on macos. This is also modular by itself.
cgroups.plugin	linux	thread	data collection	collects metrics about containers and VMs. It uses 2 helpers: cgroup-network and cgroup-name.
nfacct.plugin	linux	thread	data collection	collects netfilter metrics (accounting and connection tracker) using ibmnl. Requires root access, so it should be converted to a process.
pluginsd.plugin	all	a thread per source	data collection	collects metrics from external plugins and receiving streamed metrics. It spawns a thread for each external plugin found and each remote netdata streaming metrics.
statsd.plugin	all	2 threads	data collection	collects and processes statsd metrics. One thread is the data collector listening on the statsd ports (multiple tcp and udp ports concurrently). Another thread pushes the metrics to netdata (implementing also statsd synthetic charts).
checks.plugin	all	thread	data collection	evaluation of alternative techniques in data collection
idlejitter.plugin	all	thread	data collection	collects metrics related to the CPU latency.
tc.plugin	linux	thread with an external process	data collection	collects traffic QoS metrics on linux, using tc-qos-helper. The external tc-qos-helper process runs tc to export the current statistics of QoS on all interfaces and the internal thread parses the output of tc and pushes the metrics to netdata.
backends.plugin	all	thread	backends	The plugin for archiving metrics to time series dbs (graphite., opentsdb, json in 2 forms each). This is modular by itself (so each backend is one of its modules).
allmetrics.plugin	all	inline	api	This plugin exports all metrics in various formats. It has 3 modules: shell, prometheus and prometheus_all_hosts.
registry.plugin	all	inline	api	Implements the netdata registry.
rrdpush.plugin	all	thread	streaming	Streams all metrics to another netdata (they are received by pluginsd.plugin)
health.plugin	all	thread	alarms	Evaluates alarms and sends notifications with alarm-notify.

[paulfantom]

I suggest moving and doing cmake at the same time. And doing this on a feature branch, so we will still have working master

[ktsaou]

I have to think a bit about it.

For example:

1. a solution would be to move the files around, but still require each plugin and each module to be registered at its parent by hand (modifying the source code of the parent - normally adding an entry to an array - to add a plugin or module). This is the easiest.

2. use compiled libraries XXX.a to build all the modules and have some kind of registration code to hook it at the right place at compile time.

3. use shared libraries XXX.so to build all the modules and allow them to dynamically be hooked to the daemon at runtime.

The easiest is the 1st.
I think the 3rd is not an option, because we need netdata to be compiled statically under conditions.
The "cleaner" is the 2nd, but requires some work to develop all these hooks.

[paulfantom]

Relates to #4310