cakrit and ktsaou Updates in various READMEs while testing the link checker (#4671)
* Add checklinks.sh to validate links in MD files and fix them for both the repo and html static site

* Add checklinks.sh to validate links in MD files and fix them for both the repo and html static site

* Minor link fixes, based on output from the linkchecker

* About 50% done

* Converted absolute links to relative

* Close to being functional...

* Getting closer...

* Corrected edit files link from htmldoc branch to master

* Replaced absolute links with relative and fixed incorrect heading hierarchy

* Seems to be working. Will do all wiki conversions and then commit the final version

* Wiki links replaced by the link checker for test

* Replacement work fine, but correct location for simple patterns has moved to libnetdata/simple_pattern

* Corrected wiki links in READMEs, but most are in code files

* More simple pattern link replacements

* Want to merge current changes first and then continue with the link checker

* Add checklinks.sh to validate links in MD files and fix them for both the repo and html static site

* Minor link fixes, based on output from the linkchecker

* About 50% done

* Converted absolute links to relative

* Close to being functional...

* Getting closer...

* Corrected edit files link from htmldoc branch to master

* Replaced absolute links with relative and fixed incorrect heading hierarchy

* Seems to be working. Will do all wiki conversions and then commit the final version

* Wiki links replaced by the link checker for test

* Replacement work fine, but correct location for simple patterns has moved to libnetdata/simple_pattern

* Corrected wiki links in READMEs, but most are in code files

* More simple pattern link replacements

* Want to merge current changes first and then continue with the link checker

* ./# converted to #
Latest commit 7a5335a Nov 16, 2018
Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
adaptec_raid Disable python sudo modules by default (#4477) Oct 25, 2018
apache Fix make dist (#4401) Oct 16, 2018
beanstalk Fix documentation in beanstalk.conf. (#4639) Nov 13, 2018
bind_rndc Fix make dist (#4401) Oct 16, 2018
boinc Fix make dist (#4401) Oct 16, 2018
ceph Fix make dist (#4401) Oct 16, 2018
chrony Fix make dist (#4401) Oct 16, 2018
couchdb Fix make dist (#4401) Oct 16, 2018
cpufreq Fix make dist (#4401) Oct 16, 2018
cpuidle Fix make dist (#4401) Oct 16, 2018
dns_query_time Fix make dist (#4401) Oct 16, 2018
dnsdist Fix make dist (#4401) Oct 16, 2018
dockerd Fix make dist (#4401) Oct 16, 2018
dovecot Fix make dist (#4401) Oct 16, 2018
elasticsearch Bad text in example config (#4618) Nov 12, 2018
example Fix make dist (#4401) Oct 16, 2018
exim Fix make dist (#4401) Oct 16, 2018
fail2ban Fix make dist (#4401) Oct 16, 2018
freeradius Fix make dist (#4401) Oct 16, 2018
go_expvar Updates in various READMEs while testing the link checker (#4671) Nov 16, 2018
haproxy Fix make dist (#4401) Oct 16, 2018
hddtemp Fix make dist (#4401) Oct 16, 2018
httpcheck Fix make dist (#4401) Oct 16, 2018
icecast icecast fix (#4448) Oct 22, 2018
ipfs Fix make dist (#4401) Oct 16, 2018
isc_dhcpd Fix make dist (#4401) Oct 16, 2018
linux_power_supply Fix make dist (#4401) Oct 16, 2018
litespeed Fix make dist (#4401) Oct 16, 2018
logind Fix make dist (#4401) Oct 16, 2018
mdstat Fix make dist (#4401) Oct 16, 2018
megacli Disable python sudo modules by default (#4477) Oct 25, 2018
memcached Fix make dist (#4401) Oct 16, 2018
mongodb mongodb: add min pymongo package version (#4449) Oct 22, 2018
monit Fix make dist (#4401) Oct 16, 2018
mysql Fix make dist (#4401) Oct 16, 2018
nginx Fix make dist (#4401) Oct 16, 2018
nginx_plus Fix make dist (#4401) Oct 16, 2018
nsd Fix make dist (#4401) Oct 16, 2018
ntpd Fix make dist (#4401) Oct 16, 2018
nvidia_smi nvidia_smi: init version added (#4589) Nov 12, 2018
openldap openldap monitoring plugin added (#4513) Nov 15, 2018
ovpn_status_log Fix make dist (#4401) Oct 16, 2018
phpfpm Fix make dist (#4401) Oct 16, 2018
portcheck remove unused variables (#4437) Oct 19, 2018
postfix Fix make dist (#4401) Oct 16, 2018
postgres remove unused variables (#4437) Oct 19, 2018
powerdns Fix make dist (#4401) Oct 16, 2018
proxysql Add ProxySQL python plugin (#4112) Oct 22, 2018
puppet Fix make dist (#4401) Oct 16, 2018
python_modules Update lm_sensors and catch specific errors. (#4667) Nov 15, 2018
rabbitmq Fix make dist (#4401) Oct 16, 2018
redis Fix make dist (#4401) Oct 16, 2018
rethinkdbs Fix make dist (#4401) Oct 16, 2018
retroshare Fix make dist (#4401) Oct 16, 2018
samba Disable python sudo modules by default (#4477) Oct 25, 2018
sensors Update lm_sensors and catch specific errors. (#4667) Nov 15, 2018
smartd_log smartd_log: py2 compatibility fix (#4584) Nov 8, 2018
spigotmc Fix make dist (#4401) Oct 16, 2018
springboot Htmldoc (#4607) Nov 12, 2018
squid Fix make dist (#4401) Oct 16, 2018
tomcat Fix make dist (#4401) Oct 16, 2018
tor add tor python module (#4546) Nov 5, 2018
traefik Fix make dist (#4401) Oct 16, 2018
unbound Fix make dist (#4401) Oct 16, 2018
uwsgi Added uwsgi plugin (#4404) Oct 17, 2018
varnish Fix make dist (#4401) Oct 16, 2018
w1sensor Htmldoc (#4607) Nov 12, 2018
web_log Updates in various READMEs while testing the link checker (#4671) Nov 16, 2018
Makefile.am openldap monitoring plugin added (#4513) Nov 15, 2018
README.md Updates in various READMEs while testing the link checker (#4671) Nov 16, 2018
python.d.conf openldap monitoring plugin added (#4513) Nov 15, 2018
python.d.plugin.in Move cpufreq python module to proc plugin (#4562) Nov 15, 2018

README.md

python.d.plugin

python.d.plugin is a netdata external plugin. It is an orchestrator for data collection modules written in python.

  1. It runs as an independent process ps fax shows it
  2. It is started and stopped automatically by netdata
  3. It communicates with netdata via a unidirectional pipe (sending data to the netdata daemon)
  4. Supports any number of data collection modules
  5. Allows each module to have one or more data collection jobs
  6. Each job is collecting one or more metrics from a single data source

Pull Request Checklist for Python Plugins

This is a generic checklist for submitting a new Python plugin for Netdata. It is by no means comprehensive.

At minimum, to be buildable and testable, the PR needs to include:

  • The module itself, following proper naming conventions: python.d/<module_dir>/<module_name>.chart.py
  • A README.md file for the plugin under python.d/<module_dir>.
  • The configuration file for the module: conf.d/python.d/<module_name>.conf. Python config files are in YAML format, and should include comments describing what options are present. The instructions are also needed in the configuration section of the README.md
  • A basic configuration for the plugin in the appropriate global config file: conf.d/python.d.conf, which is also in YAML format. Either add a line that reads # <module_name>: yes if the module is to be enabled by default, or one that reads <module_name>: no if it is to be disabled by default.
  • A line for the plugin in python.d/Makefile.am under dist_python_DATA.
  • A line for the plugin configuration file in conf.d/Makefile.am, under dist_pythonconfig_DATA
  • Optionally, chart information in web/dashboard_info.js. This generally involves specifying a name and icon for the section, and may include descriptions for the section or individual charts.

Disclaimer

Every module should be compatible with python2 and python3. All third party libraries should be installed system-wide or in python_modules directory. Module configurations are written in YAML and pyYAML is required.

Every configuration file must have one of two formats:

  • Configuration for only one job:
update_every : 2 # update frequency
retries      : 1 # how many failures in update() is tolerated
priority     : 20000 # where it is shown on dashboard

other_var1   : bla  # variables passed to module
other_var2   : alb
  • Configuration for many jobs (ex. mysql):
# module defaults:
update_every : 2
retries      : 1
priority     : 20000

local:  # job name
  update_every : 5 # job update frequency
  other_var1   : some_val # module specific variable

other_job:
  priority     : 5 # job position on dashboard
  retries      : 20 # job retries
  other_var2   : val # module specific variable

update_every, retries, and priority are always optional.


How to write a new module

Writing new python module is simple. You just need to remember to include 5 major things:

  • ORDER global list
  • CHART global dictionary
  • Service class
  • _get_data method
  • all code needs to be compatible with Python 2 (≥ 2.7) and 3 (≥ 3.1)

If you plan to submit the module in a PR, make sure and go through the PR checklist for new modules beforehand to make sure you have updated all the files you need to.

Global variables ORDER and CHART

ORDER list should contain the order of chart ids. Example:

ORDER = ['first_chart', 'second_chart', 'third_chart']

CHART dictionary is a little bit trickier. It should contain the chart definition in following format:

CHART = {
    id: {
        'options': [name, title, units, family, context, charttype],
        'lines': [
            [unique_dimension_name, name, algorithm, multiplier, divisor]
        ]}

All names are better explained in the External Plugins section. Parameters like priority and update_every are handled by python.d.plugin.

Service class

Every module needs to implement its own Service class. This class should inherit from one of the framework classes:

  • SimpleService
  • UrlService
  • SocketService
  • LogService
  • ExecutableService

Also it needs to invoke the parent class constructor in a specific way as well as assign global variables to class variables.

Simple example:

from base import UrlService
class Service(UrlService):
    def __init__(self, configuration=None, name=None):
        UrlService.__init__(self, configuration=configuration, name=name)
        self.order = ORDER
        self.definitions = CHARTS

_get_data collector/parser

This method should grab raw data from _get_raw_data, parse it, and return a dictionary where keys are unique dimension names or None if no data is collected.

Example:

def _get_data(self):
    try:
        raw = self._get_raw_data().split(" ")
        return {'active': int(raw[2])}
    except (ValueError, AttributeError):
        return None

More about framework classes

Every framework class has some user-configurable variables which are specific to this particular class. Those variables should have default values initialized in the child class constructor.

If module needs some additional user-configurable variable, it can be accessed from the self.configuration list and assigned in constructor or custom check method. Example:

def __init__(self, configuration=None, name=None):
    UrlService.__init__(self, configuration=configuration, name=name)
    try:
        self.baseurl = str(self.configuration['baseurl'])
    except (KeyError, TypeError):
        self.baseurl = "http://localhost:5001"

Classes implement _get_raw_data which should be used to grab raw data. This method usually returns a list of strings.

SimpleService

This is last resort class, if a new module cannot be written by using other framework class this one can be used.

Example: mysql, sensors

It is the lowest-level class which implements most of module logic, like:

  • threading
  • handling run times
  • chart formatting
  • logging
  • chart creation and updating

LogService

Examples: apache_cache, nginx_log

Variable from config file: log_path.

Object created from this class reads new lines from file specified in log_path variable. It will check if file exists and is readable. Also _get_raw_data returns list of strings where each string is one line from file specified in log_path.

ExecutableService

Examples: exim, postfix

Variable from config file: command.

This allows to execute a shell command in a secure way. It will check for invalid characters in command variable and won't proceed if there is one of:

  • '&'
  • '|'
  • ';'
  • '>'
  • '<'

For additional security it uses python subprocess.Popen (without shell=True option) to execute command. Command can be specified with absolute or relative name. When using relative name, it will try to find command in PATH environment variable as well as in /sbin and /usr/sbin.

_get_raw_data returns list of decoded lines returned by command.

UrlService

Examples: apache, nginx, tomcat

Variables from config file: url, user, pass.

If data is grabbed by accessing service via HTTP protocol, this class can be used. It can handle HTTP Basic Auth when specified with user and pass credentials.

_get_raw_data returns list of utf-8 decoded strings (lines).

SocketService

Examples: dovecot, redis

Variables from config file: unix_socket, host, port, request.

Object will try execute request using either unix_socket or TCP/IP socket with combination of host and port. This can access unix sockets with SOCK_STREAM or SOCK_DGRAM protocols and TCP/IP sockets in version 4 and 6 with SOCK_STREAM setting.

Sockets are accessed in non-blocking mode with 15 second timeout.

After every execution of _get_raw_data socket is closed, to prevent this module needs to set _keep_alive variable to True and implement custom _check_raw_data method.

_check_raw_data should take raw data and return True if all data is received otherwise it should return False. Also it should do it in fast and efficient way.