high performance netdata

Costa Tsaousis edited this page Jul 7, 2016 · 6 revisions

General


Running Netdata


Alarms

Backends


Netdata Registry


Monitoring Info


Netdata Badges


Data Collection

Binary Modules

Python Modules

Node.js Modules

BASH Modules


API Documentation


Web Dashboards


Running behind another web server

Advanced configurations


Donations


Blog


Other monitoring tools

Clone this wiki locally

If you plan to run a netdata public on the internet, you will get the most performance out of it by following these rules:

1. run behind nginx

The internal web server is optimized to provide the best experience with few clients connected to it. Normally a web browser will make 4-6 concurrent connections to a web server, so that it can send requests in parallel. To best serve a single client, netdata spawns a thread for each connection it receives (so 4-6 threads per connected web browser).

If you plan to have your netdata public on the internet, this strategy wastes resources. It provides a lock-free environment so each thread is autonomous to serve the browser, but it does not scale well. Running netdata behind nginx, idle connections to netdata can be reused, thus improving significantly the performance of netdata.

In the following nginx configuration we do the following:

  • allow nginx to maintain up to 1024 idle connections to netdata (so netdata will have up to 1024 threads waiting for requests)

  • allow nginx to compress the responses of netdata (later we will disable gzip compression at netdata)

  • we disable wordpress pingback attacks and allow only GET, HEAD and OPTIONS requests.

upstream backend {
    server 127.0.0.1:19999;
    keepalive 1024;
}

server {
    listen *:80;
    server_name my.web.server.name;

    location / {
        proxy_set_header X-Forwarded-Host $host;
        proxy_set_header X-Forwarded-Server $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_pass_request_headers on;
        proxy_set_header Connection "keep-alive";
        proxy_store off;
        gzip on;
        gzip_proxied any;
        gzip_types *;
    }

    # WordPress Pingback Request Denial
    if ($http_user_agent ~* "WordPress") {
        return 403;
    }

    if ($request_method !~ ^(GET|HEAD|OPTIONS)$ ) {
        return 403;
    }
}

Then edit /etc/netdata/netdata.conf and set these config options:

[global]
    bind socket to IP = 127.0.0.1
    access log = none
    disconnect idle web clients after seconds = 3600
    enable web responses gzip compression = no

These options:

  • [global].bind socket to IP = 127.0.0.1 makes netdata listen only for requests from localhost (nginx).
  • [global].access log = none disables the access.log of netdata. It is not needed since netdata only listens for requests on 127.0.0.1 and thus only nginx can access it. nginx has its own access.log for your record.
  • [global].disconnect idle web clients after seconds = 3600 will kill inactive web threads after an hour of inactivity.
  • [global].enable web responses gzip compression = no disables gzip compression at netdata (nginx will compress the responses).

2. open files limits (non-systemd)

By default Linux limits open file descriptors per process to 1024. This means that less than half of this number of client connections can be accepted by both nginx and netdata. To increase them, create 2 new files:

  1. /etc/security/limits.d/nginx.conf, with these contents:

    nginx   soft    nofile  10000
    nginx   hard    nofile  30000
    
  2. /etc/security/limits.d/netdata.conf, with these contents:

    netdata   soft    nofile  10000
    netdata   hard    nofile  30000
    

and to activate them, run:

sysctl -p

2b. limit open files (systemd)

Thanks to @leleobhz, this is what you need to raise the limits using systemd:

This is based on https://ma.ttias.be/increase-open-files-limit-in-mariadb-on-centos-7-with-systemd/ and here worked as following:

  1. Create the folders in /etc:

    mkdir -p /etc/systemd/system/netdata.service.d
    mkdir -p /etc/systemd/system/nginx.service.d
    
  2. Create limits.conf in each folder as following:

    [Service]
    LimitNOFILE=30000
    
  3. Reload systemd daemon list and restart services:

    systemctl daemon-reload
    systemctl restart netdata.service
    systemctl restart nginx.service

You can check limits with following commands:

cat /proc/$(ps aux | grep "nginx: master process" | grep -v grep | awk '{print $2}')/limits | grep "Max open files"
cat /proc/$(ps aux | grep "netdata" | head -n1 | grep -v grep | awk '{print $2}')/limits | grep "Max open files"

View of the files:

# tree /etc/systemd/system/*service.d/etc/systemd/system/netdata.service.d
/etc/systemd/system/netdata.service.d
└── limits.conf
/etc/systemd/system/nginx.service.d
└── limits.conf

0 directories, 2 files

# cat /proc/$(ps aux | grep "nginx: master process" | grep -v grep | awk '{print $2}')/limits | grep "Max open files"
Max open files            30000                30000                files     

# cat /proc/$(ps aux | grep "netdata" | head -n1 | grep -v grep | awk '{print $2}')/limits | grep "Max open files"
Max open files            30000                30000                files