security concern one should take into account ?

[abdulirfan3]

Hello,

fyi ... this is more of a question than issue/bug report..

I have to say, netdata is one of the awesome system monitoring tool I have ever seen. It definitely beat a lot of enterprise grade monitoring and visualization tools. I have been playing around it for couple of days here but wanted to ask what are some of the security concerns around this tool ?? I see there are already couple of issues opened with regards to authentication(which we can run behind nginx as workaround) but besides authentication if bad guy is able to get to that data what type of damage can they do ??

I understand netdata is very light weight and should be able to serve 300+ web client requests per second per core without any delays but cant this part also be exploited as well? I am just trying to see what one need to do from security standpoint if we were to mass deploy this tool in our environment.

[ktsaou]

EDIT: the wiki page https://github.com/firehol/netdata/wiki/netdata-security replaces the information contained in this issue.

Nice you like netdata. Regarding your security concerns:

Should netdata be protected from the internet?

Of course. netdata is an internal tool you use to monitor the performance of your systems. It should be protected, the same way you protect all your admin apps.

We assume netdata will be installed inside your DMZ, privately, for your eyes only. And that your DMZ is protected and controlled enough to allow netdata focus on its core function.

We have also designed the plugins of netdata (a few of which might be running with admin privileges - like apps.plugin) in such a way that they cannot be instructed from the netdata server. They execute a hardcoded data collection job and they only send statistics to the netdata server, never receiving commands the other way around (unlike most NMSes that are capable of sending commands to the agents installed on all systems).

What will happen if you accidentally expose netdata to the internet?

1. The viewers will be able to get some information about the system netdata is running. This information is everything the dashboard provides. The dashboard includes a list of the applications you run (the legends of the charts in the Application section), the disks of the system and their names, the network interfaces and their names (not the IPs) and detailed information about the performance of the system. This information is not sensitive (meaning that it is not your business data), but is important for possible attackers. It will give them clues on what to check, what to try and in the case of DDoS against your applications, they will know if they are doing it right or not.
2. The viewers could use netdata itself to stress your server. Although netdata runs unprivileged and with the minimum process priority, some harm can be done if someone attempts a DDoS against netdata. Given the low priority of netdata, the linux kernel will most probably prefer to kill netdata in cases it starves for resources. This is however a case that should be avoided (the kernel has a complex algorithm for electing the victim process that will be killed, which includes its memory footprint - this means the kernel may elect to kill your database server, or your web server).

Why we haven't already added user access controls on netdata?

Well, the last few months and until this Friday we had... well... 100 users ! (probably I am extravagating here).

The last 3 days there have been 10.000 git clones!

So, yes! Please protect your netdata. If you can protect it using firewalls (layer 3) and provide the user access control you need utilizing other web servers (check the wiki for info), use it.

Otherwise, wait. We will add user access control soon.

[abdulirfan3]

Great summary, Thank you.

For now we will put this behind nginx and wait for a new update that will add access control.

[ktsaou]

For now we will put this behind nginx and wait for a new update that will add access control.

I just pushed this change to help you when you proxy netdata through another web server:

limit direct access to netdata

You would also need to instruct netdata to listen only to 127.0.0.1 or ::1.

To limit access to netdata only from localhost, set bind socket to IP = 127.0.0.1 or bind socket to IP = ::1 in /etc/netdata/netdata.conf.

[cbricart]

what about making the loopback bind the default and let the user override this if she wants/needs to..?
would be more a security first/per default approach..

[abdulirfan3]

@ktsaou

Thank you for making those suggestion, I was able to get couple of server behind nginx with some basic authentication. What I am struggling with is limiting direct access to netdata, I have the below config as you posted but I can access thru nginx and also directly thru servername:19999. Am I doing something wrong here ?

servername:# grep -i bind /etc/netdata/netdata.conf
bind socket to IP = 127.0.0.1
servername:#

[ktsaou]

Have you updated your netdata?
When I posted I just committed the patch that implements this.
You can also go to http://your.server:19999/netdata.conf - if your netdata is old bind socket to IP should say it is unused.

[abdulirfan3]

oh i guess that would help.

QQ, how do you update netdata. Is that just a straight removing of all the netdata file/directories and reinstall or is there a doc for updating ?

[ktsaou]

Just go to the directory you downloaded it, run git pull and install it again. The Installation page of wiki has this info.

[ktsaou]

or just re-install it using the same installer parameters.

[abdulirfan3]

just pulled the new changes and added the "bind socket to IP" and then restarted netdata, I can still access directly... am i missing something ?

`servername:~/netdata.git# head /etc/netdata/netdata.conf

NetData Configuration

You can uncomment and change any of the options below.

The value shown in the commented settings, is the default value.

global netdata configuration

bind socket to IP = 127.0.0.1

[global]
# config directory = /etc/netdata
# plugins directory = /usr/libexec/netdata/plugins.d
servername:~/netdata.git#`

[abdulirfan3]

Here is what i have done.... added "bind socket" to global config, stoped and started netdata...now when i go to "servername:19999" i get site cant be reached but also not able to see the data from ngnix server either...

servername:~# head -30 /etc/netdata/netdata.conf

NetData Configuration

You can uncomment and change any of the options below.

The value shown in the commented settings, is the default value.

global netdata configuration

[global]
# config directory = /etc/netdata
# plugins directory = /usr/libexec/netdata/plugins.d
# web files directory = /usr/share/netdata/web
# cache directory = /var/cache/netdata
# log directory = /var/log/netdata
# host access prefix =
# debug flags = 0x00000000
# memory deduplication (ksm) = yes
# debug log = /var/log/netdata/debug.log
# error log = /var/log/netdata/error.log
# access log = /var/log/netdata/access.log
# memory mode = save
# hostname = servername
# history = 3600
# update every = 1
# run as user = netdata
# web files owner = netdata
# http port listen backlog = 100
# port = 19999
# ip version = any
# disconnect idle web clients after seconds = 60
# enable web responses gzip compression = yes
bind socket to IP = 127.0.0.1

servername:# killall netdata
servername:#
servername:# netdata
16-04-05 11:12:43: ERROR: netdata: Failed to convert IP '127.0.0.1' to a valid IPv6 address. Disabling IPv6.
16-04-05 11:12:43: INFO: netdata: Listening on IP '127.0.0.1' (IPv4).
Managed to open an IPv4 socket on port 19999.servername:#
servername:~#

[ktsaou]

Your netdata is listening on 127.0.0.1. Check it:

netstat -nat | grep 19999

Then check your nginx config. The backend server should be set to 127.0.0.1:19999. Your nginx might be trying to connect to IPv6 on port 19999. In that case either set 127.0.0.1:19999 to nginx or set bind to IP = ::1 to netdata.

[abdulirfan3]

still the same thing... direct access get "site not available" accessing thru nginx i get "502 Bad Gateway"..here are the details...

servername:/tmp/netdata# netstat -nat | grep 19999
tcp 0 0 127.0.0.1:19999 0.0.0.0:* LISTEN
servername:/tmp/netdata#

here is the backend on nginx server...
upstream backend-servername {
server 127.0.0.1:19999;
keepalive 64;
}

also here is my full nginx server config file

ngixservername:~# cat /etc/nginx/nginx.conf

user nginx;
worker_processes 1;

error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
worker_connections 1024;
}

http {
include /etc/nginx/mime.types;
default_type application/octet-stream;

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

access_log  /var/log/nginx/access.log  main;

sendfile        on;
#tcp_nopush     on;

keepalive_timeout  65;

#gzip  on;

include /etc/nginx/conf.d/*.conf;

upstream backend-servername {
server 127.0.0.1:19999;
keepalive 64;
}
server {
listen IP OF ngixservername;
server_name IP OF ngixservername;
auth_basic "Protected";
auth_basic_user_file passwords;

location ~ /netdata/(?<behost>.*)/(?<ndpath>.*) {
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_pass http://backend-$behost/$ndpath$is_args$args;
    proxy_http_version 1.1;
    proxy_pass_request_headers on;
    proxy_set_header Connection "keep-alive";
    proxy_store off;
}

}
}

[ktsaou]

Can you also post what nginx logs when you attempt to access netdata?

[abdulirfan3]

access.log
10.30.105.26 - admin [06/Apr/2016:03:36:22 +1000] "GET /netdata/servername/ HTTP/1.1" 502 574 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36" "-"

error.log
2016/04/06 03:36:54 [error] 4375#0: *17 connect() failed (111: Connection refused) while connecting to upstream, client: X.X.X.26, server: X.X.X.12, request: "GET /netdata/servername/ HTTP/1.1", upstream: "http://127.0.0.1:19999/", host: "usaws604"

X.X.X.12 = NGINX server
X.X.X.26 = my laptop IP

[ktsaou]

strange. from your nginx host if you do:

telnet 127.0.0.1 19999

do you get this:

Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

(press control-] and then q to exit this)

[abdulirfan3]

ngixservername:/var/log/nginx# telnet 127.0.0.1 19999
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
ngixservername:/var/log/nginx#

[ktsaou]

is netdata running on this host?

netstat -nat | grep 19999

[abdulirfan3]

no its not, do we need to run netdata on the nginx server also?

[abdulirfan3]

i just installed it on the nginx server as well...

ngixservername:/tmp/netdata# netstat -nat | grep 19999
tcp 0 0 127.0.0.1:29347 127.0.0.1:19999 TIME_WAIT
tcp 0 0 :::19999 :::* LISTEN
ngixservername:/tmp/netdata#

..here is the telnet afterwards...
usaws604:/tmp/netdata# telnet 127.0.0.1 19999
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.

[ktsaou]

Look, 127.0.0.1 is localhost is the machine of the nginx server. The nginx configuration you used, allows you to proxy through it any number of netdata servers. But in this case, your netdata servers cannot be listening on localhost. Somehow, nginx has to talk to them.

So, use 127.0.0.1 for netdata only when nginx and netdata are on the same host. Otherwise, you don't need this, but your firewall should restrict direct access from your PCs to your netdata servers.

[abdulirfan3]

oh thanks for the explanation...that did the trick now i am able to run multiple netdata server behind one nginx box....

time to roll this out into our image ...

Thanks for your help !!!!

[abdulirfan3]

@ktsaou

i think i might have something mis-configured. when i go to http://ngnixserver/netdata/servername/

i still get data for ngnixserver and not for servername....obviously when i do http://ngnixserver/netdata/ngnixserver/ i get to nginxserver but its when i go to servername i get redirected to nginxserver....

[ktsaou]

You need to define both backend- servers. Have you?

[abdulirfan3]

yeah i have both backend server in /etc/nginx/nginx.conf file

upstream backend-nginxhostname1 {
server 127.0.0.1:19999;
keepalive 64;
}

upstream backend-hostname1 {
server 127.0.0.1:19999;
keepalive 64;
}

upstream backend-hostname2 {
server 127.0.0.1:19999;
keepalive 64;
}

[ktsaou]

All backend servers point to 127.0.0.1:19999, i.e. the same netdata, which is running on the same host with nginx.

[abdulirfan3]

that is right, what i am trying to do is. for all the backend server(hostname1, hostname2 ...) I have disable direct access (by using "bind socket to IP = 127.0.0.1") so that if someone tries to go to http://hostname1:19999 they get site not avaiable message but when someone goes to http:///nginxhostname1/netdata/hostname1 I can access it from nginxhost.

What should this be pointing to?? I tried putting the IP address of hostname1 for the backend server but I get "502 Bad Gateway" message when I try to go to nginxhostname1/netdata/hostname1

upstream backend-hostname1 {
server IP_OF_HOSTNAME1:19999;
keepalive 64;
}

[abdulirfan3]

@ktsaou

any pointers on this, as to how to block direct access and only enable access thru nginx ??

[philwhineray]

If you listen on 127.0.0.1 you can only communicate with that machine, so only an nginx on the same server will be able to talk to the netdata on that server. To do what you want you must listen on something other than 127.0.0.1 for netdata which means by default any machine can access it. This is necessary because there is nothing special about nginx when it is coming from another host; either everything and everyone gets remote access or no-one and nothing. So what can you do beyond this? You can set up firewall rules on the netdata servers so only incoming requests from the nginx server are permitted on the netdata port. Doing so is probably beyond the scope of netdata help though, since it depends on what software you use to configure your firewalls normally.

[abdulirfan3]

okay, Thank you for the information. I we will look into this from iptables standpoint or just wait until the next release or so when you guys have some kind of authentication setup...

[ktsaou]

cleaning up. If something remains, please open it again.

[ktsaou]

Just for completeness, PR #775 (apart from health monitoring - alarms) adds 2 important features related to security:

1. netdata now runs with the IDLE scheduler. This is lower than nice 19 and makes netdata use only IDLE CPU resources.
2. netdata instructs the out-of-memory kernel handler to kill it first in case it starves for resources. So, if a server is needing memory, netdata will go out first.

Since I wrote this comment, a lot more have changed, netdata is a lot faster, a lot more stable with a lot more features (check the registry, the python plugins thanks to @paulfantom, the badges, to name a few), but we still don't have authentication for the API.

The reason is simple: I am not sure if this is the way to go. As netdata matures, I see a lot of benefits to have a more complex access control. Netdata provides a lot of information that can be useful to humans with different roles, but also third party apps. Having a simple username / password to access everything, I am afraid will not suffice. I am still researching alternatives...

[LucaDev]

If someone really needs authentification he could just use an reverse proxy and basic-authentification (in the case of nginx - not sure about apache)

[ktsaou]

btw, the wiki page https://github.com/firehol/netdata/wiki/netdata-security replaces the information contained in this issue.

[netdata-community-bot]

This issue has been mentioned on the Netdata Community Forums. There might be relevant details there:

https://community.netdata.cloud/t/go-d-snmp-collector-either-not-collecting-or-not-publishing/4026/1