Docker and Apps plugin #43

titpetric · 2016-01-11T00:10:51Z

The Apps section of netdata doesn't give correct output when run with docker. Only "netdata" process is graphed. The process is run with -ch /host, data is mapped to container:

root@071d0f31f366:/host/proc# awk 'FNR==1{print ""}1' */cmdline | grep ^/
/sbin/init
/usr/sbin/dovecot
/lib/systemd/systemd-journald
/lib/systemd/systemd-udevd
/usr/lib/postfix/master
/bin/sh
/usr/sbin/mysqld
/usr/sbin/sshd
/usr/bin/redis-server 127.0.0.1:6379
/bin/bash
/usr/sbin/cron
/bin/bash
/usr/sbin/rsyslogd
/bin/bash
/usr/local/usr/sbin/netdata
/usr/local/usr/libexec/netdata/plugins.d/apps.plugin
/bin/bash
/sbin/rpcbind
/sbin/rpc.statd
/usr/sbin/rpc.idmapd
/usr/sbin/atd
/lib/systemd/systemd-logind
/usr/bin/dbus-daemon
/usr/sbin/acpid
/sbin/agetty
/root/.dropbox-dist/dropbox-lnx.x86_64-3.12.5/dropbox

It seems the basic problem is that apps.plugin doesn't support -ch prefix yet?
Opening apps_plugin.c confirms this:

// 1. support -ch option to set global /proc and /sys prefix

The text was updated successfully, but these errors were encountered:

ktsaou · 2016-01-11T07:00:39Z

check it.
Netdata passes it to apps.plugin. No configuration needed.

titpetric · 2016-01-11T10:45:43Z

Can't confirm that it works. I've attached some things which might be relevant:

run.sh /run.sh
  |-netdata -ch /host
  |   |-apps.plugin 1
  |   |-charts.d.plugin /usr/local/usr/libexec/netdata/plugins.d/charts.d.plugin 1
  |   |   `-sleep 0.484
  |   |-tc-qos-helper.s /usr/local/usr/libexec/netdata/plugins.d/tc-qos-helper.sh 1
  |   |   `-sleep 0.997
  |   `-7*[{netdata}]
  `-sleep infinity

ps auxw doesn-t show -ch being passed to apps.plugin (this might not be needed?)

root@57632d230424:/netdata.git# git log -n 1
WARNING: terminal is not fully functional
commit 50c97c72c55ae211ad43f5cc7debf194e154426f
Author: Costa Tsaousis (ktsaou) <costa@tsaousis.gr>
Date:   Mon Jan 11 09:00:07 2016 +0200

    apps.plugin inherits host prefix from netdata #43

Last commit with host prefix was included in build.
You can see the dashboard from the latest docker in action on http://cdn.si:19999/

Best,
Tit

ktsaou · 2016-01-11T12:36:06Z

There is no -ch option for apps.plugin. Netdata passes the environment variable NETDATA_HOST_PREFIX to its childs, which apps.plugin reads.

Your apps.plugin is not setuid to root, this is why it can only see netdata itself. Do this:

chown root:root /usr/libexec/netdata/plugins.d/apps.plugin
chmod 4755 /usr/libexec/netdata/plugins.d/apps.plugin

and restart netdata.

ktsaou · 2016-01-11T12:38:53Z

I corrected my response.

titpetric · 2016-01-11T15:03:25Z

root@57632d230424:/netdata.git# ls -la /usr/local/usr/libexec/netdata/plugins.d/apps.plugin
-rwsr-xr-x 1 root root 72320 Jan 11 10:13 /usr/local/usr/libexec/netdata/plugins.d/apps.plugin

Pretty sure it's fine?

titpetric · 2016-01-11T15:13:05Z

root@57632d230424:/netdata.git# ls -la /usr/local/usr/libexec/netdata/plugins.d/apps.plugin
-rwsr-xr-x 1 root root 72320 Jan 11 10:13 /usr/local/usr/libexec/netdata/plugins.d/apps.plugin
root@57632d230424:/netdata.git#
root@57632d230424:/netdata.git# chown root:root /usr/local/usr/libexec/netdata/plugins.d/apps.plugin
root@57632d230424:/netdata.git# chmod 4755 /usr/local/usr/libexec/netdata/plugins.d/apps.plugin
root@57632d230424:/netdata.git# ls -la /usr/local/usr/libexec/netdata/plugins.d/apps.plugin
-rwsr-xr-x 1 root root 72320 Jan 11 10:13 /usr/local/usr/libexec/netdata/plugins.d/apps.plugin
root@57632d230424:/netdata.git#

Seems it is/was fine. Lines are in here if you want to verify:
https://github.com/titpetric/netdata/blob/master/build.sh#L52

Also from what i understand, apps.plugin should run as root, and it does:

root     14347  0.4  0.3   6824  1856 ?        SN   14:38   0:08 /usr/local/usr/libexec/netdata/plugins.d/apps.plugin 1

titpetric · 2016-01-11T15:19:43Z

It seems it might be a docker issue, investigating.

titpetric · 2016-01-11T15:22:36Z

Yes, running docker container has to have --cap-add SYS_PTRACE option, to enable access to proc. I figured it out as I couldn't dump /proc/[pid]/environment in the container as root. Seems to be solved, I'll update the README on my side. :)

titpetric · 2016-01-11T15:23:45Z

If you can, please open http://cdn.si:19999 if you see anything out of place (missing data/charts, something that could be caused by the isolated docker environment). If everything looks good, let's close this issue :) Sorry for so many comments.

titpetric · 2016-01-11T15:32:06Z

regarding SYS_PTRACE:

Without the SYS_PTRACE capability, the environment variables don't get passed from the netdata daemon, to the apps.plugin and other plugins. In this case it means that apps.plugin is reading info from /proc location, instead of the mapped volume /host/proc.

Can we pass this as an argument, or as part of the exec call itself (AAA=xxx ./apps.plugin?). I know it's not ideal, but it would drop the need for --cap-add SYS_PTRACE requirement.

ktsaou · 2016-01-11T17:56:27Z

Where do you add SYS_PTRACE? Is it something I can add to netdata?

titpetric · 2016-01-11T18:55:28Z

No, SYS_PTRACE is a capability that needs to be enabled on the docker
container.

Without it, I couldn't print the environment from a process (even as root).
I suspected that it also affects the way environment is passed between
netdata and apps plugin. I found this issue by googling a bit, read a bunch
of comments, and decided to try to enable SYS_PTRACE, just to dump the
environment vars by PID. After running the docker image, I checked the apps
output & voila, issue resolved.

But, I think netdata can be modified slightly, so this capability is not
needed. In pseudo code, something like this should work: system("ENV=value
./apps.plugin"); - This way the environment is not inherited, but
explicitly defined when the plugin is spawned (fork-ed, or however it's
done). This is only feasible if there's not many of such variables - and it
seems there are not many?

On Mon, Jan 11, 2016 at 6:56 PM, Costa Tsaousis notifications@github.com
wrote:

Where do you add SYS_PTRACE? Is it something I can add to netdata?

—
Reply to this email directly or view it on GitHub
#43 (comment).

ktsaou · 2016-01-11T22:58:01Z

Well, this does not sound right to me.

Environment variables between the same process tree in the same docker should not be affected by anything else.

I have added a log line.
Start netdata and then do this:

 # tail -n 1000 /var/log/netdata/error.log  | grep apps.plugin

Mine logs this:

16-01-12 00:55:20: INFO: apps.plugin: Found NETDATA_HOST_PREFIX=''

I have not set it, so it found it empty.
It can also log: NETDATA_HOST_PREFIX is not passed from netdata

Which one does yours log?

titpetric · 2016-01-12T08:03:56Z

I updated the docker image with the new netdata code, and ran the container without --cap-add. It seems you were correct, the environment is passed to apps.plugin. It also seems the --cap-add switch is here to stay:

16-01-12 07:58:16: ERROR: apps.plugin: Cannot process /host/proc/1/io (errno 13, Permission denied)
...

So, to amend my theory, it seems the mapped proc filesystem is not readable without SYS_PTRACE. Closing the issue, thanks for everything.

Fetch chart type from the database

* httpcheck charts fix * httpcheck wip * httpcheck wip * HttpCheck => HTTPCheck * HttpCheck => HTTPCheck * HTTPCheck some tests * minor

ktsaou added a commit that referenced this issue Jan 11, 2016

apps.plugin inherits host prefix from netdata #43

50c97c7

ktsaou added a commit that referenced this issue Jan 11, 2016

apps.plugin logs NETDATA_HOST_PREFIX variable #43

7ad2b63

titpetric closed this as completed Jan 12, 2016

pando85 mentioned this issue Dec 6, 2016

Apps plugin not working titpetric/netdata#24

Closed

ilyam8 added the docker docker related issues label May 30, 2020

stelfrag added a commit that referenced this issue Jun 8, 2021

Merge pull request #43 from stelfrag/fetch_chart_type_from_db

f71e6cf

Fetch chart type from the database

vkalintiris pushed a commit to vkalintiris/netdata that referenced this issue Dec 13, 2023

Httpcheck fix (netdata#43)

4038919

* httpcheck charts fix * httpcheck wip * httpcheck wip * HttpCheck => HTTPCheck * HttpCheck => HTTPCheck * HTTPCheck some tests * minor

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker and Apps plugin #43

Docker and Apps plugin #43

titpetric commented Jan 11, 2016

ktsaou commented Jan 11, 2016

titpetric commented Jan 11, 2016

ktsaou commented Jan 11, 2016

ktsaou commented Jan 11, 2016

titpetric commented Jan 11, 2016

titpetric commented Jan 11, 2016

titpetric commented Jan 11, 2016

titpetric commented Jan 11, 2016

titpetric commented Jan 11, 2016

titpetric commented Jan 11, 2016

ktsaou commented Jan 11, 2016

titpetric commented Jan 11, 2016

ktsaou commented Jan 11, 2016

titpetric commented Jan 12, 2016

Docker and Apps plugin #43

Docker and Apps plugin #43

Comments

titpetric commented Jan 11, 2016

ktsaou commented Jan 11, 2016

titpetric commented Jan 11, 2016

ktsaou commented Jan 11, 2016

ktsaou commented Jan 11, 2016

titpetric commented Jan 11, 2016

titpetric commented Jan 11, 2016

titpetric commented Jan 11, 2016

titpetric commented Jan 11, 2016

titpetric commented Jan 11, 2016

titpetric commented Jan 11, 2016

ktsaou commented Jan 11, 2016

titpetric commented Jan 11, 2016

ktsaou commented Jan 11, 2016

titpetric commented Jan 12, 2016