Add docker plugin by tuxity · Pull Request #4191 · netdata/netdata

tuxity · 2018-09-13T19:59:02Z

As discussed in #4111, I have created a plugin to allow us to have some docker metrics.
For now only running containers, healthy and unhealthy container with an alarm when unhealthy containers > 0

ktsaou · 2018-09-13T21:08:46Z

conf.d/python.d/dockerd.conf

+# only one of them will run (they have the same name)
+#
+local:
+    url: 'unix://var/run/docker.sock'


I think we can have another one for the case netdata runs in a container itself:

local: url: 'unix://host/var/run/docker.sock'

So both at the same time. Only one will work.

Hmmm I don't understand, a 2nd

local: url: 'unix://host/var/run/docker.sock'

?

I also don't understand that. @ktsaou what do you mean? Containerized netdata shouldn't have unix://host/var/run/docker.sock socket, it is mounted as unix://var/run/docker.sock

As per: https://github.com/firehol/netdata/wiki/Install-netdata-with-Docker#run-netdata-with-docker-command

ok,

python plugin supports jobs. When 2 jobs have the same name, the code picks the first that works.

We use this feature for auto-detection. Let's assume nginx files on ubuntu are on /var/log/www and on centos on /var/log/httpd. We create 2 jobs with the same name, and the plugin will pick the first job that works. The same is true for example when we try to connect to a port. We try localhost, 127.0.0.1 and ::1 - only one will actually be used - the first that succeeds will obsolete the others.

Though the instructions I gave you above are wrong. The right config is:

local_run: name: 'local' url: 'unix://var/run/docker.sock' local_host: name: 'local' url: 'unix://host/var/run/docker.sock'

So, the first of the above that will succeed, will obsolete the other. Only one job for each name is used.

And now I saw that containerized netdata has the socket as /var/run/docker.sock, so the above are not needed.
Sorry...

ilyam8 · 2018-09-14T00:53:22Z

python.d/dockerd.chart.py

+    import docker
+    DOCKER_PY = True
+except ImportError:
+    DOCKER_PY = False


name it HAS_DOCKER

ilyam8 · 2018-09-14T01:00:34Z

python.d/dockerd.chart.py

+
+        self.client = docker.DockerClient(base_url=self.configuration.get('url', 'unix://var/run/docker.sock'))
+
+        return True if self.client else False


self.client is always True. It is just initialization.

ilyam8 · 2018-09-14T01:13:10Z

python.d/dockerd.chart.py

+        data['healthy_containers'] = len(self.client.containers.list(filters={'health': 'healthy'}))
+        data['unhealthy_containers'] = len(self.client.containers.list(filters={'health': 'unhealthy'}))
+
+        return data or None


self.client.containers.list() returns a list of Containers objects. Does object provide method/attribute about health status?

If so we can call self.client.containers.list() only once then.

It doesn't seems possible, we have to call it multiple times https://docker-py.readthedocs.io/en/stable/containers.html

Ok, i see list method has sparse option

sparse (bool) – Do not inspect containers. Returns partial information, but guaranteed not to block

Can we use it together with health filter?

Oh yes we can do that, I'm adding it

ilyam8 · 2018-09-14T17:29:41Z

python.d/dockerd.chart.py

+
+        self.client = docker.DockerClient(base_url=self.configuration.get('url', 'unix://var/run/docker.sock'))
+
+        return True


check checks only if python docker package is installed.

We need to check if we can to connect and retrieve data.

I can do a docker.version() with a try to see if it's ok for more safety

I used docker.ping() instead https://docker-py.readthedocs.io/en/stable/client.html#docker.client.DockerClient.ping

conf.d/health.d/dockerd.conf

paulfantom · 2018-09-14T21:59:55Z

@ktsaou while we are on the subject of monitoring healthy containers, does netdata have HTTP endpoint for healthchecks?

ktsaou · 2018-09-14T23:22:54Z

@ktsaou while we are on the subject of monitoring healthy containers, does netdata have HTTP endpoint for healthchecks?

You mean to check remote http endpoints for their health (it has http_check plugin), or receive traps of some kind from remote apps?

The later does not exist yet. Do you have something in mind? You know something similar?

paulfantom · 2018-09-15T11:04:03Z

I mean, do netdata by itself have some endpoint which could be checked if it finished bootstraping process (is healthy)? Something that would show that netdata finished initial discovery stage and it is ready to collect metrics (fully operational).

I don't mean to check health status of other services.

It doesn't need to be HTTP endpoint, just something that can be easily checked in automated fashion (not by parsing logs). This would allow us to use HEALTHCHECK directive in Dockerfile

ilyam8 · 2018-09-15T17:11:13Z

python.d/README.md

+**Requirement:**
+* `docker` package
+
+### configuration


please add plugin charts info

ktsaou · 2018-09-16T22:39:31Z

so, merge?

ilyam8 · 2018-09-17T21:26:08Z

yes

ktsaou reviewed Sep 13, 2018

View reviewed changes

ktsaou requested a review from ilyam8 September 13, 2018 21:09

ilyam8 reviewed Sep 14, 2018

View reviewed changes

ilyam8 requested changes Sep 14, 2018

View reviewed changes

paulfantom suggested changes Sep 14, 2018

View reviewed changes

conf.d/health.d/dockerd.conf Show resolved Hide resolved

paulfantom approved these changes Sep 15, 2018

View reviewed changes

ilyam8 requested changes Sep 15, 2018

View reviewed changes

python.d/README.md

**Requirement:**

* `docker` package

### configuration

Copy link

Member

ilyam8 Sep 15, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add plugin charts info

Add docker plugin

2cf2371

ilyam8 approved these changes Sep 16, 2018

View reviewed changes

ktsaou merged commit b3578ca into netdata:master Sep 17, 2018

tuxity mentioned this pull request Sep 21, 2018

Raise an alarm when a docker container is unhealthy #4111

Closed


		self.client = docker.DockerClient(base_url=self.configuration.get('url', 'unix://var/run/docker.sock'))

		return True if self.client else False


		self.client = docker.DockerClient(base_url=self.configuration.get('url', 'unix://var/run/docker.sock'))

		return True

Conversation

tuxity commented Sep 13, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tuxity Sep 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paulfantom Sep 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tuxity Sep 14, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

paulfantom commented Sep 14, 2018

Uh oh!

ktsaou commented Sep 14, 2018

Uh oh!

paulfantom commented Sep 15, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ktsaou commented Sep 16, 2018

Uh oh!

ilyam8 commented Sep 17, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tuxity Sep 14, 2018 •

edited

Loading

paulfantom Sep 15, 2018 •

edited

Loading

tuxity Sep 14, 2018 •

edited

Loading

paulfantom commented Sep 15, 2018 •

edited

Loading