Can't find mounted cgroup on Debian

[benvoi]

Getting the following error while enabling the docker_daemon on the agent running on my host (and not in a container)
2016-02-07 03:20:06 CET | CRITICAL | dd.collector | checks.docker_daemon(docker_daemon.py:201) | Can't find mounted memory cgroups. 2016-02-07 03:20:06 CET | WARNING | dd.collector | checks.docker_daemon(__init__.py:649) | Initialization failed. Will retry at next iteration

Neither #1896 nor #1908 are fixing my problem.

Thanks in advance.

[yannmh]

Thanks for reporting the issue @aieeeeeeeee

Could you reach out support using flare ? This would be a valuable feedback to help us troubleshooting it.

Thanks

[gtaylor]

Has anyone found a workaround for this?

[yannmh]

@aieeeeeeeee @gtaylor, were you able to reach out the support team using the flare command ? If so, let's follow the discussion on these threads.

Additionally please note that Datadog Agent 5.7.0 (released on 03/07/2016) ships many bugfixes related to the docker_daemon check: if you haven't upgraded yet, it's worth giving a try. More information is available in the CHANGELOG

[gtaylor]

I'm running 5.7.0-1. I haven't had a chance to get into a conversation with support just yet.

[adrien-candiotti]

I'm running the agent from inside a container and have the same error.

This is my docker-cloud.yml (docker-compose) file, just in case:

datadog:
  image: 'datadog/docker-dd-agent:latest'
  deployment_strategy: every_node
  environment:
    - API_KEY=$MY_API_KEY
    - HOSTNAME=$DOCKERCLOUD_NODE_HOSTNAME
  privileged: true
  restart: on-failure
  tags:
    - production
  volumes:
    - '/var/run/docker.sock:/var/run/docker.sock'
    - '/proc:/host/proc:ro'
    - '/sys/fs/cgroup:/host/sys/fs/cgroup:ro'

It worked perfectly on another host.

[adrien-candiotti]

I just read a bit of your source and checked my /proc/mounts file.
There is no /sys/fs/cgroup/memory cgroup on debian like on ubuntu for example.

After some reading I found this:

http://serverfault.com/questions/525162/what-is-required-to-activate-cgroups-in-linux

Referencing:

https://wiki.debian.org/LXC

It seems like the debian we are using is missing the cgroup memory kernel option.

@gtaylor @aieeeeeeeee could you check your /proc/cmdline and confirm the absence of the cgroup_enable=memory option?

[gtaylor]

Hey, so I actually had to manually enable this on all of our Wheezy VMs. Though you might not be using Ansible, this may help get the basic idea of the steps I took to work around this issue:

---
# Based on instructions here: https://docs.docker.com/engine/installation/linux/debian/#debian-wheezy-stable-7-x-64-bit
- name: adding Docker APT key
  apt_key: >
    keyserver=p80.pool.sks-keyservers.net
    id=58118E89F3A912897C070ADBF76221572C52609D

- name: install ca-certificates
  apt: name=ca-certificates state=present

- name: install apt-transport-https
  apt: name=apt-transport-https state=present force=yes

- name: add Docker APT repo
  apt_repository: >
    repo='deb https://apt.dockerproject.org/repo debian-{{ ansible_distribution_release }} main'
    state=present update_cache=yes

- name: install Docker Engine
  apt: >
    name=docker-engine=1.10.3-0~{{ ansible_distribution_release }}
    state=present

- name: install docker-py
  pip: name=docker-py version=1.7.2

- name: enable memory cgroups
  lineinfile: >
    dest=/etc/default/grub
    line='GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT cgroup_enable=memory"'
  when: ansible_distribution_release == "wheezy"
  notify: update grub

- meta: flush_handlers

For those with Ansible, these are the matching handlers:

- name: update grub
  command: update-grub
  notify: restart machine

- name: restart machine
  command: shutdown -r now
  async: 0
  poll: 0
  ignore_errors: true
  notify: waiting for server to come back

- name: waiting for server to come back
  local_action: wait_for port=22 host="{{ inventory_hostname }}" state=started delay=10
  become: false

[adrien-candiotti]

For the dockercloud + debian hosts user, just added this line to /etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT cgroup_enable=memory"

Used grub-update and rebooted the machine.
Check that the /proc/cmdline displays the cgroup_enable=memory option and everythin works :)

[rmoriz]

While I understand that Debian misses the cgroup_enable setting by default, dd-agent should not completely crash the docker integration when not available.

e.g. docker stats works without the option, it just doesn't display memory usage. I think dd-agent should behave the same way