New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System error in /sys/fs/cgroup/cpu,cpuactt #16256

Open
fommil opened this Issue Sep 12, 2015 · 51 comments

Comments

Projects
None yet
@fommil

fommil commented Sep 12, 2015

I'm on Debian Unstable with the docker.io images (debian no longer repackages). This is what happens when I try to run any image:

~ docker run -it debian:jessie
Error response from daemon: Cannot start container c35192b15227f0a93170415f688bd7ff974aaf7af63a168398d865d857ed2f75: [8] System error: open /sys/fs/cgroup/cpu,cpuacct/init.scope/system.slice/docker-c35192b15227f0a93170415f688bd7ff974aaf7af63a168398d865d857ed2f75.scope/cpu.shares: no such file or directory

and indeed, that structure does not exist on my directory

~ ls /sys/fs/cgroup/cpu,cpuacct/init.scope/
cgroup.clone_children  cgroup.procs  cpuacct.stat  cpuacct.usage  cpuacct.usage_percpu  cpu.shares  notify_on_release  tasks

i.e. no system.slice.

I'm running a very new linux kernel, has the sys directory perhaps changed to the point that docker breaks?


~ uname -a
Linux Samskara 4.1.0-2-amd64 #1 SMP Debian 4.1.6-1 (2015-08-23) x86_64 GNU/Linux

~ docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/amd64
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/amd64

~ docker -D info
Containers: 3
Images: 13
Storage Driver: devicemapper
 Pool Name: docker-8:5-27000835-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: extfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 1.601 GB
 Data Space Total: 107.4 GB
 Data Space Available: 105.8 GB
 Metadata Space Used: 1.978 MB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.146 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Data loop file: /mnt/archive/docker/devicemapper/devicemapper/data
 Metadata loop file: /mnt/archive/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.104 (2015-08-10)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.1.0-2-amd64
Operating System: Debian GNU/Linux stretch/sid (containerized)
CPUs: 4
Total Memory: 15.62 GiB
Name: Samskara
ID: EM6A:FAHK:YV4K:3AUD:LWIU:RJ2Q:ZPOP:Q4C7:IIAX:ZIR6:P5UY:67GB
Username: fommil
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Sep 12, 2015

that auto response is very useful... are you using a service for that or did you hack something up? (I want to use it for my projects if it is possible to customise it to only respond once to a submitter to avoid spamming people)

fommil commented Sep 12, 2015

that auto response is very useful... are you using a service for that or did you hack something up? (I want to use it for my projects if it is possible to customise it to only respond once to a submitter to avoid spamming people)

@bosr

This comment has been minimized.

Show comment
Hide comment
@bosr

bosr Sep 12, 2015

Hi @fommil

If you are on Debian Unstable, there is a chance you have just upgraded to systemd 226, which introduces the init.scope and places init (pid 1) inside it:

$ cat /proc/1/cgroup
9:cpuset:/
8:memory:/
7:devices:/init.scope
6:cpu,cpuacct:/
5:perf_event:/
4:net_cls,net_prio:/
3:freezer:/
2:blkio:/
1:name=systemd:/init.scope

source: http://news.softpedia.com/news/systemd-226-adds-new-features-to-the-dhcp-implementation-of-networkd-more-491265.shtml

I have the same error message as you.

bosr commented Sep 12, 2015

Hi @fommil

If you are on Debian Unstable, there is a chance you have just upgraded to systemd 226, which introduces the init.scope and places init (pid 1) inside it:

$ cat /proc/1/cgroup
9:cpuset:/
8:memory:/
7:devices:/init.scope
6:cpu,cpuacct:/
5:perf_event:/
4:net_cls,net_prio:/
3:freezer:/
2:blkio:/
1:name=systemd:/init.scope

source: http://news.softpedia.com/news/systemd-226-adds-new-features-to-the-dhcp-implementation-of-networkd-more-491265.shtml

I have the same error message as you.

@cdd-aix

This comment has been minimized.

Show comment
Hide comment
@cdd-aix

cdd-aix Sep 12, 2015

Downgrading all systemd components to 225 on Debian resolves the issue.

Bug #798778 opened against debian sid's systemd package.

Long term, docker should handle systemd's new init.scope. 4 days notice is insufficient :-).
Short term, no distribution should release new versions of systemd that make major changes to cgroup space without testing them against other packages that use cgroup space first.

cdd-aix commented Sep 12, 2015

Downgrading all systemd components to 225 on Debian resolves the issue.

Bug #798778 opened against debian sid's systemd package.

Long term, docker should handle systemd's new init.scope. 4 days notice is insufficient :-).
Short term, no distribution should release new versions of systemd that make major changes to cgroup space without testing them against other packages that use cgroup space first.

@bosr

This comment has been minimized.

Show comment
Hide comment
@bosr

bosr Sep 12, 2015

Agreed!

Romain

On 12 Sep 2015, at 16:48, Chris Dukes notifications@github.com wrote:

Downgrading all systemd components to 225 on Debian resolves the issue.

Bug #798778 opened against debian sid's systemd package.

Long term, docker should handle systemd's new init.scope. 4 days notice is insufficient :-).
Short term, no distribution should release new versions of systemd that make major changes to cgroup space without testing them against other packages that use cgroup space first.


Reply to this email directly or view it on GitHub.

bosr commented Sep 12, 2015

Agreed!

Romain

On 12 Sep 2015, at 16:48, Chris Dukes notifications@github.com wrote:

Downgrading all systemd components to 225 on Debian resolves the issue.

Bug #798778 opened against debian sid's systemd package.

Long term, docker should handle systemd's new init.scope. 4 days notice is insufficient :-).
Short term, no distribution should release new versions of systemd that make major changes to cgroup space without testing them against other packages that use cgroup space first.


Reply to this email directly or view it on GitHub.

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Sep 13, 2015

Contributor

You can use fs cgroups if you want, they're much more stable and has more features.
--exec-opt native.cgroupdriver=cgroupfs to daemon flags.

Contributor

LK4D4 commented Sep 13, 2015

You can use fs cgroups if you want, they're much more stable and has more features.
--exec-opt native.cgroupdriver=cgroupfs to daemon flags.

@cdd-aix

This comment has been minimized.

Show comment
Hide comment
@cdd-aix

cdd-aix Sep 13, 2015

Alexander,

Interesting. That does appear to resolve the issue.

Thank you.

I have questions.

  1. If cgroupfs is preferable to systemd, why is the default to check for
    systemd then fallback to cgroupfs?
  2. Has systemd done other experiments with cgroup space with similar
    fallout?
  3. If not, what were the other historical instabilities?
  4. What are the additional features provided by cgroupfs as the driver?
    https://docs.docker.com/reference/commandline/daemon/
    is somewhat silent on the above questions.

I see the differences in /sys/fs/cgroup between the two. But the context
on the nature of the differences eludes me.

Again, Thank You.

On Sat, Sep 12, 2015 at 11:54 PM, Alexander Morozov <
notifications@github.com> wrote:

You can use fs cgroups if you want, they're much more stable and has more
features.
--exec-opt native.cgroupdriver=cgroupfs to daemon flags.


Reply to this email directly or view it on GitHub
#16256 (comment).

cdd-aix commented Sep 13, 2015

Alexander,

Interesting. That does appear to resolve the issue.

Thank you.

I have questions.

  1. If cgroupfs is preferable to systemd, why is the default to check for
    systemd then fallback to cgroupfs?
  2. Has systemd done other experiments with cgroup space with similar
    fallout?
  3. If not, what were the other historical instabilities?
  4. What are the additional features provided by cgroupfs as the driver?
    https://docs.docker.com/reference/commandline/daemon/
    is somewhat silent on the above questions.

I see the differences in /sys/fs/cgroup between the two. But the context
on the nature of the differences eludes me.

Again, Thank You.

On Sat, Sep 12, 2015 at 11:54 PM, Alexander Morozov <
notifications@github.com> wrote:

You can use fs cgroups if you want, they're much more stable and has more
features.
--exec-opt native.cgroupdriver=cgroupfs to daemon flags.


Reply to this email directly or view it on GitHub
#16256 (comment).

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Sep 13, 2015

--exec-opt native.cgroupdriver=cgroupfs works for me! :-)

fommil commented Sep 13, 2015

--exec-opt native.cgroupdriver=cgroupfs works for me! :-)

@fommil fommil closed this Sep 13, 2015

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Sep 13, 2015

Member

@LK4D4 are there changes to be made, or docs to be added w.r.t #16256 (comment)? I don't know the answers to those, but let me know if the documentation should have extra information about that 👍

Member

thaJeztah commented Sep 13, 2015

@LK4D4 are there changes to be made, or docs to be added w.r.t #16256 (comment)? I don't know the answers to those, but let me know if the documentation should have extra information about that 👍

@fommil

This comment has been minimized.

Show comment
Hide comment
@fommil

fommil Sep 13, 2015

It would be good if this setting was automatically enabled for relevant systems

fommil commented Sep 13, 2015

It would be good if this setting was automatically enabled for relevant systems

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Sep 13, 2015

Member

@fommil feel free to create a separate issue for that, if you can clearly describe what you think needs to be done (and please add a link to this issue as well, for reference).

Member

thaJeztah commented Sep 13, 2015

@fommil feel free to create a separate issue for that, if you can clearly describe what you think needs to be done (and please add a link to this issue as well, for reference).

@runcom

This comment has been minimized.

Show comment
Hide comment
@runcom

runcom Sep 13, 2015

Member

Or libcontainer should handle the fact that with systemd >225 cgroups paths are different?

Member

runcom commented Sep 13, 2015

Or libcontainer should handle the fact that with systemd >225 cgroups paths are different?

@epifanio

This comment has been minimized.

Show comment
Hide comment
@epifanio

epifanio Sep 14, 2015

Hi, i'm running in the same problem (debian sid) how/whre should I use the option : --exec-opt native.cgroupdriver=cgroupfs ?

epifanio commented Sep 14, 2015

Hi, i'm running in the same problem (debian sid) how/whre should I use the option : --exec-opt native.cgroupdriver=cgroupfs ?

@cdd-aix

This comment has been minimized.

Show comment
Hide comment
@cdd-aix

cdd-aix Sep 14, 2015

/etc/default
On Sep 14, 2015 2:03 PM, "epifanio" notifications@github.com wrote:

Hi, i'm running in the same problem (debian sid) how/whre should I use the
option : --exec-opt native.cgroupdriver=cgroupfs ?


Reply to this email directly or view it on GitHub
#16256 (comment).

cdd-aix commented Sep 14, 2015

/etc/default
On Sep 14, 2015 2:03 PM, "epifanio" notifications@github.com wrote:

Hi, i'm running in the same problem (debian sid) how/whre should I use the
option : --exec-opt native.cgroupdriver=cgroupfs ?


Reply to this email directly or view it on GitHub
#16256 (comment).

@epifanio

This comment has been minimized.

Show comment
Hide comment
@epifanio

epifanio Sep 14, 2015

I tried .. that's what i got :

service docker stop
service docker start
docker daemon --exec-opt native.cgroupdriver=cgroupfs
docker: 'daemon' is not a docker command.
See 'docker --help'.

epifanio commented Sep 14, 2015

I tried .. that's what i got :

service docker stop
service docker start
docker daemon --exec-opt native.cgroupdriver=cgroupfs
docker: 'daemon' is not a docker command.
See 'docker --help'.
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Sep 14, 2015

Member

@epifanio the docker daemon command was added in docker 1.8, so either upgrade to docker 1.8, or use docker -d, which was the "old" way of starting the docker daemon

Member

thaJeztah commented Sep 14, 2015

@epifanio the docker daemon command was added in docker 1.8, so either upgrade to docker 1.8, or use docker -d, which was the "old" way of starting the docker daemon

@epifanio

This comment has been minimized.

Show comment
Hide comment
@epifanio

epifanio Sep 14, 2015

@thaJeztah , thanks!
docker -d --exec-opt native.cgroupdriver=cgroupfs
works fine (eventually I'll upgrade my docker installation).
Question, i runned docker -d ... from a standard ssh shell on my server using nohup to avoid to kill the demon when i close the ssh session. How can i make those changes permanent, so that I don't need to issue it and leave supervisord to do the job?

epifanio commented Sep 14, 2015

@thaJeztah , thanks!
docker -d --exec-opt native.cgroupdriver=cgroupfs
works fine (eventually I'll upgrade my docker installation).
Question, i runned docker -d ... from a standard ssh shell on my server using nohup to avoid to kill the demon when i close the ssh session. How can i make those changes permanent, so that I don't need to issue it and leave supervisord to do the job?

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Sep 15, 2015

Contributor

btw, it's still an issue, it breaks oom for systemd as well. I'm not sure from where this init.scope crap is came, because mine devices is in user.slice, but inside container it's mounted as /init.scope/system.slice
Also I don't see mention of init.scope anywhere in systemd docs. Which is another reason to use fs implementation.

Contributor

LK4D4 commented Sep 15, 2015

btw, it's still an issue, it breaks oom for systemd as well. I'm not sure from where this init.scope crap is came, because mine devices is in user.slice, but inside container it's mounted as /init.scope/system.slice
Also I don't see mention of init.scope anywhere in systemd docs. Which is another reason to use fs implementation.

@LK4D4 LK4D4 reopened this Sep 15, 2015

@cdd-aix

This comment has been minimized.

Show comment
Hide comment
@cdd-aix

cdd-aix Sep 15, 2015

Take a peek at /proc/1/cgroup.
devices and blkio are in /init.scope.

In systemd 225 and older, those are in /.

Suspicion:
native.cgroupdriver=systemd uses systemd's cgroups in creating paths for
docker cgroups.

The release notes for systemd 226 does not address the why.

systemd/init's metrics are easy to track when fenced in a separate cgroup
hierarchy.

I have not rebooted since enabling memory cgroups and swap accounting, but
I suspect I will be pleasantly surprised.

On Tue, Sep 15, 2015 at 12:42 PM, Alexander Morozov <
notifications@github.com> wrote:

btw, it's still an issue, it breaks oom for systemd as well. I'm not sure
from where this init.scope crap is came, because mine devices is in
user.slice, but inside container it's mounted as /init.scope/system.slice


Reply to this email directly or view it on GitHub
#16256 (comment).

cdd-aix commented Sep 15, 2015

Take a peek at /proc/1/cgroup.
devices and blkio are in /init.scope.

In systemd 225 and older, those are in /.

Suspicion:
native.cgroupdriver=systemd uses systemd's cgroups in creating paths for
docker cgroups.

The release notes for systemd 226 does not address the why.

systemd/init's metrics are easy to track when fenced in a separate cgroup
hierarchy.

I have not rebooted since enabling memory cgroups and swap accounting, but
I suspect I will be pleasantly surprised.

On Tue, Sep 15, 2015 at 12:42 PM, Alexander Morozov <
notifications@github.com> wrote:

btw, it's still an issue, it breaks oom for systemd as well. I'm not sure
from where this init.scope crap is came, because mine devices is in
user.slice, but inside container it's mounted as /init.scope/system.slice


Reply to this email directly or view it on GitHub
#16256 (comment).

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Sep 15, 2015

Contributor

@cdd-aix funny. Seems like we're messed /proc/self with /proc/1 somehow.

Contributor

LK4D4 commented Sep 15, 2015

@cdd-aix funny. Seems like we're messed /proc/self with /proc/1 somehow.

@jeremyeder

This comment has been minimized.

Show comment
Hide comment
@jeremyeder

jeremyeder Sep 16, 2015

Seems no such issue exists in Fedora rawhide:

# rpm -q docker systemd
docker-1.9.0-2.gitf8950e0.fc24.x86_64
systemd-226-1.fc24.x86_64

# cat /proc/1/cgroup
10:hugetlb:/
9:memory:/init.scope
8:cpuset:/
7:net_cls,net_prio:/
6:perf_event:/
5:devices:/init.scope
4:blkio:/init.scope
3:freezer:/
2:cpu,cpuacct:/init.scope
1:name=systemd:/init.scope

# docker run fedora echo hi
hi
#

jeremyeder commented Sep 16, 2015

Seems no such issue exists in Fedora rawhide:

# rpm -q docker systemd
docker-1.9.0-2.gitf8950e0.fc24.x86_64
systemd-226-1.fc24.x86_64

# cat /proc/1/cgroup
10:hugetlb:/
9:memory:/init.scope
8:cpuset:/
7:net_cls,net_prio:/
6:perf_event:/
5:devices:/init.scope
4:blkio:/init.scope
3:freezer:/
2:cpu,cpuacct:/init.scope
1:name=systemd:/init.scope

# docker run fedora echo hi
hi
#
@venthur

This comment has been minimized.

Show comment
Hide comment
@venthur

venthur Sep 17, 2015

docker -d --exec-opt native.cgroupdriver=cgroupfs works on current Debian/Unstable.

venthur commented Sep 17, 2015

docker -d --exec-opt native.cgroupdriver=cgroupfs works on current Debian/Unstable.

@paulbdavis

This comment has been minimized.

Show comment
Hide comment
@paulbdavis

paulbdavis Sep 21, 2015

The issue only seems to manifest when using -c (though I guess some other options that utilize cgroups would also cause the issue).

% docker version
Client:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Sat Aug 15 17:29:10 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Sat Aug 15 17:29:10 UTC 2015
 OS/Arch:      linux/amd64
 % pacman -Qi systemd
Name           : systemd
Version        : 226-1
Description    : system and service manager
Architecture   : x86_64
URL            : http://www.freedesktop.org/wiki/Software/systemd
Licenses       : GPL2  LGPL2.1
Groups         : None
Provides       : nss-myhostname  systemd-tools=226  udev=226
Depends On     : acl  bash  dbus  iptables  kbd  kmod  hwids  libcap  libgcrypt  libsystemd  libidn  lz4  pam  libseccomp  util-linux  xz
Optional Deps  : cryptsetup: required for encrypted block devices [installed]
                 libmicrohttpd: remote journald capabilities
                 quota-tools: kernel-level quota management
                 systemd-sysvcompat: symlink package to provide sysvinit binaries [installed]
                 polkit: allow administration as unprivileged user [installed]
Required By    : accountsservice  android-udev  bluez-utils  chromium  colord  device-mapper  dhcpcd  docker  gnome-system-monitor  lib32-systemd  libgusb  libinput  libpulse
                 libusb  lvm2  media-player-info  mesa  mkinitcpio  netctl  pcmciautils  pcsclite  plymouth-legacy  polkit  python-pyudev  qt5-base  qtwebkit  rtkit  subversion
                 systemd-sysvcompat  udisks2  upower  xf86-input-evdev
Optional For   : None
Conflicts With : nss-myhostname  systemd-tools  udev
Replaces       : nss-myhostname  systemd-tools  udev
Installed Size :  27.24 MiB
Packager       : Dave Reisner <dreisner@archlinux.org>
Build Date     : Tue 08 Sep 2015 01:57:54 PM EDT
Install Date   : Sun 20 Sep 2015 03:17:27 PM EDT
Install Reason : Installed as a dependency for another package
Install Script : Yes
Validated By   : Signature
 % uname -r
4.1.6-1-ARCH
 % docker run --rm ubuntu echo hi
hi
 % docker run --rm -c 1 ubuntu echo hi
Error response from daemon: Cannot start container c44478b6b516c75fa72e090e61db49a08453f25c7fe59e877e641214dc3dd721: [8] System error: open /sys/fs/cgroup/cpu,cpuacct/init.scope/system.slice/docker-c44478b6b516c75fa72e090e61db49a08453f25c7fe59e877e641214dc3dd721.scope/cpu.shares: no such file or directory

paulbdavis commented Sep 21, 2015

The issue only seems to manifest when using -c (though I guess some other options that utilize cgroups would also cause the issue).

% docker version
Client:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Sat Aug 15 17:29:10 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Sat Aug 15 17:29:10 UTC 2015
 OS/Arch:      linux/amd64
 % pacman -Qi systemd
Name           : systemd
Version        : 226-1
Description    : system and service manager
Architecture   : x86_64
URL            : http://www.freedesktop.org/wiki/Software/systemd
Licenses       : GPL2  LGPL2.1
Groups         : None
Provides       : nss-myhostname  systemd-tools=226  udev=226
Depends On     : acl  bash  dbus  iptables  kbd  kmod  hwids  libcap  libgcrypt  libsystemd  libidn  lz4  pam  libseccomp  util-linux  xz
Optional Deps  : cryptsetup: required for encrypted block devices [installed]
                 libmicrohttpd: remote journald capabilities
                 quota-tools: kernel-level quota management
                 systemd-sysvcompat: symlink package to provide sysvinit binaries [installed]
                 polkit: allow administration as unprivileged user [installed]
Required By    : accountsservice  android-udev  bluez-utils  chromium  colord  device-mapper  dhcpcd  docker  gnome-system-monitor  lib32-systemd  libgusb  libinput  libpulse
                 libusb  lvm2  media-player-info  mesa  mkinitcpio  netctl  pcmciautils  pcsclite  plymouth-legacy  polkit  python-pyudev  qt5-base  qtwebkit  rtkit  subversion
                 systemd-sysvcompat  udisks2  upower  xf86-input-evdev
Optional For   : None
Conflicts With : nss-myhostname  systemd-tools  udev
Replaces       : nss-myhostname  systemd-tools  udev
Installed Size :  27.24 MiB
Packager       : Dave Reisner <dreisner@archlinux.org>
Build Date     : Tue 08 Sep 2015 01:57:54 PM EDT
Install Date   : Sun 20 Sep 2015 03:17:27 PM EDT
Install Reason : Installed as a dependency for another package
Install Script : Yes
Validated By   : Signature
 % uname -r
4.1.6-1-ARCH
 % docker run --rm ubuntu echo hi
hi
 % docker run --rm -c 1 ubuntu echo hi
Error response from daemon: Cannot start container c44478b6b516c75fa72e090e61db49a08453f25c7fe59e877e641214dc3dd721: [8] System error: open /sys/fs/cgroup/cpu,cpuacct/init.scope/system.slice/docker-c44478b6b516c75fa72e090e61db49a08453f25c7fe59e877e641214dc3dd721.scope/cpu.shares: no such file or directory
@flixi

This comment has been minimized.

Show comment
Hide comment
@flixi

flixi Sep 23, 2015


[root@alarm services]# docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/arm
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/arm
uname -r
4.2.0-1-ARCH

If I try to

docker run redis:latest

I get the error:

689d0208f5907a16b153bc213b9fb78777b7b2486890ddf8b8039c6c982e6396
Error response from daemon: Cannot start container 689d0208f5907a16b153bc213b9fb78777b7b2486890ddf8b8039c6c982e6396: [8] System error: open /sys/fs/cgroup/cpu,cpuacct/init.scope/system.slice/docker-689d0208f5907a16b153bc213b9fb78777b7b2486890ddf8b8039c6c982e6396.scope/cpu.shares: no such file or directory

Running ArchLinux on BeagleBone Black rev4.

flixi commented Sep 23, 2015


[root@alarm services]# docker version
Client version: 1.7.1
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 786b29d
OS/Arch (client): linux/arm
Server version: 1.7.1
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 786b29d
OS/Arch (server): linux/arm
uname -r
4.2.0-1-ARCH

If I try to

docker run redis:latest

I get the error:

689d0208f5907a16b153bc213b9fb78777b7b2486890ddf8b8039c6c982e6396
Error response from daemon: Cannot start container 689d0208f5907a16b153bc213b9fb78777b7b2486890ddf8b8039c6c982e6396: [8] System error: open /sys/fs/cgroup/cpu,cpuacct/init.scope/system.slice/docker-689d0208f5907a16b153bc213b9fb78777b7b2486890ddf8b8039c6c982e6396.scope/cpu.shares: no such file or directory

Running ArchLinux on BeagleBone Black rev4.

@paulbdavis

This comment has been minimized.

Show comment
Hide comment
@paulbdavis

paulbdavis Sep 23, 2015

On arch, the fd it is looking for exists at /sys/fs/cgroup/cpu,cpuacct/system.slice/docker-blahblah/cpu.shares instead of /sys/fs/cgroup/cpu,cpuacct/init.scope/system.slice/docker-blahblah/cpu.shares

It's there, just not under the init.scope directory

paulbdavis commented Sep 23, 2015

On arch, the fd it is looking for exists at /sys/fs/cgroup/cpu,cpuacct/system.slice/docker-blahblah/cpu.shares instead of /sys/fs/cgroup/cpu,cpuacct/init.scope/system.slice/docker-blahblah/cpu.shares

It's there, just not under the init.scope directory

@tinti

This comment has been minimized.

Show comment
Hide comment
@tinti

tinti Sep 25, 2015

docker -d --exec-opt native.cgroupdriver=cgroupfs works on Arch.

docker

$ docker version
Client:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.5.1
 Git commit:   0a8c2e3-dirty
 Built:        Mon Sep 14 12:09:36 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.5.1
 Git commit:   0a8c2e3-dirty
 Built:        Mon Sep 14 12:09:36 UTC 2015
 OS/Arch:      linux/amd64

kernel

uname -r
4.1.6-1-ARCH

systemd

$ pacman -Qi systemd
Name           : systemd
Version        : 226-1
Description    : system and service manager
Architecture   : x86_64
URL            : http://www.freedesktop.org/wiki/Software/systemd
Licenses       : GPL2  LGPL2.1
Groups         : None
Provides       : nss-myhostname  systemd-tools=226  udev=226
Depends On     : acl  bash  dbus  iptables  kbd  kmod  hwids  libcap  libgcrypt  libsystemd  libidn  lz4  pam  libseccomp  util-linux  xz
Optional Deps  : cryptsetup: required for encrypted block devices [installed]
                 libmicrohttpd: remote journald capabilities
                 quota-tools: kernel-level quota management
                 systemd-sysvcompat: symlink package to provide sysvinit binaries [installed]
                 polkit: allow administration as unprivileged user [installed]
Required By    : accountsservice  android-udev  chromium  colord  device-mapper  dhcpcd  docker  lib32-systemd  libgusb  libinput  libmbim  libpulse  libusb  lvm2  media-player-info  mesa  mkinitcpio  modemmanager  netctl  pcmciautils
                 polkit  qt5-base  qtwebkit  rtkit  subversion  systemd-sysvcompat  udisks2  upower  xf86-input-evdev
Optional For   : None
Conflicts With : nss-myhostname  systemd-tools  udev
Replaces       : nss-myhostname  systemd-tools  udev
Installed Size :  27.24 MiB
Packager       : Dave Reisner <dreisner@archlinux.org>
Build Date     : Tue 08 Sep 2015 02:57:54 PM BRT
Install Date   : Mon 21 Sep 2015 07:45:44 AM BRT
Install Reason : Installed as a dependency for another package
Install Script : Yes
Validated By   : Signature

tinti commented Sep 25, 2015

docker -d --exec-opt native.cgroupdriver=cgroupfs works on Arch.

docker

$ docker version
Client:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.5.1
 Git commit:   0a8c2e3-dirty
 Built:        Mon Sep 14 12:09:36 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.8.2
 API version:  1.20
 Go version:   go1.5.1
 Git commit:   0a8c2e3-dirty
 Built:        Mon Sep 14 12:09:36 UTC 2015
 OS/Arch:      linux/amd64

kernel

uname -r
4.1.6-1-ARCH

systemd

$ pacman -Qi systemd
Name           : systemd
Version        : 226-1
Description    : system and service manager
Architecture   : x86_64
URL            : http://www.freedesktop.org/wiki/Software/systemd
Licenses       : GPL2  LGPL2.1
Groups         : None
Provides       : nss-myhostname  systemd-tools=226  udev=226
Depends On     : acl  bash  dbus  iptables  kbd  kmod  hwids  libcap  libgcrypt  libsystemd  libidn  lz4  pam  libseccomp  util-linux  xz
Optional Deps  : cryptsetup: required for encrypted block devices [installed]
                 libmicrohttpd: remote journald capabilities
                 quota-tools: kernel-level quota management
                 systemd-sysvcompat: symlink package to provide sysvinit binaries [installed]
                 polkit: allow administration as unprivileged user [installed]
Required By    : accountsservice  android-udev  chromium  colord  device-mapper  dhcpcd  docker  lib32-systemd  libgusb  libinput  libmbim  libpulse  libusb  lvm2  media-player-info  mesa  mkinitcpio  modemmanager  netctl  pcmciautils
                 polkit  qt5-base  qtwebkit  rtkit  subversion  systemd-sysvcompat  udisks2  upower  xf86-input-evdev
Optional For   : None
Conflicts With : nss-myhostname  systemd-tools  udev
Replaces       : nss-myhostname  systemd-tools  udev
Installed Size :  27.24 MiB
Packager       : Dave Reisner <dreisner@archlinux.org>
Build Date     : Tue 08 Sep 2015 02:57:54 PM BRT
Install Date   : Mon 21 Sep 2015 07:45:44 AM BRT
Install Reason : Installed as a dependency for another package
Install Script : Yes
Validated By   : Signature

@Chris00

This comment has been minimized.

Show comment
Hide comment
@Chris00

Chris00 Oct 6, 2015

Contributor

Note that on Debian testing (with systemd), you have to add the option --exec-opt native.cgroupdriver=cgroupfs in /lib/systemd/system/docker.service:

...
[Service]
ExecStart=/usr/bin/docker -d -H fd:// --exec-opt native.cgroupdriver=cgroupfs
...
Contributor

Chris00 commented Oct 6, 2015

Note that on Debian testing (with systemd), you have to add the option --exec-opt native.cgroupdriver=cgroupfs in /lib/systemd/system/docker.service:

...
[Service]
ExecStart=/usr/bin/docker -d -H fd:// --exec-opt native.cgroupdriver=cgroupfs
...
@dubwoc

This comment has been minimized.

Show comment
Hide comment
@dubwoc

dubwoc Oct 10, 2015

You really should be making that change in "/etc/default/docker" not directly to the systemd service file. In /etc/default/docker add the argument to the DOCKER_OPS variable.

dubwoc commented Oct 10, 2015

You really should be making that change in "/etc/default/docker" not directly to the systemd service file. In /etc/default/docker add the argument to the DOCKER_OPS variable.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 10, 2015

Member

@dubwoc agreed, however note that the /etc/default/docker file is no longer used in the newer packages; using drop-ins is the recommended method. See https://docs.docker.com/articles/systemd/#custom-docker-daemon-options for reference

Member

thaJeztah commented Oct 10, 2015

@dubwoc agreed, however note that the /etc/default/docker file is no longer used in the newer packages; using drop-ins is the recommended method. See https://docs.docker.com/articles/systemd/#custom-docker-daemon-options for reference

@Chris00

This comment has been minimized.

Show comment
Hide comment
@Chris00

Chris00 Oct 10, 2015

Contributor

Well, in the package http://get.docker.io/ubuntu/ , /lib/systemd/system/docker.service does not declare /etc/default/docker as an environment file — so its settings are not used anymore.

Contributor

Chris00 commented Oct 10, 2015

Well, in the package http://get.docker.io/ubuntu/ , /lib/systemd/system/docker.service does not declare /etc/default/docker as an environment file — so its settings are not used anymore.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 10, 2015

Member

@Chris00 see the article I linked, which contains instructions on how to override the default settings, or add an environment file, without changing the default docker.service

Member

thaJeztah commented Oct 10, 2015

@Chris00 see the article I linked, which contains instructions on how to override the default settings, or add an environment file, without changing the default docker.service

@Chris00

This comment has been minimized.

Show comment
Hide comment
@Chris00

Chris00 Oct 10, 2015

Contributor

@thaJeztah Right. I read your linked page more carefully and it suffices indeed to add a file /lib/systemd/system/docker.service containing:

[Service]
ExecStart=
ExecStart=/usr/bin/docker -d -H fd:// --exec-opt native.cgroupdriver=cgroupfs

Thanks.

Contributor

Chris00 commented Oct 10, 2015

@thaJeztah Right. I read your linked page more carefully and it suffices indeed to add a file /lib/systemd/system/docker.service containing:

[Service]
ExecStart=
ExecStart=/usr/bin/docker -d -H fd:// --exec-opt native.cgroupdriver=cgroupfs

Thanks.

@vimagick vimagick referenced this issue Oct 13, 2015

Closed

Default gateway #63

@Corwin7

This comment has been minimized.

Show comment
Hide comment
@Corwin7

Corwin7 Oct 13, 2015

I have this problem and can not resolve it. I run Debian Testing (kernel 4.2.0-1-amd64) and as a result have Docker 1.7.1~dfsg1-1 . Going to packages.debian.org it looks like that's the absolute latest version; it's the same version for Debian Unstable. I guess CoreOS is right, SystemD and Docker just don't work together.

docker run ... --exec-opt native.cgroupdriver=cgroupfs  -d ...
flag provided but not defined: --exec-opt

Corwin7 commented Oct 13, 2015

I have this problem and can not resolve it. I run Debian Testing (kernel 4.2.0-1-amd64) and as a result have Docker 1.7.1~dfsg1-1 . Going to packages.debian.org it looks like that's the absolute latest version; it's the same version for Debian Unstable. I guess CoreOS is right, SystemD and Docker just don't work together.

docker run ... --exec-opt native.cgroupdriver=cgroupfs  -d ...
flag provided but not defined: --exec-opt
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 13, 2015

Member

@Corwin7 those are the packages that are packaged by Debian; the apt-repository maintained by docker does have docker 1.8.3 for stretch: https://apt.dockerproject.org/repo/dists/debian-stretch

and the installation docs: https://docs.docker.com/installation/debian/

Member

thaJeztah commented Oct 13, 2015

@Corwin7 those are the packages that are packaged by Debian; the apt-repository maintained by docker does have docker 1.8.3 for stretch: https://apt.dockerproject.org/repo/dists/debian-stretch

and the installation docs: https://docs.docker.com/installation/debian/

@Corwin7

This comment has been minimized.

Show comment
Hide comment
@Corwin7

Corwin7 Oct 13, 2015

Thanks @thaJeztah :) I hate figuring out what stupid toy story of the month club character matches up to testing but I guessed one and it worked.

Corwin7 commented Oct 13, 2015

Thanks @thaJeztah :) I hate figuring out what stupid toy story of the month club character matches up to testing but I guessed one and it worked.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 13, 2015

Member

@Corwin7 yeah.. I know the feeling 😸

Member

thaJeztah commented Oct 13, 2015

@Corwin7 yeah.. I know the feeling 😸

@wsw70 wsw70 referenced this issue Oct 14, 2015

Closed

cpu.shares missing #1

@fgsch

This comment has been minimized.

Show comment
Hide comment
@fgsch

fgsch Oct 14, 2015

@Corwin7 In Debian just edit /etc/default/docker. I have:

DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 --exec-opt native.cgroupdriver=cgroupfs"

And works fine.

fgsch commented Oct 14, 2015

@Corwin7 In Debian just edit /etc/default/docker. I have:

DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4 --exec-opt native.cgroupdriver=cgroupfs"

And works fine.

@delcypher

This comment has been minimized.

Show comment
Hide comment
@delcypher

delcypher Oct 19, 2015

I also came across this issue using Arch Linux. Passing --exec-opt native.cgroupdriver=cgroupfs to the Docker daemon fixed the issue. Downgrading from system-d 227-1 to 225-3 also fixed the problem so system-d is definitely the problem (226 is also broken)

I had a quick look at system-d's NEWS file was version 226. This looks interesting

https://github.com/systemd/systemd/blob/1e0adaa45d2c1a300199069bfdeb494281b54086/NEWS#L252

        * systemd now optionally supports the new Linux kernel
          "unified" control group hierarchy. If enabled via the kernel
          command-line option 'systemd.unified_cgroup_hierarchy=1',
          systemd will try to mount the unified cgroup hierarchy
          directly on /sys/fs/cgroup. If not enabled, or not
          available, systemd will fall back to the legacy cgroup
          hierarchy setup, as before. Host system and containers can
          mix and match legacy and unified hierarchies as they
          wish. nspawn understands the $UNIFIED_CROUP_HIERARCHY
          environment variable to individually select the hierarchy to
          use for executed containers. By default, nspawn will use the
          unified hierarchy for the containers if the host uses the
          unified hierarchy, and the legacy hierarchy otherwise.
          Please note that at this point the unified hierarchy is an
          experimental kernel feature and is likely to change in one
          of the next kernel releases.  Therefore, it should not be
          enabled by default in downstream distributions yet. The
          minimum required kernel version for the unified hierarchy to
          work is 4.2. Note that when the unified hierarchy is used
          for the first time delegated access to controllers is
          safe. Because of this systemd-nspawn containers will get
          access to controllers now, as will systemd user
          sessions. This means containers and user sessions may now
          manage their own resources, partitioning up what the system
          grants them.

I tried passing systemd.unified_cgroup_hierarchy=0 as a kernel parameter when using system-d 227 but unfortunately I still experienced the issue :(

delcypher commented Oct 19, 2015

I also came across this issue using Arch Linux. Passing --exec-opt native.cgroupdriver=cgroupfs to the Docker daemon fixed the issue. Downgrading from system-d 227-1 to 225-3 also fixed the problem so system-d is definitely the problem (226 is also broken)

I had a quick look at system-d's NEWS file was version 226. This looks interesting

https://github.com/systemd/systemd/blob/1e0adaa45d2c1a300199069bfdeb494281b54086/NEWS#L252

        * systemd now optionally supports the new Linux kernel
          "unified" control group hierarchy. If enabled via the kernel
          command-line option 'systemd.unified_cgroup_hierarchy=1',
          systemd will try to mount the unified cgroup hierarchy
          directly on /sys/fs/cgroup. If not enabled, or not
          available, systemd will fall back to the legacy cgroup
          hierarchy setup, as before. Host system and containers can
          mix and match legacy and unified hierarchies as they
          wish. nspawn understands the $UNIFIED_CROUP_HIERARCHY
          environment variable to individually select the hierarchy to
          use for executed containers. By default, nspawn will use the
          unified hierarchy for the containers if the host uses the
          unified hierarchy, and the legacy hierarchy otherwise.
          Please note that at this point the unified hierarchy is an
          experimental kernel feature and is likely to change in one
          of the next kernel releases.  Therefore, it should not be
          enabled by default in downstream distributions yet. The
          minimum required kernel version for the unified hierarchy to
          work is 4.2. Note that when the unified hierarchy is used
          for the first time delegated access to controllers is
          safe. Because of this systemd-nspawn containers will get
          access to controllers now, as will systemd user
          sessions. This means containers and user sessions may now
          manage their own resources, partitioning up what the system
          grants them.

I tried passing systemd.unified_cgroup_hierarchy=0 as a kernel parameter when using system-d 227 but unfortunately I still experienced the issue :(

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Oct 19, 2015

Member

also see #16238

Member

thaJeztah commented Oct 19, 2015

also see #16238

@runcom

This comment has been minimized.

Show comment
Hide comment
@runcom

runcom Oct 21, 2015

Member

fedora rawhide seems to have the same issue btw (need to specify -c to trigger this behavior):

[root@localhost ~]# docker run -c 2 busybox echo hello
[  738.765697] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
[  738.921473] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
[  738.952694] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
[  738.955775] device veth955a37f entered promiscuous mode
[  738.956541] IPv6: ADDRCONF(NETDEV_UP): veth955a37f: link is not ready
[  738.988624] eth0: renamed from veth1f32ef3
[  738.989210] IPv6: ADDRCONF(NETDEV_CHANGE): veth955a37f: link becomes ready
[  738.989777] docker0: port 2(veth955a37f) entered forwarding state
[  738.990189] docker0: port 2(veth955a37f) entered forwarding state
[  739.003745] docker0: port 2(veth955a37f) entered disabled state
[  739.006178] veth1f32ef3: renamed from eth0
[  739.008221] docker0: port 2(veth955a37f) entered disabled state
[  739.008776] device veth955a37f left promiscuous mode
[  739.009252] docker0: port 2(veth955a37f) entered disabled state
Error response from daemon: Cannot start container a76b4423b7d02ef76c68eae8f6d9263ae06cd0406fe4338575888da425d88b73: [8] System error: open /sys/fs/cgroup/cpu,cpuacct/init.scope/system.slice/docker-a76b4423b7d02ef76c68eae8f6d9263ae06cd0406fe4338575888da425d88b73.scope/cpu.shares: no such file or directory

@jeremyeder

Member

runcom commented Oct 21, 2015

fedora rawhide seems to have the same issue btw (need to specify -c to trigger this behavior):

[root@localhost ~]# docker run -c 2 busybox echo hello
[  738.765697] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
[  738.921473] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
[  738.952694] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
[  738.955775] device veth955a37f entered promiscuous mode
[  738.956541] IPv6: ADDRCONF(NETDEV_UP): veth955a37f: link is not ready
[  738.988624] eth0: renamed from veth1f32ef3
[  738.989210] IPv6: ADDRCONF(NETDEV_CHANGE): veth955a37f: link becomes ready
[  738.989777] docker0: port 2(veth955a37f) entered forwarding state
[  738.990189] docker0: port 2(veth955a37f) entered forwarding state
[  739.003745] docker0: port 2(veth955a37f) entered disabled state
[  739.006178] veth1f32ef3: renamed from eth0
[  739.008221] docker0: port 2(veth955a37f) entered disabled state
[  739.008776] device veth955a37f left promiscuous mode
[  739.009252] docker0: port 2(veth955a37f) entered disabled state
Error response from daemon: Cannot start container a76b4423b7d02ef76c68eae8f6d9263ae06cd0406fe4338575888da425d88b73: [8] System error: open /sys/fs/cgroup/cpu,cpuacct/init.scope/system.slice/docker-a76b4423b7d02ef76c68eae8f6d9263ae06cd0406fe4338575888da425d88b73.scope/cpu.shares: no such file or directory

@jeremyeder

@johnjelinek

This comment has been minimized.

Show comment
Hide comment
@johnjelinek

johnjelinek Dec 2, 2015

👍 --exec-opt native.cgroupdriver=cgroupfs works for CentOS 7.1 too. I wish docker-machine create would configure this automatically.

johnjelinek commented Dec 2, 2015

👍 --exec-opt native.cgroupdriver=cgroupfs works for CentOS 7.1 too. I wish docker-machine create would configure this automatically.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Dec 2, 2015

Member

@johnjelinek we recently merged a PR to use if as default for docker 1.10; #17704 there's still some discussion on that though.

@LK4D4 should this be resolved with the changes in #17704?

Member

thaJeztah commented Dec 2, 2015

@johnjelinek we recently merged a PR to use if as default for docker 1.10; #17704 there's still some discussion on that though.

@LK4D4 should this be resolved with the changes in #17704?

@johnjelinek

This comment has been minimized.

Show comment
Hide comment
@johnjelinek

johnjelinek Dec 2, 2015

hrmm .. I think I spoke too soon:

$ eval "$(docker-machine env --swarm my-swarm)"
$ docker run --rm redis

                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 3.0.5 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 1
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'
...

but then, with this docker-compose.yml:

web:
  build: .
  command: python app.py
  ports:
     - "5000"
  hostname: hello.weave.local
  environment:
     - "affinity:container!=app_web_*"
redis:
  image: redis
  hostname: redis.weave.local

it fails on the redis spin up:

$ docker-compose up -d --force-recreate
Recreating app_web_1
Recreating app_redis_1
ERROR: Cannot start container 3d19005a57b6237bddbd868d5f158126001eed9cbcc8027e0c330f3410b23d69: [8] System error: write /sys/fs/cgroup/memory/system.slice/docker/3d19005a57b6237bddbd868d5f158126001eed9cbcc8027e0c330f3410b23d69/memory.swappiness: invalid argument

All the swarm dockers are configured with --exec-opt native.cgroupdriver=cgroupfs.

johnjelinek commented Dec 2, 2015

hrmm .. I think I spoke too soon:

$ eval "$(docker-machine env --swarm my-swarm)"
$ docker run --rm redis

                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 3.0.5 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in standalone mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 1
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           http://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'
...

but then, with this docker-compose.yml:

web:
  build: .
  command: python app.py
  ports:
     - "5000"
  hostname: hello.weave.local
  environment:
     - "affinity:container!=app_web_*"
redis:
  image: redis
  hostname: redis.weave.local

it fails on the redis spin up:

$ docker-compose up -d --force-recreate
Recreating app_web_1
Recreating app_redis_1
ERROR: Cannot start container 3d19005a57b6237bddbd868d5f158126001eed9cbcc8027e0c330f3410b23d69: [8] System error: write /sys/fs/cgroup/memory/system.slice/docker/3d19005a57b6237bddbd868d5f158126001eed9cbcc8027e0c330f3410b23d69/memory.swappiness: invalid argument

All the swarm dockers are configured with --exec-opt native.cgroupdriver=cgroupfs.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Dec 3, 2015

Member

@johnjelinek could that be related to #15159?

Member

thaJeztah commented Dec 3, 2015

@johnjelinek could that be related to #15159?

@johnjelinek

This comment has been minimized.

Show comment
Hide comment
@johnjelinek

johnjelinek Dec 3, 2015

I don't think so -- I didn't unmount anything. The problem existed before I did the --exec-opt native.cgroupdriver=cgroupfs fix above. How can I troubleshoot further?

johnjelinek commented Dec 3, 2015

I don't think so -- I didn't unmount anything. The problem existed before I did the --exec-opt native.cgroupdriver=cgroupfs fix above. How can I troubleshoot further?

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Dec 3, 2015

Member

@johnjelinek hm actually, it looks like #17879, could be related to some API changes w.r.t default values, but not sure

Member

thaJeztah commented Dec 3, 2015

@johnjelinek hm actually, it looks like #17879, could be related to some API changes w.r.t default values, but not sure

@johnjelinek

This comment has been minimized.

Show comment
Hide comment
@johnjelinek

johnjelinek Dec 3, 2015

I manually created this file: /sys/fs/cgroup/memory/system.slice/docker/8983d7cd867be40dcfd68747c79fac2f713243a979c3011dc1997dcdf8665150/memory.swappiness because that's what was complaining. When I did docker-compose up -d again, the 898...150 folder went missing. I guess it was deleted when the container was recreated.

johnjelinek commented Dec 3, 2015

I manually created this file: /sys/fs/cgroup/memory/system.slice/docker/8983d7cd867be40dcfd68747c79fac2f713243a979c3011dc1997dcdf8665150/memory.swappiness because that's what was complaining. When I did docker-compose up -d again, the 898...150 folder went missing. I guess it was deleted when the container was recreated.

@johnjelinek

This comment has been minimized.

Show comment
Hide comment
@johnjelinek

johnjelinek Dec 3, 2015

@thaJeztah: I think you're right. It's likely #17879.

johnjelinek commented Dec 3, 2015

@thaJeztah: I think you're right. It's likely #17879.

@marineam

This comment has been minimized.

Show comment
Hide comment
@marineam

marineam May 13, 2016

Just for the sake of completeness since I didn't see it explicitly stated, for recent versions of systemd that use init.scope for pid 1 libcontainer should not be including it in this join: https://github.com/opencontainers/runc/blob/master/libcontainer/cgroups/systemd/apply_systemd.go#L384

marineam commented May 13, 2016

Just for the sake of completeness since I didn't see it explicitly stated, for recent versions of systemd that use init.scope for pid 1 libcontainer should not be including it in this join: https://github.com/opencontainers/runc/blob/master/libcontainer/cgroups/systemd/apply_systemd.go#L384

@marineam

This comment has been minimized.

Show comment
Hide comment
@marineam

marineam May 13, 2016

Instead of assuming things about systemd's cgroup behavior it could be asked instead:

$ busctl get-property org.freedesktop.systemd1 /org/freedesktop/systemd1/unit/system_2eslice org.freedesktop.systemd1.Slice ControlGroup
s "/system.slice"

That should work regardless of systemd version

marineam commented May 13, 2016

Instead of assuming things about systemd's cgroup behavior it could be asked instead:

$ busctl get-property org.freedesktop.systemd1 /org/freedesktop/systemd1/unit/system_2eslice org.freedesktop.systemd1.Slice ControlGroup
s "/system.slice"

That should work regardless of systemd version

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah May 13, 2016

Member

@marineam are you still seeing this on docker 1.11.1? We're using runC now for the runtime, and a lot has changed since this was reported

Member

thaJeztah commented May 13, 2016

@marineam are you still seeing this on docker 1.11.1? We're using runC now for the runtime, and a lot has changed since this was reported

@marineam

This comment has been minimized.

Show comment
Hide comment
@marineam

marineam May 13, 2016

This code is in runc and unchanged since docker 1.10 but I haven't actually tried docker 1.11 yet.

marineam commented May 13, 2016

This code is in runc and unchanged since docker 1.10 but I haven't actually tried docker 1.11 yet.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment