Unable to launch container, error complaining about SHM size must be greater than 0

[derekwaynecarr]

I am getting errors like the following:

$ cluster/kubectl.sh describe pods
Name:       nginx-2040093540-0ietl
Namespace:  default
Node:       127.0.0.1/127.0.0.1
Start Time: Thu, 21 Apr 2016 02:10:40 -0400
Labels:     pod-template-hash=2040093540,run=nginx
Status:     Pending
IP:     
Controllers:    ReplicaSet/nginx-2040093540
Containers:
  nginx:
    Container ID:   
    Image:      nginx
    Image ID:       
    Port:       
    QoS Tier:
      cpu:          BestEffort
      memory:           BestEffort
    State:          Waiting
      Reason:           ContainerCreating
    Ready:          False
    Restart Count:      0
    Environment Variables:  <none>
Conditions:
  Type      Status
  Ready     False 
Volumes:
  default-token-h7ms3:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-h7ms3
Events:
  FirstSeen LastSeen    Count   From            SubobjectPath   Type        Reason          Message
  --------- --------    -----   ----            -------------   --------    ------          -------
  48s       48s     1   {default-scheduler }            Warning     FailedScheduling    no nodes available to schedule pods
  47s       47s     1   {default-scheduler }            Normal      Scheduled       Successfully assigned nginx-2040093540-0ietl to 127.0.0.1
  44s       8s      4   {kubelet 127.0.0.1}         Warning     FailedSync      Error syncing pod, skipping: failed to "StartContainer" for "POD" with RunContainerError: "runContainer: Error response from daemon: SHM size must be greater then 0"

I am running the following:

$ uname -r
4.4.7-300.fc23.x86_64$ 
$ docker version
Client:
 Version:         1.9.1
 API version:     1.21
 Package version: docker-1.9.1-9.gitee06d03.fc23.x86_64
 Go version:      go1.5.3
 Git commit:      ee06d03/1.9.1
 Built:           
 OS/Arch:         linux/amd64

Server:
 Version:         1.9.1
 API version:     1.21
 Package version: docker-1.9.1-9.gitee06d03.fc23.x86_64
 Go version:      go1.5.3
 Git commit:      ee06d03/1.9.1
 Built:           
 OS/Arch:         linux/amd64

Are we now required to set a ShmSize moving forward now that we have switched to the docker engine API? I was able to work around it on my system by supplying ShmSize like the following:

    hc := &dockercontainer.HostConfig{
        Binds:          binds,
        NetworkMode:    dockercontainer.NetworkMode(netMode),
        IpcMode:        dockercontainer.IpcMode(ipcMode),
        UTSMode:        dockercontainer.UTSMode(utsMode),
        PidMode:        dockercontainer.PidMode(pidMode),
        ReadonlyRootfs: readOnlyRootFilesystem(container),
        Resources: dockercontainer.Resources{
            Memory:     memoryLimit,
            MemorySwap: -1,
            CPUShares:  cpuShares,
        },
        ShmSize:     1024 * 1024 * 64,
        SecurityOpt: securityOpts,
    }

which I think is consistent with what docker CLI is doing by default.

/cc @kubernetes/sig-node @vishh @Random-Liu @mrunalp @kubernetes/rh-cluster-infra

[derekwaynecarr]

Since this blocks my ability to launch containers, bump to p0. See associated PR that sets ShmSize to match what docker CLI does by default. We could probably get fancier with ShmSize in future.

[Random-Liu]

@derekwaynecarr Thanks a looooot~
ShmSize is added after docker 1.9+, and is documented in docker 1.10.
I suspects that it's added after 1.9.1(I'll find the specific PR later). If so, the reason may be that the docker version in our e2e cluster is different from yours. :)

As is documented:

ShmSize - Size of /dev/shm in bytes. The size must be greater than 0. If omitted the system uses 64MB.

Go-dockerclient doesn't have this issue because it doesn't even support ShmSize now see, so the filed is omitted during json encoding.

While the engine-api supports ShmSize see, and the field is not omitted even when it is empty - 0, which causes this problem.

[Random-Liu]

@derekwaynecarr My theory is proved. :)
In our ContainerVM, we are using the earliest docker 1.9.1:
moby/moby@a34a1d5

While the ShmSize PR got in after that:
moby/moby#16168

The version you are using is quite new:
moby/moby@ee06d03

That's why we didn't catch this in our e2e cluster.

When I validated docker 1.10, we were still using go-dockerclient, which doesn't have this problem (see #24588 (comment)).

I think we should:

1. Set default ShmSize value if it is empty in our docker client wrapper - kube_docker_client
2. File an issue to docker engine-api to see whether they can set ShmSize to be "omitempty".

For older docker version (before 1.9.1), an extra field won't matter, it will be automatically omitted during json decoding.

Also Ref #19720, #23563

[ncdc]

We have a new Fedora Docker RPM pending to fix this.

On Thursday, April 21, 2016, Lantao Liu notifications@github.com wrote:

@derekwaynecarr https://github.com/derekwaynecarr
In our ContainerVM, we are using the earliest docker 1.9.1:
moby/moby@a34a1d5 moby/moby@a34a1d5

While the ShmSize PR gets in after that:
moby/moby#16168 moby/moby#16168

The version you are using quite new:
moby/moby@ee06d03 moby/moby@ee06d03

When I validated docker 1.10, we were still using go-dockerclient, which
doesn't have this problem (see #24588 (comment)
#24588 (comment)
).

—
You are receiving this because you are on a team that was mentioned.
Reply to this email directly or view it on GitHub
#24588 (comment)

[derekwaynecarr]

@Random-Liu - I am fine with option 1, I will update this PR.

[derekwaynecarr]

Per discussion this morning with @eparis and @ncdc , I am not a fan of hidden default flags that have a history of breaking us. As a result, I would like kube to set an explicit default ShmSize using the option 1 proposed by @Random-Liu and I wonder if we should look to expose ShmSize as a per container option in the future.

[Random-Liu]

As @ncdc said in #24589 (comment)

It seems that docker doesn't accept ShmSize of 0 from moby/moby#18261, but change to accept that after moby/moby@5190794.

There is no public docker release between them, so this should not happen in any docker released version. Maybe we don't need to care about it too much for now. :)