permission denied to /var/run/docker.sock

[moix]

Hi,

today I tried to install cluster-insight on my minions but it is failing with a permission denied to docker.sock.

I'm running latest docker version which I've realized that come now with --selinux-enabled option set by default. Just a guess, Can it be related? any clue how to fix it?

Thanks!

[supriyagarg]

Hi moix,

Can you please send more details about your environment? Especially OS, and kubernetes/docker versions.

Also, please try to run this command manually on a minion node, and see if it works:

sudo docker run -priviledged -d --net=host -p 4243:4243 --name cluster-insight -e CLUSTER_INSIGHT_MODE=minion -v /var/run/docker.sock:/var/run/docker.sock:ro kubernetes/cluster-insight

[EranGabber]

Moix:
If you run the minion collector in the way that Supriya recommended, the container you are running is not a part of any pod. This may trigger a failure of the cluster insight master as explained in #70.

Once you run the minion collector, try to access it from the same VM using the command:

curl http://localhost:4243/containers/json

It should show a few lines of JSON output and it should not fail. If it fails, please report the error message.

[moix]

Hi supreyagarg/EranGabber,

yes thanks! it was indeed a problem with selinux and privilegies for docker containers. I could make it work with the following to options but both running out of a pod so now facing the issue you pointed, #70:

{
  "error_message": "\"u'missing or invalid parent pod ID in container drunk_franklin'\"", 
  "success": false, 
  "timestamp": "2015-06-17T09:19:00.804892"
}

With these 2 commands cluster-insight runs fine, curl now responds fine:

• docker run -d --security-opt label:type:docker_t ...
• docker run -priviledged ...

Now trying to know how to include one of these 2 options in the pod spec, any clue? have tried some combinations but cannot make it work.

[moix]

Alright, setting --allow_privileged=true in kubelet in minions and adding "privileged": true, to the ReplicationController definition did the trick and now is running withing pods.

I would prefer obviously to set --security-opt label:type:docker_t in the pod spec but couldnt find how to do it, Any idea?

Thanks!

[EranGabber]

moix,

I will try to update the specification and push a new version to Github soon.

[supriyagarg]

Moix,

Can you please tell us about the rest of your environment, so we can replicate the issue.
The cluster-insight on minions is running fine on our test clusters (without running in privileged mode), and they use the latest docker version (1.6.0)

Thanks.

[supriyagarg]

Moix,

Please add the following line to the pod spec, exactly where you had added the "privileged": true, option

"security-opt": "label:type:docker_t",

Let us know if this works.

[moix]

Sorry, was not in my laptop when reported the issue and couldnt detail the environment. It is a setup on 3 centos-7 servers, one master and 2 minions. Version of the packages are:

kubernetes-master-0.17.1-3.el7.x86_64
kubernetes-node-0.17.1-3.el7.x86_64
kubernetes-0.17.1-3.el7.x86_64
flannel-0.2.0-7.el7.x86_64
docker-master-1.7.0-4.git56481a3.el7.x86_64

this docker runs with --selinux-enabled option enabled by default, not sure if already in docker 1.6 as well.

More information about the environment you can check at kubernetes/kubernetes#9580

---

Sure, I'll try "security-opt": "label:type:docker_t", and let you know. I tried with something like:
"securityContext": { "seLinuxOptions": { "type": "docker_t" } }, but did not work.

[supriyagarg]

Moix - I finally got around to setting up a CentOS kubernetes cluster. The option you specified ("securityContext": { "seLinuxOptions": { "type": "docker_t" } },) works fine for me. Can you try once more? Maybe some formatting issue?

I added this line right between "image" and "resources" in the containers spec.

Thanks.

[EranGabber]

Moix,

Supriya just updated the collector/cluster-insight-controller.json file in the repository. Please fetch it from https://github.com/google/cluster-insight and try again.

Thanks.

[moix]

Hi,

nop, master is not able to deploy pods, error in log is (https://github.com/GoogleCloudPlatform/kubernetes/blob/release-0.17/pkg/controller/controller_utils.go):

Jun 19 03:14:54 kubernetes-master kube-controller-manager: E0619 03:14:54.864560     554 replication_controller.go:310] unable to create pod replica: pods "" is forbidden: SecurityContext.SELinuxOptions is forbidden

I've tried to investigate where it comes and how to enable but sorry, just a beginner in kubernetes :) I guess kubelet service in minions should run with some option to allow security context modifications, https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/design/security_context.md

Will continue looking for a solution and let you know if I find the proper configuration.

Thanks!

[EranGabber]

Moix,

We changed the configuration of the Cluster-Insight collector. It is now a service with its own replication controller. Could you try to install it again with the latest installation script and the latest container image?

Thanks,

Eran

[moix]

Sure, I'm not at the office now but will try as soon as I'm back.

Thanks, Moix.

[moix]

Sorry for the delay, yeah now worked fine! thanks @EranGabber