sudo: unable to send audit message: Operation not permitted (Ubuntu 14.04)

[rarkins]

I'm using docker compose, here is the config YAML:

  che:
    image: codenvy/che:latest
    command: >
      bash -c "
      sudo rm -rf /home/user/che/lib-copy/* &&
      sudo rm -rf /home/user/che/tomcat/temp/local-storage/* &&
      mkdir -p /home/user/che/lib-copy/ &&
      sudo chown -R user:user /home/user/che &&
      cp -rf /home/user/che/lib/* /home/user/che/lib-copy &&
      /home/user/che/bin/che.sh run -d
      "
    network_mode: "host"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "/home/user/che/lib:/home/user/che/lib-copy"
      - "/home/user/che/workspaces:/home/user/che/workspaces"
      - "/home/user/che/tomcat/temp/local-storage:/home/user/che/tomcat/temp/local-storage"
    container_name: che

Note that I'm using version 2 of the compose yml so instead of net: host it's now network_mode: host.

I'm running a standard Ubuntu 14.04 server VM on EC2 with Docker 1.10.3 installed.

Here's what I see:

ubuntu@rarkins2:/app$ docker version
Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64
ubuntu@rarkins2:/app$
ubuntu@rarkins2:/app$ ls -l /var/run/docker.sock
srwxrwxrwx 1 root docker 0 May 15 17:15 /var/run/docker.sock
ubuntu@rarkins2:/app$ docker-compose up che
Starting che
Attaching to che
che               | sudo: unable to send audit message: Operation not permitted
che               | bash: line 1:     6 Segmentation fault      (core dumped) sudo rm -rf /home/user/che/lib-copy/*
che exited with code 139

It seems like it's a combination of problems with Ubuntu 14.04, host networking, and sudo. Removing host networking instead gets the Running 'docker' succeeded, but 'docker ps' failed. This usually means that docker cannot reach its daemon. error.

[TylerJewell]

@rarkins Hi. As a guide on what we do to run Docker within a Vagrantfile, you can take a look at the bash script within this file. It gives you a sense of the steps that we take to configure a Linux host to run the Docker container properly. There is some setup of the groups and file permissions.

Typically, you get the docker ps issue when rights to /var/run/docker.sock are not setup in a way that we can use it. Could you chmod 777 /var/run/docker.sock and rerun the compose file, letting us know how it goes?

https://github.com/eclipse/che/blob/2fca5b224ad3cf940556ec0d8e3659f472a76a2c/Vagrantfile

[rarkins]

@TylerJewell I ran chmod on docker.sock previously and you can check the permissions in my above bash log actually - looks correct to you?

[TylerJewell]

@rarkins -- will you humor me and run the pull request Vagrantfile. Is this successful for you? If so, then I think I see some errors in the compose file format that I'll need to work on. This Vagrantfile has the Linux syntax for running Docker Che. It's a bit more manual than compose, but if this file works then we have to back track to review the compose file. I might even attempt a Vagrantfile that launches compose instead of Docker run to verify the syntax.

https://github.com/eclipse/che/pull/1279/files

[TylerJewell]

@rarkins - Hi. I have been working on using Docker compose with our nightly build. We have made some adjustments. This compose file was able to work for me. I'll also publish a Vagrantfile so that you can see the syntax that we use to create the VM to run this in as well. In this particular example, my VM was IP address of 192.168.28.100, so you would replace that value with the IP of the machine you are on.

 che:
    image: codenvy/che:nightly
    command: --remote:192.168.28.100 --skip:client run
    net: "host"
    restart: always
    environment:
      CHE_LOCAL_CONF_DIR: /container
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /home/user/che/lib:/home/user/che/lib-copy
      - /home/user/che/workspaces:/home/user/che/workspaces
      - /home/user/che/storage:/home/user/che/storage
      - /home/user/che/che.properties:/container/che.properties
    container_name: che

[TylerJewell]

@rarkins - I have also created a Vagrantfile which will run this syntax. This is using the nightly build of Eclipse Che, which is 4.3 - due to be released soon.
https://eclipse-che.readme.io/v4.3/docs/usage-docker#section-docker-compose

[rarkins]

@TylerJewell I really appreciate your updates on this.

First of all I thought I'd try using the updated docker-compose definition above. Essentially the same result:

$ sudo chmod 777 /var/run/docker.sock
$ ls -l /var/run/docker.sock
srwxrwxrwx 1 root docker 0 May 20 05:05 /var/run/docker.sock
ubuntu@rarkins:/app$ docker-compose up che
che is up-to-date
Attaching to che
che                |
che                | !!!
che                | !!! Running 'docker' succeeded, but 'docker ps' failed. This usually means that docker cannot reach its daemon. On Mac and Linux, check the read / write permissions on '/var/run/docker.sock'. Consider running 'sudo chmod 777 /var/run/docker.sock'.
che                | !!!

By the way, I wasn't 100% sure if the IP address in command should be the public IP of this EC2 VM, or the docker0 IP, or eth0, so I tried them all of them and got the same result each time.

[rarkins]

@TylerJewell I was able to run your Vagrantfile on my Mac. In addition, I switched the box to ubuntu/trusty64 and modified the Docker install script to suit and it still worked. i.e. I have che running inside an Ubuntu 14.04 VM via Vagrant. So now to figure out why it fails on an EC2 Ubuntu VM.

[ghost]

@rarkins what you can do it:

docker run -ti --net=host -v /var/run/docker.sock:/var/run/docker.sock codenvy/che bash

You will have a shell in a container. Run docker ps there. Response from Docker should be more informative.

Permissions for /var/run/docker.sock look ok for me. I wonder if it's different Docker versions (host vs container). Can you update Docker to the latest on your EC2 instance?

[rarkins]

I think I've been able to progress this further by switching my VM's docker version from 1.10.3 to 1.11.3. However it's starting to mess with my regular config so I'm going to spin up a dedicated EC2 instance for this before continuing.

I have some questions:

1. What is the relationship between che and docker versions? e.g. is it that che requires a minimum version x, or that che requires any system it runs on to have the exact same version of docker?
2. For this use case (running remotely on EC2 instance, with other containers in default bridge mode), what value for --remote:1.1.1.1 should be used? e.g. docker0, eth0, the EC2 public IP, etc?

[rarkins]

Also, after performing a sudo reboot I realised that running chmod on /var/run/docker.sock is not a persistent solution so ideally there is an alternative.

[TylerJewell]

@rarkins:

We currently require 1.8.x+ of a Docker version for Che. But we will be moving to 1.11.x soon for stability reasons. We certainly make use of certain Docker API syntax that depends upon a version. But docker ps issues generally means that something is wrong with the configuration of the Docker daemon. But given what @eivantsov has said, wondering if I should add a minimum Docker daemon version check into our core launching scripts.

For the --remote command, it requires the IP address that you would want your browser clients to use when connecting to a workspace. So it's the public IP, not docker0 or etho. A lot of this information is included in our networking docs. It's a long page, but it maps it all out.

[ghost]

@rarkins We mount /var/run/docker.sock.. so when Docker in a container is invoked it triggers Docker on the host. If Docker versions on the host and in the container differ, that might be a problem.

@TylerJewell for Che there are no Docker version requirements. 1.8+ is ok

[ghost]

@rarkins @TylerJewell I found this issue - moby/moby#5899

Looks like we have hit it with Che, using --net=host

[ghost]

--pid=host solved it for me.

[TylerJewell]

@rarkins - if this resolves your issue, then we will update our various compose file, docker run docs, and the Vagrantfiles that make use of this.

This is the first time I have seen this issue surface.

[rarkins]

@TylerJewell I think that the originally reported issue (sudo problem) was resolved by the new approach @eivantsov is taking in version 4.3. Therefore I'm not sure there's a need for any update apart from that.

The issue discussion kind of evolved though and my current challenge is getting che-docker running on a "vanilla" Ubuntu 14.04 VM on EC2, which doesn't seem to work with the default instructions. Perhaps this:

If Docker versions on the host and in the container differ, that might be a problem.

I'm fine if this issue is closed though.

[TylerJewell]

We made this commit this morning for a new che.sh script that will make a test to see if the versions are different after docker ps fails. The "nightly" docker container image for che was built with this new script, so if you run the nightly image now, it should give you a better diagnostic.

[rarkins]

I've been able to get this working on an EC2 Ubuntu 14.04 server as originally intended. Steps:

1. Allow ports 8080 and 22
2. Install latest docker engine
3. change permissions on /var/run/docker.sock
4. Then this exact command:

$ docker run --net=host \
           --name che \
           -p 8080:8080 \
           -v /var/run/docker.sock:/var/run/docker.sock  \
           -v /home/user/che/lib:/home/user/che/lib-copy  \
           -v /home/user/che/workspaces:/home/user/che/workspaces  \
           -v /home/user/che/storage:/home/user/che/storage  \
           codenvy/che:nightly --remote:52.50.157.12

I think this can therefore be closed now. Thanks to everyone for the assistance.

[ghost]

When me start kali nethunter show a error "sudo: unable to send audit message: Operation not permitted"
nh
sudo: unable to send audit message: Operation not permitted
┌──(kali㉿localhost)-[~]
└─$