Documentation for running Che using docker-compose + potentially nginx-proxy & on carina

[danpolanco]

Background: I'm trying to run Che on Carina. I'm also using nginx-proxy, but that's important for #238.
Problem: I can't create a new workspace.

I have the following docker-compose.yml:

che_app:
  image: "codenvy/che:no_dnd"
  restart: "always"
  environment:
    - "VIRTUAL_HOST=sub.yourdomain.org"
    - "VIRTUAL_PORT=8080"
    - "TZ=America/Mountains"
  volumes:
    - "/var/run/docker.sock:/var/run/docker.sock"
  volumes_from:
    - "che_data"

che_data:
  image: "codenvy/che:no_dnd"
  environment:
    - "TZ=America/Mountains"
  volumes:
    - "/home/user/che/lib-copy"
    - "/home/user/che/workspaces"
    - "/home/user/che/tomcat/temp/local-storage"
  entrypoint: "true"

The trouble is I'm seeing [ERROR] org.eclipse.che.plugin.docker.client.DockerException: Error response from docker API, status: 500, message: Cannot start container a9ab4a017b3f7a68dc952eb7440cb964af60900a42ef620e86110368ca7b0c1a: [8] System error: permission denied.

I'm guessing the problem is that by leaving DOCKER_MACHINE_HOST as localhost, Che is unable to get permission to spin up new containers.

I'm considering (and briefly tried a few combinations of):

• adding net: "host"
• adding pid: "host"
• sed -e "s/HOSTIP/$(docker inspect --format '{{ .NetworkSettings.IPAddress }}' swarm-manager)/g" docker-compose.yml | docker-compose --file - u

And suggestions as to how to get past permission denied would be greatly appreciated. I'll post anything I figure out.

Spun out of #257.
I release all rights to the following information (i.e. feel free to use it in the docs if it's useful and correct). I'm not a pro though so take what I say and double check it's true.

[danpolanco]

Update: tried without nginx-proxy and added net: "host" and permission denied still happens.

[danpolanco]

Interesting... The container is being created. docker ps -a reveals:

903f0e8b391e        eclipse-che/che_workspace7ff5fg7ivfou9816_ws-machine_vq29m   "/bin/sh -c 'sudo /us"   31 seconds ago                                                                                            e4db2ccb-67ff-44e7-8862-b9b322211b8e-n1/che_workspace7ff5fg7ivfou9816_ws-machine_vq29m

[TylerJewell]

We have seen issues with vbox where running this container a second time in the same vm can cause problems such as hanging . We have only seen this on Windows though, never on Linux.

[danpolanco]

Ok. Let me try creating a brand new cluster.

[ghost]

What is the host os and Docker version? What is host UID?

[danpolanco]

Before I get to that, I noticed it's printing:

1. Open browser to http://localhost:8080, or:

Instead of the DOCKER_MACHINE_HOST ip I have in the docker-compose.yml. I added USE_DEBUG=true and the output looks the same from docker-compose logs. I also added the environment variables to the editor_data just in case and nothing changed.

Could that be related?

[danpolanco]

docker info

Containers: 8
 Running: 0
 Paused: 0
 Stopped: 0
Images: 4
Role: primary
Strategy: spread
Filters: health, port, dependency, affinity, constraint
Nodes: 1
 09a2bff8-2b32-40bf-a4bd-928bf8ef5715-n1: 172.99.73.168:42376
  └ Containers: 8
  └ Reserved CPUs: 0 / 12
  └ Reserved Memory: 1000 MiB / 4.2 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=3.18.21-1-rackos, operatingsystem=Debian GNU/Linux 7 (wheezy) (containerized), storagedriver=aufs
Plugins: 
 Volume: 
 Network: 
CPUs: 12
Total Memory: 4.2 GiB
Name: 6a222652137d
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: No cpu shares support
WARNING: No cpuset support

[danpolanco]

Server:
 Version:      swarm/1.0.0
 API version:  1.21
 Go version:   go1.5.1
 Git commit:   087e245
 Built:        
 OS/Arch:      linux/amd64

[danpolanco]

Updated for @eivantsov. I'm not sure if that has the UUID though? If you need more info, let me know what command to run :)

[danpolanco]

Getting a wee bit tired, but at the moment I think the issue is the data container.

I thought by using the data container I could get past cannot create directory '/home/user/che/lib-copy/': Permission denied but that was a silly assumption. Tomorrow I'll use a data container that che_app can write to.

[ghost]

Yes, it is important that when doing mounts a host directory can be created. It is used then by a workspace container. ws-agent and terminal are mounted into a workspace container, and it is only after a ws-agent successfully starts that a workspace is considered as running.

[danpolanco]

Ok. So perhaps che.sh changes the permissions back to root:root then?

[danpolanco]

Nvm. It doesn't look like that's the problem. Still looking...

[danpolanco]

He is the full report just incase it's helpful to anyone:
http://pastebin.com/DARDxM1j

[danpolanco]

Important

➜  test-cluster  docker run -it --rm --volumes-from cf8479fb4e6c codenvy/ubuntu_jdk8 /bin/bash
Timestamp: 2016-02-05 17:20:16.070706608 +0000 UTC
Code: System error

Message: permission denied

Frames:
---
0: setupRootfs
Package: github.com/opencontainers/runc/libcontainer
File: rootfs_linux.go@40
---
1: Init
Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go@57
---
2: StartInitialization
Package: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
File: factory_linux.go@242
---
3: initializer
Package: github.com/docker/docker/daemon/execdriver/native
File: init.go@35
---
4: Init
Package: github.com/docker/docker/pkg/reexec
File: reexec.go@26
---
5: main
Package: main
File: docker.go@18
---
6: main
Package: runtime
File: proc.go@63
---
7: goexit
Package: runtime
File: asm_amd64.s@2232
Error response from daemon: Cannot start container d960163005c317719ee7dc44daa4d5152d614c6a831721067626a30abfdd500f: [8] System error: permission denied

[danpolanco]

So it's a Carina specific issue: https://getcarina.com/docs/troubleshooting/common-problems/

[TylerJewell]

Quite the thread. In reading the Carina issue, looks like you have some workarounds. Are you unblocked?

[danpolanco]

Hopefully the length of the thread isn't a problem? I've just been trying to keep track of my progress.

I haven't been able to get it up and running yet. I was just about to post on the Carina Forums to see if they know what is up.

I just ran docker inspect and it shows a few mounts plus all the other settings. It is my next lead. I'm trying to figure out what volume is causing [8] System error: permission denied or if it isn't a volume, why it's having permission issues.

If you'd like me to post less, let me know. Also feel free to close this issue if needed :)

Edit: Carina forum post: https://community.getcarina.com/t/running-eclipse-che-on-carina-error-response-from-docker-api-status-500-message-cannot-start-container/111

[TylerJewell]

Oh, I love the commentary. It's a reflection that there is real engagement + usage + value in a very new product. That is essential to the future success. I like having a record of the activity, and I just updated our Contribution.MD to give people additional pointers on how to file tickets around issues so that they provide more data, not less.

[TylerJewell]

I assume that there is closure and success??

[danpolanco]

Haha that was fast. Not yet. Carina isn't at 1.10. I'm going to come back the issue once it is.