Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Privileged Containers fail in LXD with lstat /dev/.lxc/proc: no such file or directory Error #32968

Closed
caleblloyd opened this issue May 2, 2017 · 9 comments

Comments

Projects
None yet
5 participants
@caleblloyd
Copy link

commented May 2, 2017

Description

Privileged Containers fail in LXD with error linux runtime spec devices: lstat /dev/.lxc/proc/#/fdinfo/#: no such file or directory.

Steps to reproduce the issue:

  1. Create a privileged LXD container:
root@prec:~# lxc launch ubuntu:16.04 docker -p default -p docker
Creating docker
Starting docker
root@prec:~# lxc config set docker security.privileged true
root@prec:~# lxc restart docker
root@prec:~# lxc exec -t docker bash
  1. Install Docker in the LXD Container
curl -sSL https://get.docker.com/ | sh
  1. Try to start a privileged container:
root@docker:~# docker run --privileged --rm alpine bash
docker: Error response from daemon: linux runtime spec devices: lstat /dev/.lxc/proc/3114/fdinfo/16: no such file or directory.

Describe the results you received:

Error response from daemon: linux runtime spec devices: lstat /dev/.lxc/proc/3114/fdinfo/16: no such file or directory.

Describe the results you expected:

A running docker container

Additional information you deem important (e.g. issue happens only occasionally):

I think this is the same issue reported at Rancher here: rancher/rancher#7968, and fixed in Runc here: opencontainers/runc#1327

If that is the case, then the Runc commit needs to be updated in moby

Output of docker version:

root@docker:~# docker version
Client:
 Version:      17.04.0-ce
 API version:  1.28
 Go version:   go1.7.5
 Git commit:   4845c56
 Built:        Mon Apr  3 18:07:42 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.04.0-ce
 API version:  1.28 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   4845c56
 Built:        Mon Apr  3 18:07:42 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 17.04.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 3
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: 
containerd version: 422e31ce907fd9c3833a38d7b8fdd023e5a76e73
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.10.0-15-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.4GiB
Name: docker
ID: 7LKZ:SCVT:QRNM:TJJ4:EKRS:QCLC:QDRJ:54IU:HE23:ROCY:4ZWK:WEPI
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Additional environment details (AWS, VirtualBox, physical, etc.):

Ubuntu 16.04 running LXD

@caleblloyd

This comment has been minimized.

Copy link
Author

commented May 15, 2017

Works in 17.06-dev

@ronaldpetty

This comment has been minimized.

Copy link

commented Jun 7, 2017

@caleblloyd sorry to add unrelated question, but how did you install 17.06-dev? I can't seem to figure it out via the normal instructions.

root@nodeg:~# history
    1  apt-get update
    2  sudo apt-get install     apt-transport-https     ca-certificates     curl     software-properties-common
    3  curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
    4  sudo add-apt-repository    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   edge"
    5  sudo apt-get update
    6  apt-cache madison docker-ce
    7  sudo apt-get install docker-ce=17.06-dev
root@nodeg:~# sudo apt-get install docker-ce=17.06-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Version '17.06-dev' for 'docker-ce' was not found
root@nodeg:~#
@caleblloyd

This comment has been minimized.

Copy link
Author

commented Jun 7, 2017

@ronaldpetty cloned the repository and built from source. 17.06-ce should be out any day now though​.

@ronaldpetty

This comment has been minimized.

Copy link

commented Jun 10, 2017

I did not build from source, but used RC2, I appear to still be stuck.

user@ubuntu:~$ lxc exec nodea bash
root@nodea:~# sudo apt-get install \
> apt-transport-https \
> ca-certificates \
> curl \
> software-properties-common
Reading package lists... Done
Building dependency tree       
Reading state information... Done
ca-certificates is already the newest version (20160104ubuntu1).
apt-transport-https is already the newest version (1.2.20).
curl is already the newest version (7.47.0-1ubuntu2.2).
software-properties-common is already the newest version (0.96.20.6).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@nodea:~# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
OK
root@nodea:~# sudo apt-key fingerprint 0EBFCD88
pub   4096R/0EBFCD88 2017-02-22
      Key fingerprint = 9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
uid                  Docker Release (CE deb) <docker@docker.com>
sub   4096R/F273FCD8 2017-02-22

root@nodea:~# sudo add-apt-repository \
> "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
> $(lsb_release -cs) \
> test"
root@nodea:~# sudo apt-get update
Get:1 https://download.docker.com/linux/ubuntu xenial InRelease [29.5 kB]
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB] 
Hit:3 http://archive.ubuntu.com/ubuntu xenial InRelease 
Get:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease [102 kB]           
Get:5 https://download.docker.com/linux/ubuntu xenial/test amd64 Packages [3,166 B]
Get:6 http://security.ubuntu.com/ubuntu xenial-security/main Sources [76.2 kB]             
Get:7 http://archive.ubuntu.com/ubuntu xenial-backports InRelease [102 kB]                    
Get:8 http://security.ubuntu.com/ubuntu xenial-security/restricted Sources [2,604 B]
Get:9 http://archive.ubuntu.com/ubuntu xenial/main Sources [868 kB]               
Get:10 http://security.ubuntu.com/ubuntu xenial-security/universe Sources [30.9 kB]
Get:11 http://security.ubuntu.com/ubuntu xenial-security/multiverse Sources [1,144 B]
Get:12 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages [284 kB]
Get:13 http://archive.ubuntu.com/ubuntu xenial/restricted Sources [4,808 B]    
Get:14 http://security.ubuntu.com/ubuntu xenial-security/main Translation-en [121 kB]
Get:15 http://archive.ubuntu.com/ubuntu xenial/universe Sources [7,728 kB]
Get:16 http://security.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages [7,420 B]
Get:17 http://security.ubuntu.com/ubuntu xenial-security/restricted Translation-en [2,428 B]
Get:18 http://security.ubuntu.com/ubuntu xenial-security/universe amd64 Packages [131 kB] 
Get:19 http://security.ubuntu.com/ubuntu xenial-security/universe Translation-en [67.5 kB]
Get:20 http://security.ubuntu.com/ubuntu xenial-security/multiverse amd64 Packages [2,752 B]
Get:21 http://security.ubuntu.com/ubuntu xenial-security/multiverse Translation-en [1,232 B]
Get:22 http://archive.ubuntu.com/ubuntu xenial/multiverse Sources [179 kB]
Get:23 http://archive.ubuntu.com/ubuntu xenial/universe amd64 Packages [7,532 kB]
Get:24 http://archive.ubuntu.com/ubuntu xenial/universe Translation-en [4,354 kB]
Get:25 http://archive.ubuntu.com/ubuntu xenial/multiverse amd64 Packages [144 kB]
Get:26 http://archive.ubuntu.com/ubuntu xenial/multiverse Translation-en [106 kB]
Get:27 http://archive.ubuntu.com/ubuntu xenial-updates/main Sources [252 kB]
Get:28 http://archive.ubuntu.com/ubuntu xenial-updates/restricted Sources [3,000 B]
Get:29 http://archive.ubuntu.com/ubuntu xenial-updates/universe Sources [157 kB]                                                                                                                                                             
Get:30 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse Sources [5,672 B]                                                                                                                                                          
Get:31 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages [552 kB]                                                                                                                                                          
Get:32 http://archive.ubuntu.com/ubuntu xenial-updates/main Translation-en [224 kB]                                                                                                                                                          
Get:33 http://archive.ubuntu.com/ubuntu xenial-updates/restricted amd64 Packages [7,772 B]                                                                                                                                                   
Get:34 http://archive.ubuntu.com/ubuntu xenial-updates/restricted Translation-en [2,548 B]                                                                                                                                                   
Get:35 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages [477 kB]                                                                                                                                                      
Get:36 http://archive.ubuntu.com/ubuntu xenial-updates/universe Translation-en [188 kB]                                                                                                                                                      
Get:37 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse amd64 Packages [8,928 B]                                                                                                                                                   
Get:38 http://archive.ubuntu.com/ubuntu xenial-updates/multiverse Translation-en [4,460 B]                                                                                                                                                   
Get:39 http://archive.ubuntu.com/ubuntu xenial-backports/main Sources [3,304 B]                                                                                                                                                              
Get:40 http://archive.ubuntu.com/ubuntu xenial-backports/universe Sources [4,052 B]                                                                                                                                                          
Get:41 http://archive.ubuntu.com/ubuntu xenial-backports/main amd64 Packages [4,684 B]                                                                                                                                                       
Get:42 http://archive.ubuntu.com/ubuntu xenial-backports/main Translation-en [3,216 B]                                                                                                                                                       
Get:43 http://archive.ubuntu.com/ubuntu xenial-backports/universe amd64 Packages [5,624 B]                                                                                                                                                   
Get:44 http://archive.ubuntu.com/ubuntu xenial-backports/universe Translation-en [2,872 B]                                                                                                                                                   
Fetched 23.9 MB in 8s (2,687 kB/s)                                                                                                                                                                                                           
Reading package lists... Done
root@nodea:~# apt-cache search docker-ce
docker-ce - Docker: the open-source application container engine
root@nodea:~# apt-cache madison docker-ce
 docker-ce | 17.06.0~ce~rc2-0~ubuntu | https://download.docker.com/linux/ubuntu xenial/test amd64 Packages
 docker-ce | 17.06.0~ce~rc1-0~ubuntu | https://download.docker.com/linux/ubuntu xenial/test amd64 Packages
 docker-ce | 17.05.0~ce~rc3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/test amd64 Packages
 docker-ce | 17.05.0~ce~rc2-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/test amd64 Packages
 docker-ce | 17.05.0~ce~rc1-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/test amd64 Packages
 docker-ce | 17.04.0~ce~rc2-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/test amd64 Packages
 docker-ce | 17.04.0~ce~rc1-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/test amd64 Packages
 docker-ce | 17.03.2~ce~rc1-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/test amd64 Packages
 docker-ce | 17.03.1~ce~rc1-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/test amd64 Packages
root@nodea:~# apt-get install 17.06.0~ce~rc2-0~ubuntu
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package 17.06.0~ce~rc2-0~ubuntu
E: Couldn't find any package by glob '17.06.0~ce~rc2-0~ubuntu'
E: Couldn't find any package by regex '17.06.0~ce~rc2-0~ubuntu'
root@nodea:~# apt-get install docker-ce=17.06.0~ce~rc2-0~ubuntu
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  aufs-tools cgroupfs-mount libltdl7
Suggested packages:
  mountall
The following NEW packages will be installed:
  aufs-tools cgroupfs-mount docker-ce libltdl7
0 upgraded, 4 newly installed, 0 to remove and 23 not upgraded.
Need to get 20.4 MB of archives.
After this operation, 96.5 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://archive.ubuntu.com/ubuntu xenial/universe amd64 aufs-tools amd64 1:3.2+20130722-1.1ubuntu1 [92.9 kB]
Get:2 https://download.docker.com/linux/ubuntu xenial/test amd64 docker-ce amd64 17.06.0~ce~rc2-0~ubuntu [20.3 MB]
Get:3 http://archive.ubuntu.com/ubuntu xenial/universe amd64 cgroupfs-mount all 1.2 [4970 B]
Get:4 http://archive.ubuntu.com/ubuntu xenial/main amd64 libltdl7 amd64 2.4.6-0.1 [38.3 kB]
Fetched 20.4 MB in 2s (9237 kB/s)                                     
Selecting previously unselected package aufs-tools.
(Reading database ... 25504 files and directories currently installed.)
Preparing to unpack .../aufs-tools_1%3a3.2+20130722-1.1ubuntu1_amd64.deb ...
Unpacking aufs-tools (1:3.2+20130722-1.1ubuntu1) ...
Selecting previously unselected package cgroupfs-mount.
Preparing to unpack .../cgroupfs-mount_1.2_all.deb ...
Unpacking cgroupfs-mount (1.2) ...
Selecting previously unselected package libltdl7:amd64.
Preparing to unpack .../libltdl7_2.4.6-0.1_amd64.deb ...
Unpacking libltdl7:amd64 (2.4.6-0.1) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../docker-ce_17.06.0~ce~rc2-0~ubuntu_amd64.deb ...
Unpacking docker-ce (17.06.0~ce~rc2-0~ubuntu) ...
Processing triggers for libc-bin (2.23-0ubuntu7) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (229-4ubuntu17) ...
Setting up aufs-tools (1:3.2+20130722-1.1ubuntu1) ...
Setting up cgroupfs-mount (1.2) ...
Setting up libltdl7:amd64 (2.4.6-0.1) ...
Setting up docker-ce (17.06.0~ce~rc2-0~ubuntu) ...
Processing triggers for libc-bin (2.23-0ubuntu7) ...
Processing triggers for systemd (229-4ubuntu17) ...
Processing triggers for ureadahead (0.100.0-19) ...
root@nodea:~# docker version
Client:
 Version:      17.06.0-ce-rc2
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   402dd4a
 Built:        Wed Jun  7 10:04:47 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.06.0-ce-rc2
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   402dd4a
 Built:        Wed Jun  7 10:03:40 2017
 OS/Arch:      linux/amd64
 Experimental: false
root@nodea:~# docker container run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
78445dd45222: Pull complete 
Digest: sha256:c5515758d4c5e1e838e9cd307f6c6a0d620b5e07e6f927b07d05f6d12a1ac8d7
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: Could not check if docker-default AppArmor profile was loaded: open /sys/kernel/security/apparmor/profiles: permission denied.
ERRO[0002] error waiting for container: context canceled 
root@nodea:~#
@thaJeztah

This comment has been minimized.

Copy link
Member

commented Jun 10, 2017

Issue looks different?

Error response from daemon:
Could not check if docker-default AppArmor profile was loaded:
open /sys/kernel/security/apparmor/profiles: permission denied
@caleblloyd

This comment has been minimized.

Copy link
Author

commented Jun 10, 2017

@ronaldpetty make sure your LXD profile has all of this stuff. Edit it with lxc profile edit [profile name]

config:
  linux.kernel_modules: bridge,br_netfilter,ip_tables,ip6_tables,ip_vs,netlink_diag,nf_nat,overlay,xt_conntrack
  raw.lxc: |-
    lxc.aa_profile = unconfined
    lxc.cgroup.devices.allow = a
    lxc.mount.auto=proc:rw sys:rw
    lxc.cap.drop =
  security.nesting: "true"
  security.privileged: "true"
@caleblloyd

This comment has been minimized.

Copy link
Author

commented Jun 10, 2017

Just tried 17.06 RC 2 with the above profile in an LXD container and everything worked:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo apt-key fingerprint 0EBFCD88
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) test"
sudo apt-get update
apt-cache madison docker-ce
sudo apt-get install docker-ce=17.06.0~ce~rc2-0~ubuntu
sudo docker run --privileged hello-world
@ronaldpetty

This comment has been minimized.

Copy link

commented Jun 10, 2017

@caleblloyd thank you, your modification worked. I am new to LXD so not sure how it all works under. I appreciate the help.

@wmontes

This comment has been minimized.

Copy link

commented Jul 13, 2017

sorry, I had no luck:
sudo docker run --privileged hello-world

container_linux.go:262: starting container process caused "process_linux.go:339: container init caused "rootfs_linux.go:69: creating device nodes caused \"open /var/lib/docker/vfs/dir/696a6d770ccd1de678dbfabf25343f2970081af86e4b38dd597960953b55412d/dev/tty: no such device or address\"""
docker: Error response from daemon: oci runtime error: container_linux.go:262: starting container process caused "process_linux.go:339: container init caused "rootfs_linux.go:69: creating device nodes caused \"open /var/lib/docker/vfs/dir/696a6d770ccd1de678dbfabf25343f2970081af86e4b38dd597960953b55412d/dev/tty: no such device or address\""".
ERRO[0005] error waiting for container: context canceled

lxc config show master
architecture: x86_64
config:
image.architecture: x86_64
image.description: Ubuntu 16.04 LTS server (20170619.1)
image.os: ubuntu
image.release: xenial
volatile.base_image: 7a7ff654cbd8f5f09bec03aa19d8d7d92649127d18659036a963b1ea63f90d25
volatile.eth0.hwaddr: 00:16:3e:0e:21:91
volatile.eth0.name: eth0
volatile.idmap.base: "0"
volatile.idmap.next: '[]'
volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":100000,"Nsid":0,"Maprange":65536},{"Isuid":false,"Isgid":true,"Hostid":100000,"Nsid":0,"Maprange":65536}]'
volatile.last_state.power: RUNNING
devices: {}
ephemeral: false
profiles:

  • default

$ lxc profile show default
config:
linux.kernel_modules: bridge,br_netfilter,ip_tables,ip6_tables,ip_vs,netlink_diag,nf_nat,overlay,xt_conntrack
raw.lxc: |-
lxc.aa_profile = unconfined
lxc.cgroup.devices.allow = a
lxc.mount.auto=proc:rw sys:rw
lxc.cap.drop =
security.nesting: "true"
security.privileged: "true"
description: Default LXD profile
devices:
eth0:
nictype: bridged
parent: lxdbr0
type: nic
root:
path: /
pool: pan
type: disk
name: default

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.