lxc-attach stopped working since 0.6.4

[kiorky]

I dont know why and how, but i was used to use lxc-attach to run an arbitrary command on a running docker container. i get a "lxc-attach: failed to get the init pid" error

root@judith:/var/lib/docker/containers# lxc-attach -n 535c7ada817bd8e2d75b5e0c597ebccf310f6e76ca2e0c5b814d51d77bbf688c -- bash
lxc-attach: failed to get the init pid

I dont know if it can be a clue, but it seems lxc-attach didnt find a socket:

root@judith:/var/lib/docker/containers# docker ps
ID                  IMAGE                         COMMAND                CREATED             STATUS              PORTS
535c7ada817b        makinacorpus/upstart:latest   /sbin/init             15 minutes ago      Up 15 minutes                

root@judith:/var/lib/docker/containers# strace lxc-attach -n 535c7ada817bd8e2d75b5e0c597ebccf310f6e76ca2e0c5b814d51d77bbf688c  /bin
/bash 2>&1
execve("/usr/bin/lxc-attach", ["lxc-attach", "-n", "535c7ada817bd8e2d75b5e0c597ebccf"..., "/bin/bash"], [/* 23 vars */]) = 0
brk(0)                                  = 0x7fd426402000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd4252c4000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=166397, ...}) = 0
mmap(NULL, 166397, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fd42529b000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/liblxc.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\246\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=329120, ...}) = 0
mmap(NULL, 2424448, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fd424e54000
mprotect(0x7fd424ea2000, 2097152, PROT_NONE) = 0
mmap(0x7fd4250a2000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4e000) = 0x7fd4250a2000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\37\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1848024, ...}) = 0
mmap(NULL, 3961912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fd424a8c000
mprotect(0x7fd424c4a000, 2093056, PROT_NONE) = 0
mmap(0x7fd424e49000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bd000) = 0x7fd424e49000
mmap(0x7fd424e4f000, 17464, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fd424e4f000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 \26\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=18952, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd42529a000
mmap(NULL, 2114160, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fd424887000
mprotect(0x7fd42488b000, 2093056, PROT_NONE) = 0
mmap(0x7fd424a8a000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7fd424a8a000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/libapparmor.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\30\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=47992, ...}) = 0
mmap(NULL, 2143184, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fd42467b000
mprotect(0x7fd424685000, 2097152, PROT_NONE) = 0
mmap(0x7fd424885000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xa000) = 0x7fd424885000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/usr/lib/x86_64-linux-gnu/libseccomp.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0 V\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=67744, ...}) = 0
mmap(NULL, 2162872, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fd42446a000
mprotect(0x7fd424477000, 2097152, PROT_NONE) = 0
mmap(0x7fd424677000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd000) = 0x7fd424677000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20m\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=67744, ...}) = 0
mmap(NULL, 2162872, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fd42446a000
mprotect(0x7fd424477000, 2097152, PROT_NONE) = 0
mmap(0x7fd424677000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd000) = 0x7fd424677000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20m\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=135175, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd425299000
mmap(NULL, 2212936, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fd42424d000
mprotect(0x7fd424265000, 2093056, PROT_NONE) = 0
mmap(0x7fd424464000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7fd424464000
mmap(0x7fd424466000, 13384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fd424466000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libutil.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20\17\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=10680, ...}) = 0
mmap(NULL, 2105624, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fd42404a000
mprotect(0x7fd42404c000, 2093056, PROT_NONE) = 0
mmap(0x7fd42424b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1000) = 0x7fd42424b000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd425298000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fd425296000
arch_prctl(ARCH_SET_FS, 0x7fd425296740) = 0
mprotect(0x7fd424e49000, 16384, PROT_READ) = 0
mprotect(0x7fd42424b000, 4096, PROT_READ) = 0
mprotect(0x7fd424464000, 4096, PROT_READ) = 0
mprotect(0x7fd424677000, 12288, PROT_READ) = 0
mprotect(0x7fd424885000, 4096, PROT_READ) = 0
mprotect(0x7fd424a8a000, 4096, PROT_READ) = 0
mprotect(0x7fd4250a2000, 4096, PROT_READ) = 0
mprotect(0x7fd4254cb000, 4096, PROT_READ) = 0
mprotect(0x7fd4252c6000, 4096, PROT_READ) = 0
munmap(0x7fd42529b000, 166397)          = 0
set_tid_address(0x7fd425296a10)         = 27974
set_robust_list(0x7fd425296a20, 0x18)   = 0
futex(0x7fffceefd69c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, 7fd425296740) = -1 EAGAIN (Resource temporarily unavailable)
rt_sigaction(SIGRTMIN, {0x7fd424253800, [], SA_RESTORER|SA_SIGINFO, 0x7fd42425cbd0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x7fd424253880, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7fd42425cbd0}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
getuid()                                = 0
getgid()                                = 0
geteuid()                               = 0
brk(0)                                  = 0x7fd426402000
brk(0x7fd426423000)                     = 0x7fd426423000
open("/etc/lxc/lxc.conf", O_RDONLY)     = -1 ENOENT (No such file or directory)
access("/var", F_OK)                    = 0
access("/var/log", F_OK)                = 0
access("/var/log/lxc", F_OK)            = 0
getuid()                                = 0
open("/var/log/lxc/535c7ada817bd8e2d75b5e0c597ebccf310f6e76ca2e0c5b814d51d77bbf688c.log", O_WRONLY|O_CREAT|O_APPEND|O_CLOEXEC, 0666) = 3
getuid()                                = 0
socket(PF_FILE, SOCK_STREAM, 0)         = 4
connect(4, {sa_family=AF_FILE, path=@"/var/lib/lxc/535c7ada817bd8e2d75b5e0c597ebccf310f6e76ca2e0c5b814d51d77bbf688c/command"}, 110) = -1 ECONNREFUSED (Connection refused)
close(4)                                = 0
write(3, "     lxc-attach 1382050385.226 E"..., 80) = 80
write(2, "lxc-attach: ", 12lxc-attach: )            = 12
write(2, "failed to get the init pid", 26failed to get the init pid) = 26
write(2, "\n", 1
)                       = 1
exit_group(-1)  

[spahl]

One thing that always works is "nsenter". It's a command part of util-linux (newer version needed depending on your distro)

You use it like this:
nsenter -m -u -i -n -p -t PID -r -w -- /bin/su -

PID is the pid of the first process in the container (not lxc-start but the one with pid 1 inside)

Obviously that does not solve the problem with lxc-attach but at least it will always work even if/when docker drops lxc.

[kiorky]

Yep, or drop bare lxc over to libvirt-lxc, thx for the idea !
Main main use case is in saltstack modules to do RPC on running containers:
https://github.com/saltstack/salt/blob/develop/salt/modules/dockerio.py#L1745

[kiorky]

Seems to rework on this setup:

• Ubuntu 13.04 as a host from this vagrant file: https://github.com/makinacorpus/vms/blob/master/Vagrantfile
• docker: c241388
• lxc: lxc/lxc@f371aca

[kiorky]

I think this commit from lxc: lxc/lxc@69733b5 may have fixed it all, and so we may urgently document that ...

[kiorky]

Yes, confirmed that this bug is on the LXC side of the moon.
For ubuntu users, please have a look on https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1244301

[kiorky]

cc @regilero @jpetazzo

[jpetazzo]

@spahl, do you often use nsenter? Any remarks (e.g. "stuff that needs the terminal tends to ENOWORK")? I wonder if we should make it the semi-official way to enter containers (instead of lxc-attach).

[spahl]

I only discovered a few months ago and haven't used it as much as lxc-attach yet. No weird bugs yet. It works well but has less features than lxc-attach. It's not aware of lxc so it won't try to match the config and won't drop capabilities. It's just much more lower level.

[0xqd]

@spahl thanks for pointing out the useful command :D

[mikexstudios]

Sorry for hijacking this old thread, but just wanted to link to: https://jpetazzo.github.io/2014/03/23/lxc-attach-nsinit-nsenter-docker-0-9/. Newer docker versions use libcontainer instead of the lxc driver so lxc-attach does not work anymore.

See also: https://github.com/jpetazzo/nsenter