Container fails to start with x86_64 Kernel and x86 userland

[copyninja]

Hi,
I've bit of weird setup in my laptop. I run Debian unstable with x86_64 kernel but 386 userland. uname command output on my system looks like below
Linux rudra 4.1.0-2-amd64 #1 SMP Debian 4.1.6-1 (2015-08-23) x86_64 GNU/Linux

I've created a container using below configuration

lxc.cgroup.cpuset.cpus = 0,1
lxc.cgroup.cpu.shares = 200

lxc.cgroup.memory.limit_in_bytes = 900M
lxc.cgroup.memory.soft_limit_in_bytes = 512M

lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = natbr0
lxc.network.name = eth0

lxc.utsname = sidbuilder.copyninja.info
lxc.arch = x86

Now when I try to start the container with lxc-start I see below error

lxc-start: seccomp.c: get_new_ctx: 167 Seccomp error -17 (Unknown error -17) adding arch: 2
lxc-start: start.c: lxc_init: 382 failed loading seccomp policy
lxc-start: start.c: __lxc_start: 1045 failed to initialize the container
lxc-start: lxc_start.c: main: 342 The container failed to start.
lxc-start: lxc_start.c: main: 346 Additional information can be obtained by setting the --logfile and --logpriority options.

Is it possible to run lxc undex x86_64 kernel with 386 (x86) user land?..

[jirutka]

Just curious, why you run x86_64 kernel with x86 userland?

[copyninja]

Better performance, I have a old 64 bit capable laptop which gets strained when running full 64 bit system. So running 64bit kernel with 32 bit userland which has improved memory footprints compared to full 64 bit system (Well haven't benchmarked it though).

[hallyn]

Hi,

this seems like it may be a problem in your libseccomp. Exactly what release are you on?

On Ubuntu 15.10, I do:

sudo lxc-create -t download -n t1 -- -d ubuntu -r trusty -a i386
sudo grep -i arch /var/lib/lxc/t1/config
lxc.arch = x86
sudo lxc-start -n t1
sudo lxc-attach -n t1 -- grep -i seccomp /proc/self/status
Seccomp: 2

[copyninja]

@hallyn Sorry for delayed response on my system ldd shows below output.

vasudev@rudra:~/Documents/Debian/collab-maint/lcdf-typetools$ ldd /usr/bin/lxc-create 
        linux-gate.so.1 (0xf774a000)
        liblxc.so.1 => /usr/lib/i386-linux-gnu/liblxc.so.1 (0xf76b1000)
        libcap.so.2 => /lib/i386-linux-gnu/libcap.so.2 (0xf76ab000)
        libapparmor.so.1 => /lib/i386-linux-gnu/libapparmor.so.1 (0xf7698000)
        libselinux.so.1 => /lib/i386-linux-gnu/libselinux.so.1 (0xf7670000)
        libseccomp.so.2 => /lib/i386-linux-gnu/libseccomp.so.2 (0xf7655000)
        libutil.so.1 => /lib/i386-linux-gnu/i686/cmov/libutil.so.1 (0xf7651000)
        libpthread.so.0 => /lib/i386-linux-gnu/i686/cmov/libpthread.so.0 (0xf7635000)
        libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xf7489000)
        libcgmanager.so.0 => /lib/i386-linux-gnu/libcgmanager.so.0 (0xf745b000)
        libnih.so.1 => /lib/libnih.so.1 (0xf7444000)
        libnih-dbus.so.1 => /lib/libnih-dbus.so.1 (0xf743b000)
        libdbus-1.so.3 => /lib/i386-linux-gnu/libdbus-1.so.3 (0xf73df000)
        /lib/ld-linux.so.2 (0x565b7000)
        libattr.so.1 => /lib/i386-linux-gnu/libattr.so.1 (0xf73d8000)
        libpcre.so.3 => /lib/i386-linux-gnu/libpcre.so.3 (0xf7365000)
        libdl.so.2 => /lib/i386-linux-gnu/i686/cmov/libdl.so.2 (0xf7360000)
        librt.so.1 => /lib/i386-linux-gnu/i686/cmov/librt.so.1 (0xf7357000)
        libsystemd.so.0 => /lib/i386-linux-gnu/libsystemd.so.0 (0xf72c9000)
        libm.so.6 => /lib/i386-linux-gnu/i686/cmov/libm.so.6 (0xf7282000)
        libresolv.so.2 => /lib/i386-linux-gnu/i686/cmov/libresolv.so.2 (0xf726b000)
        liblzma.so.5 => /lib/i386-linux-gnu/liblzma.so.5 (0xf7242000)
        libgcrypt.so.20 => /lib/i386-linux-gnu/libgcrypt.so.20 (0xf7191000)
        libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xf7174000)
        libgpg-error.so.0 => /lib/i386-linux-gnu/libgpg-error.so.0 (0xf715e000)

And

vasudev@rudra:~/Documents/Debian/collab-maint/lcdf-typetools$ dpkg -S /lib/i386-linux-gnu/libseccomp.so.2.2.3 
libseccomp2:i386: /lib/i386-linux-gnu/libseccomp.so.2.2.3

[hallyn]

Oh, I see. So your host is 32-bit userspace. I haven't looked (and won't be able to for a week) but I can easily imagine that case being mishandled in the lxc seccomp code.

[hallyn]

The problem occurs when:

1. lxc checks uname, which is returning x86_64 for you.
2. lxc decides to create a 32-bit compat context for you
   (using get_new_ctx())
   2a. seccomp creates a context for 32-bit x86
   2b. lxc adds SCMP_ARCH_X86 to the list of arches for that rule, but
   that was already there

I'm still trying to decide what lxc should do differently. You can
most easily get past this problem bug adding a check for

seccomp_arch_exist(compat_ctx, arch)

before the seccomp_arch_add is called. But I'm wondering what we should
be doing for your case in general. Should we continue to create a (default)
64-bit and a (compat) 32-bit context?

[copyninja]

Thanks guys for fixing this issue!.

[evgeni]

It seems 473ebc7 was reverted again and thus this issue still persists?

@copyninja @stgraber shall this be reopened again?

[copyninja]

@evgeni Is the reverted lxc released?. I basically first tested this on lxc shipped from Debian so if its really reverted then I think issue needs to be reopened.

[evgeni]

@copyninja as far as I can tell even the fixed one is not released as both commits are only in the 2.0 (master) branch.

[copyninja]

@evgeni that is bad :(. If the fix is reverted I think this issue should be reopened but I can't do it.

[nboullis]

Hi,

For those who are interested, I could patch debian’s LXC 1:1.0.6-6+deb8u5 to run on a i386 userspace with an amd64 kernel. The patch is here: 0099-seccomp-64bit-kernel-32bit-userspace.patch

For more info, see my comment.

[brauner]

@copyninja, @nboullis, @evgeni see #2274 and #2275.