lxc-start: segfault #1280

ne0zer0 · Nov 6, 2016

Hi,
Seems since the latest libvirt0 and libvirt-* packages update (2.3.0-3 > 2.4.0-1), I can not start anymore lxc.

zero@debian:~$ lxc-start --name ubuntu

*** Error in `lxc-start': free(): invalid pointer: 0x00007f05f849cb58 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x70bcb)[0x7f05f8174bcb]
/lib/x86_64-linux-gnu/libc.so.6(+0x76fa6)[0x7f05f817afa6]
/lib/x86_64-linux-gnu/libc.so.6(+0x7779e)[0x7f05f817b79e]
/usr/lib/x86_64-linux-gnu/liblxc.so.1(lxc_free_array+0x2a)[0x7f05f918043a]
/usr/lib/x86_64-linux-gnu/liblxc.so.1(+0x22701)[0x7f05f9168701]
/usr/lib/x86_64-linux-gnu/liblxc.so.1(+0x23b00)[0x7f05f9169b00]
/usr/lib/x86_64-linux-gnu/liblxc.so.1(+0x23c6f)[0x7f05f9169c6f]
/usr/lib/x86_64-linux-gnu/liblxc.so.1(freezer_state+0x2f)[0x7f05f917c98f]
/usr/lib/x86_64-linux-gnu/liblxc.so.1(lxc_getstate+0x11)[0x7f05f91966d1]
/usr/lib/x86_64-linux-gnu/liblxc.so.1(+0x5efb5)[0x7f05f91a4fb5]
/usr/lib/x86_64-linux-gnu/liblxc.so.1(+0x5f03c)[0x7f05f91a503c]
lxc-start(main+0x13e)[0x564f673c13fe]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f05f81242b1]
lxc-start(_start+0x2a)[0x564f673c1b1a]
======= Memory map: ========
564f673c0000-564f673c3000 r-xp 00000000 fe:01 547695                     /usr/bin/lxc-start
564f675c3000-564f675c4000 r--p 00003000 fe:01 547695                     /usr/bin/lxc-start
564f675c4000-564f675c5000 rw-p 00004000 fe:01 547695                     /usr/bin/lxc-start
564f6949a000-564f694bb000 rw-p 00000000 00:00 0                          [heap]
7f05f0000000-7f05f0021000 rw-p 00000000 00:00 0 
7f05f0021000-7f05f4000000 ---p 00000000 00:00 0 
7f05f7a76000-7f05f7a8c000 r-xp 00000000 fe:00 1426                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7f05f7a8c000-7f05f7c8b000 ---p 00016000 fe:00 1426                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7f05f7c8b000-7f05f7c8c000 r--p 00015000 fe:00 1426                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7f05f7c8c000-7f05f7c8d000 rw-p 00016000 fe:00 1426                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7f05f7c8d000-7f05f7c8f000 r-xp 00000000 fe:00 2436                       /lib/x86_64-linux-gnu/libdl-2.24.so
7f05f7c8f000-7f05f7e8f000 ---p 00002000 fe:00 2436                       /lib/x86_64-linux-gnu/libdl-2.24.so
7f05f7e8f000-7f05f7e90000 r--p 00002000 fe:00 2436                       /lib/x86_64-linux-gnu/libdl-2.24.so
7f05f7e90000-7f05f7e91000 rw-p 00003000 fe:00 2436                       /lib/x86_64-linux-gnu/libdl-2.24.so
7f05f7e91000-7f05f7f03000 r-xp 00000000 fe:00 368                        /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7f05f7f03000-7f05f8102000 ---p 00072000 fe:00 368                        /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7f05f8102000-7f05f8103000 r--p 00071000 fe:00 368                        /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7f05f8103000-7f05f8104000 rw-p 00072000 fe:00 368                        /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7f05f8104000-7f05f8299000 r-xp 00000000 fe:00 2307                       /lib/x86_64-linux-gnu/libc-2.24.so
7f05f8299000-7f05f8498000 ---p 00195000 fe:00 2307                       /lib/x86_64-linux-gnu/libc-2.24.so
7f05f8498000-7f05f849c000 r--p 00194000 fe:00 2307                       /lib/x86_64-linux-gnu/libc-2.24.so
7f05f849c000-7f05f849e000 rw-p 00198000 fe:00 2307                       /lib/x86_64-linux-gnu/libc-2.24.so
7f05f849e000-7f05f84a2000 rw-p 00000000 00:00 0 
7f05f84a2000-7f05f84b2000 r-xp 00000000 fe:00 2510                       /lib/x86_64-linux-gnu/libapparmor.so.1.4.0
7f05f84b2000-7f05f86b1000 ---p 00010000 fe:00 2510                       /lib/x86_64-linux-gnu/libapparmor.so.1.4.0
7f05f86b1000-7f05f86b2000 r--p 0000f000 fe:00 2510                       /lib/x86_64-linux-gnu/libapparmor.so.1.4.0
7f05f86b2000-7f05f86b3000 rw-p 00010000 fe:00 2510                       /lib/x86_64-linux-gnu/libapparmor.so.1.4.0
7f05f86b3000-7f05f86cb000 r-xp 00000000 fe:00 3085                       /lib/x86_64-linux-gnu/libpthread-2.24.so
7f05f86cb000-7f05f88ca000 ---p 00018000 fe:00 3085                       /lib/x86_64-linux-gnu/libpthread-2.24.so
7f05f88ca000-7f05f88cb000 r--p 00017000 fe:00 3085                       /lib/x86_64-linux-gnu/libpthread-2.24.so
7f05f88cb000-7f05f88cc000 rw-p 00018000 fe:00 3085                       /lib/x86_64-linux-gnu/libpthread-2.24.so
7f05f88cc000-7f05f88d0000 rw-p 00000000 00:00 0 
7f05f88d0000-7f05f88d2000 r-xp 00000000 fe:00 3157                       /lib/x86_64-linux-gnu/libutil-2.24.so
7f05f88d2000-7f05f8ad1000 ---p 00002000 fe:00 3157                       /lib/x86_64-linux-gnu/libutil-2.24.so
7f05f8ad1000-7f05f8ad2000 r--p 00001000 fe:00 3157                       /lib/x86_64-linux-gnu/libutil-2.24.so
7f05f8ad2000-7f05f8ad3000 rw-p 00002000 fe:00 3157                       /lib/x86_64-linux-gnu/libutil-2.24.so
7f05f8ad3000-7f05f8b01000 r-xp 00000000 fe:00 4272                       /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
7f05f8b01000-7f05f8d01000 ---p 0002e000 fe:00 4272                       /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
7f05f8d01000-7f05f8d17000 r--p 0002e000 fe:00 4272                       /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
7f05f8d17000-7f05f8d18000 rw-p 00044000 fe:00 4272                       /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
7f05f8d18000-7f05f8d3d000 r-xp 00000000 fe:00 733                        /lib/x86_64-linux-gnu/libselinux.so.1
7f05f8d3d000-7f05f8f3c000 ---p 00025000 fe:00 733                        /lib/x86_64-linux-gnu/libselinux.so.1
7f05f8f3c000-7f05f8f3d000 r--p 00024000 fe:00 733                        /lib/x86_64-linux-gnu/libselinux.so.1
7f05f8f3d000-7f05f8f3e000 rw-p 00025000 fe:00 733                        /lib/x86_64-linux-gnu/libselinux.so.1
7f05f8f3e000-7f05f8f40000 rw-p 00000000 00:00 0 
7f05f8f40000-7f05f8f44000 r-xp 00000000 fe:00 1525                       /lib/x86_64-linux-gnu/libcap.so.2.25
7f05f8f44000-7f05f9144000 ---p 00004000 fe:00 1525                       /lib/x86_64-linux-gnu/libcap.so.2.25
7f05f9144000-7f05f9145000 r--p 00004000 fe:00 1525                       /lib/x86_64-linux-gnu/libcap.so.2.25
7f05f9145000-7f05f9146000 rw-p 00005000 fe:00 1525                       /lib/x86_64-linux-gnu/libcap.so.2.25
7f05f9146000-7f05f91d4000 r-xp 00000000 fe:01 787209                     /usr/lib/x86_64-linux-gnu/liblxc.so.1.2.0
7f05f91d4000-7f05f93d3000 ---p 0008e000 fe:01 787209                     /usr/lib/x86_64-linux-gnu/liblxc.so.1.2.0
7f05f93d3000-7f05f93d6000 r--p 0008d000 fe:01 787209                     /usr/lib/x86_64-linux-gnu/liblxc.so.1.2.0
7f05f93d6000-7f05f93d8000 rw-p 00090000 fe:01 787209                     /usr/lib/x86_64-linux-gnu/liblxc.so.1.2.0
7f05f93d8000-7f05f93fb000 r-xp 00000000 fe:00 2296                       /lib/x86_64-linux-gnu/ld-2.24.so
7f05f95cf000-7f05f95d5000 rw-p 00000000 00:00 0 
7f05f95f6000-7f05f95fa000 rw-p 00000000 00:00 0 
7f05f95fa000-7f05f95fb000 r--p 00022000 fe:00 2296                       /lib/x86_64-linux-gnu/ld-2.24.so
7f05f95fb000-7f05f95fc000 rw-p 00023000 fe:00 2296                       /lib/x86_64-linux-gnu/ld-2.24.so
7f05f95fc000-7f05f95fd000 rw-p 00000000 00:00 0 
7fff3b2d6000-7fff3b2f7000 rw-p 00000000 00:00 0                          [stack]
7fff3b304000-7fff3b306000 r--p 00000000 00:00 0                          [vvar]
7fff3b306000-7fff3b308000 r-xp 00000000 00:00 0                          [vdso]
Aborted

Here debug

  lxc-start 20161106104952.401 INFO     lxc_start_ui - tools/lxc_start.c:main:264 - using rcfile /home/zero/.local/share/lxc/ubuntu/config
  lxc-start 20161106104952.401 INFO     lxc_confile - confile.c:config_idmap:1500 - read uid map: type u nsid 0 hostid 1214112 range 1000
  lxc-start 20161106104952.401 INFO     lxc_confile - confile.c:config_idmap:1500 - read uid map: type g nsid 0 hostid 1214112 range 1000
  lxc-start 20161106104952.401 INFO     lxc_confile - confile.c:config_idmap:1500 - read uid map: type u nsid 1000 hostid 1000 range 1
  lxc-start 20161106104952.401 INFO     lxc_confile - confile.c:config_idmap:1500 - read uid map: type g nsid 1000 hostid 1000 range 1
  lxc-start 20161106104952.401 INFO     lxc_confile - confile.c:config_idmap:1500 - read uid map: type u nsid 1001 hostid 1215113 range 64535
  lxc-start 20161106104952.401 INFO     lxc_confile - confile.c:config_idmap:1500 - read uid map: type g nsid 1001 hostid 1215113 range 64535

evgeni · Nov 6, 2016

does this also happen with privileged containers?

ne0zer0 · Nov 6, 2016

Here what I get with privileged container:

  lxc-start 20161106115555.620 WARN     lxc_confile - confile.c:config_pivotdir:1879 - lxc.pivotdir is ignored.  It will soon become an error.
  lxc-start 20161106115555.620 INFO     lxc_confile - confile.c:config_idmap:1500 - read uid map: type u nsid 0 hostid 1214112 range 1000
  lxc-start 20161106115555.620 INFO     lxc_confile - confile.c:config_idmap:1500 - read uid map: type g nsid 0 hostid 1214112 range 1000
  lxc-start 20161106115555.620 INFO     lxc_confile - confile.c:config_idmap:1500 - read uid map: type u nsid 1000 hostid 1000 range 1
  lxc-start 20161106115555.621 INFO     lxc_confile - confile.c:config_idmap:1500 - read uid map: type g nsid 1000 hostid 1000 range 1
  lxc-start 20161106115555.621 INFO     lxc_confile - confile.c:config_idmap:1500 - read uid map: type u nsid 1001 hostid 1215113 range 64535
  lxc-start 20161106115555.621 INFO     lxc_confile - confile.c:config_idmap:1500 - read uid map: type g nsid 1001 hostid 1215113 range 64535
  lxc-start 20161106115555.621 INFO     lxc_start - start.c:lxc_check_inherited:252 - closed inherited fd 4
  lxc-start 20161106115555.623 INFO     lxc_container - lxccontainer.c:do_lxcapi_start:797 - Attempting to set proc title to [lxc monitor] /var/lib/lxc ubuntu
  lxc-start 20161106115555.623 INFO     lxc_lsm - lsm/lsm.c:lsm_init:48 - LSM security driver nop
  lxc-start 20161106115555.623 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .reject_force_umount  # comment this to allow umount -f;  not recommended.
  lxc-start 20161106115555.623 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for reject_force_umount action 0.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:251 - Setting Seccomp rule to reject force umounts.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for reject_force_umount action 0.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:do_resolve_add_rule:251 - Setting Seccomp rule to reject force umounts.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .[all].
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .kexec_load errno 1.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for kexec_load action 327681.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for kexec_load action 327681.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .open_by_handle_at errno 1.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for open_by_handle_at action 327681.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for open_by_handle_at action 327681.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .init_module errno 1.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for init_module action 327681.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for init_module action 327681.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .finit_module errno 1.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for finit_module action 327681.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for finit_module action 327681.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:402 - processing: .delete_module errno 1.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:567 - Adding native rule for delete_module action 327681.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:570 - Adding compat rule for delete_module action 327681.
  lxc-start 20161106115555.624 INFO     lxc_seccomp - seccomp.c:parse_config_v2:580 - Merging in the compat Seccomp ctx into the main one.
  lxc-start 20161106115555.624 INFO     lxc_conf - conf.c:run_script_argv:367 - Executing script '/home/zero/.local/share/lxc/ubuntu/setup-pulse.sh' for container 'ubuntu', config section 'lxc'
  lxc-start 20161106115555.624 INFO     lxc_start - start.c:lxc_check_inherited:252 - closed inherited fd 4
  lxc-start 20161106115555.625 INFO     lxc_monitor - monitor.c:lxc_monitor_sock_name:178 - using monitor sock name lxc/ad055575fe28ddd5//var/lib/lxc
  lxc-start 20161106115555.629 ERROR    lxc_conf - conf.c:run_buffer:347 - Script exited with status 1
  lxc-start 20161106115555.629 ERROR    lxc_start - start.c:lxc_init:465 - failed to run pre-start hooks for container 'ubuntu'.
  lxc-start 20161106115555.629 ERROR    lxc_start - start.c:__lxc_start:1313 - failed to initialize the container
  lxc-start 20161106115555.629 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:172 - command get_cgroup failed to receive response
  lxc-start 20161106115600.634 ERROR    lxc_start_ui - tools/lxc_start.c:main:344 - The container failed to start.
  lxc-start 20161106115600.634 ERROR    lxc_start_ui - tools/lxc_start.c:main:346 - To get more details, run the container in foreground mode.
  lxc-start 20161106115600.634 ERROR    lxc_start_ui - tools/lxc_start.c:main:348 - Additional information can be obtained by setting the --logfile and --logpriority options.

ne0zer0 · Nov 6, 2016

Find also this in journalctl:

Nov 05 21:30:05 debian cgmanager[2335]: cgmanager: Error mounting unified: No such file or directory
Nov 05 21:30:05 debian cgmanager[2335]: cgmanager: failed to collect cgroup subsystems
Nov 05 21:30:05 debian systemd[1]: cgmanager.service: Main process exited, code=exited, status=1/FAILURE
Nov 05 21:30:05 debian systemd[1]: cgmanager.service: Unit entered failed state.
Nov 05 21:30:05 debian systemd[1]: cgmanager.service: Failed with result 'exit-code'.
Nov 05 21:30:05 debian systemd[1]: cgmanager.service: Service hold-off time over, scheduling restart.
Nov 05 21:30:05 debian systemd[1]: cgmanager.service: Start request repeated too quickly.
Nov 05 21:30:05 debian systemd[1]: cgmanager.service: Unit entered failed state.
Nov 05 21:30:05 debian systemd[1]: cgmanager.service: Failed with result 'exit-code'.


Nov 05 21:30:05 debian systemd[1]: Started Cgroup management daemon.
Nov 05 21:30:05 debian cgmanager[2335]: cgmanager: failed to collect cgroup subsystems
Nov 05 21:30:05 debian systemd[1]: Stopped Cgroup management daemon.
Nov 05 21:30:05 debian systemd[1]: Failed to start Cgroup management daemon.
Nov 05 21:30:13 debian PAM-CGFS[2447]: Corrupt /proc/self/cgroup

evgeni · Nov 6, 2016

can you please try removing cgmanager, rebooting and re-trying?

ne0zer0 · Nov 6, 2016

Done, here the output:

● cgmanager.service - Cgroup management daemon
   Loaded: loaded (/lib/systemd/system/cgmanager.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2016-11-06 13:54:39 CET; 3s ago
  Process: 3528 ExecStart=/sbin/cgmanager -m name=systemd (code=exited, status=1/FAILURE)
 Main PID: 3528 (code=exited, status=1/FAILURE)
      CPU: 2ms

Nov 06 13:54:39 debian systemd[1]: cgmanager.service: Failed with result 'exit-code'.
Nov 06 13:54:39 debian systemd[1]: cgmanager.service: Service hold-off time over, scheduling restart.
Nov 06 13:54:39 debian systemd[1]: Stopped Cgroup management daemon.
Nov 06 13:54:39 debian systemd[1]: cgmanager.service: Start request repeated too quickly.
Nov 06 13:54:39 debian systemd[1]: Failed to start Cgroup management daemon.
Nov 06 13:54:39 debian systemd[1]: cgmanager.service: Unit entered failed state.
Nov 06 13:54:39 debian systemd[1]: cgmanager.service: Failed with result 'exit-code'.

brauner · Nov 6, 2016

This strongly reminds me of: #1258. In any case, the segfault you're seeing should be fixed in current git master as soon as @hallyn finds the time to look at #1262.
That is not your main issue though. I suspect what you're seeing is caused by incorrectly mounted cgroups. The error message you're seeing

Nov 05 21:30:13 debian PAM-CGFS[2447]: Corrupt /proc/self/cgroup

makes this especially likely. In any case, cgmanager is deprecated and should not be used to setup cgroup mounts especially with LXC 2.*.* versions which provide a new cgroup backend. So please make sure that the following things items are checked:

uninstall cgmanger and stop any instance of the daemon that might still be running
make sure that you've not recently changed init systems: aka switched from sysvinit to systemd or vica versa. If you did switch init systems, cleanly remove the one you don't want to keep. Install any packages that are required to correctly setup cgroup mounts on your system (This only affects sysvinit.) in the standard cross-distro way and reboot your system to drop any references to the removed init systems that might linger in your current cgroup mounts.

Please report back if the issue still persists. If it does, please show the output of:

cat /proc/1/mountinfo | grep cgroup
cat /proc/self/cgroup
attach any logs the container has produced or start it with -l debug -o container. log and append it
cat /etc/fstab

I might need to request further information later on. :)

ne0zer0 · Nov 6, 2016

The issue still persists:

zero@debian:~$ cat /proc/1/mountinfo | grep cgroup
26 16 0:21 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:10 - tmpfs tmpfs ro,mode=755
27 26 0:22 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:11 - cgroup2 cgroup rw
29 26 0:24 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:14 - cgroup cgroup rw,perf_event
30 26 0:25 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:15 - cgroup cgroup rw,freezer
31 26 0:26 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:16 - cgroup cgroup rw,cpu,cpuacct
32 26 0:27 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,devices
33 26 0:28 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:18 - cgroup cgroup rw,net_cls,net_prio
34 26 0:29 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:19 - cgroup cgroup rw,memory
35 26 0:30 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime shared:20 - cgroup cgroup rw,pids
36 26 0:31 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:21 - cgroup cgroup rw,blkio
37 26 0:32 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:22 - cgroup cgroup rw,cpuset

zero@debian:~$ cat /proc/self/cgroup
9:cpuset:/
8:blkio:/user.slice
7:pids:/user.slice/user-1000.slice/user@1000.service
6:memory:/user.slice
5:net_cls,net_prio:/
4:devices:/user.slice
3:cpu,cpuacct:/user.slice
2:freezer:/
1:perf_event:/
0::/user.slice/user-1000.slice/user@1000.service/gnome-terminal-server.service

zero@debian:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
UUID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /boot           ext4 defaults 0   2
/dev/mapper/sda1_crypt /               ext4    discard,errors=remount-ro 0       1
/dev/mapper/sda5_crypt /usr            ext4    discard        0       2
/dev/mapper/sda6_crypt /usr/local      ext4    discard        0       2
/dev/mapper/sda7_crypt /opt            ext4    discard        0       2
/dev/mapper/sda9_crypt /home           ext4    discard        0       2
/dev/mapper/sda8_crypt /var            ext4    discard        0       2
/dev/mapper/sdb1_crypt /var/tmp        ext4    defaults        0       2
/dev/mapper/sdb5_crypt /var/log        ext4    defaults        0       2
/dev/mapper/sdb6_crypt none            swap    sw              0       0
/dev/mapper/sdb7_crypt /tmp            ext4    defaults        0       2

zero@debian:~$ lxc-start -n ubuntu -l DEBUG -o /tmp/lxc_test.log
*** Error in `lxc-start': free(): invalid pointer: 0x00007f8a4b794b58 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x70bcb)[0x7f8a4b46cbcb]
/lib/x86_64-linux-gnu/libc.so.6(+0x76fa6)[0x7f8a4b472fa6]
/lib/x86_64-linux-gnu/libc.so.6(+0x7779e)[0x7f8a4b47379e]
/usr/lib/x86_64-linux-gnu/liblxc.so.1(lxc_free_array+0x2a)[0x7f8a4c47843a]
/usr/lib/x86_64-linux-gnu/liblxc.so.1(+0x22701)[0x7f8a4c460701]
/usr/lib/x86_64-linux-gnu/liblxc.so.1(+0x23b00)[0x7f8a4c461b00]
/usr/lib/x86_64-linux-gnu/liblxc.so.1(+0x23c6f)[0x7f8a4c461c6f]
/usr/lib/x86_64-linux-gnu/liblxc.so.1(freezer_state+0x2f)[0x7f8a4c47498f]
/usr/lib/x86_64-linux-gnu/liblxc.so.1(lxc_getstate+0x11)[0x7f8a4c48e6d1]
/usr/lib/x86_64-linux-gnu/liblxc.so.1(+0x5efb5)[0x7f8a4c49cfb5]
/usr/lib/x86_64-linux-gnu/liblxc.so.1(+0x5f03c)[0x7f8a4c49d03c]
lxc-start(main+0x13e)[0x558155aa33fe]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f8a4b41c2b1]
lxc-start(_start+0x2a)[0x558155aa3b1a]
======= Memory map: ========
558155aa2000-558155aa5000 r-xp 00000000 fe:01 547695                     /usr/bin/lxc-start
558155ca5000-558155ca6000 r--p 00003000 fe:01 547695                     /usr/bin/lxc-start
558155ca6000-558155ca7000 rw-p 00004000 fe:01 547695                     /usr/bin/lxc-start
55815799c000-5581579bd000 rw-p 00000000 00:00 0                          [heap]
7f8a44000000-7f8a44021000 rw-p 00000000 00:00 0 
7f8a44021000-7f8a48000000 ---p 00000000 00:00 0 
7f8a4ad6e000-7f8a4ad84000 r-xp 00000000 fe:00 1426                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7f8a4ad84000-7f8a4af83000 ---p 00016000 fe:00 1426                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7f8a4af83000-7f8a4af84000 r--p 00015000 fe:00 1426                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7f8a4af84000-7f8a4af85000 rw-p 00016000 fe:00 1426                       /lib/x86_64-linux-gnu/libgcc_s.so.1
7f8a4af85000-7f8a4af87000 r-xp 00000000 fe:00 2436                       /lib/x86_64-linux-gnu/libdl-2.24.so
7f8a4af87000-7f8a4b187000 ---p 00002000 fe:00 2436                       /lib/x86_64-linux-gnu/libdl-2.24.so
7f8a4b187000-7f8a4b188000 r--p 00002000 fe:00 2436                       /lib/x86_64-linux-gnu/libdl-2.24.so
7f8a4b188000-7f8a4b189000 rw-p 00003000 fe:00 2436                       /lib/x86_64-linux-gnu/libdl-2.24.so
7f8a4b189000-7f8a4b1fb000 r-xp 00000000 fe:00 368                        /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7f8a4b1fb000-7f8a4b3fa000 ---p 00072000 fe:00 368                        /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7f8a4b3fa000-7f8a4b3fb000 r--p 00071000 fe:00 368                        /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7f8a4b3fb000-7f8a4b3fc000 rw-p 00072000 fe:00 368                        /lib/x86_64-linux-gnu/libpcre.so.3.13.3
7f8a4b3fc000-7f8a4b591000 r-xp 00000000 fe:00 2307                       /lib/x86_64-linux-gnu/libc-2.24.so
7f8a4b591000-7f8a4b790000 ---p 00195000 fe:00 2307                       /lib/x86_64-linux-gnu/libc-2.24.so
7f8a4b790000-7f8a4b794000 r--p 00194000 fe:00 2307                       /lib/x86_64-linux-gnu/libc-2.24.so
7f8a4b794000-7f8a4b796000 rw-p 00198000 fe:00 2307                       /lib/x86_64-linux-gnu/libc-2.24.so
7f8a4b796000-7f8a4b79a000 rw-p 00000000 00:00 0 
7f8a4b79a000-7f8a4b7aa000 r-xp 00000000 fe:00 2510                       /lib/x86_64-linux-gnu/libapparmor.so.1.4.0
7f8a4b7aa000-7f8a4b9a9000 ---p 00010000 fe:00 2510                       /lib/x86_64-linux-gnu/libapparmor.so.1.4.0
7f8a4b9a9000-7f8a4b9aa000 r--p 0000f000 fe:00 2510                       /lib/x86_64-linux-gnu/libapparmor.so.1.4.0
7f8a4b9aa000-7f8a4b9ab000 rw-p 00010000 fe:00 2510                       /lib/x86_64-linux-gnu/libapparmor.so.1.4.0
7f8a4b9ab000-7f8a4b9c3000 r-xp 00000000 fe:00 3085                       /lib/x86_64-linux-gnu/libpthread-2.24.so
7f8a4b9c3000-7f8a4bbc2000 ---p 00018000 fe:00 3085                       /lib/x86_64-linux-gnu/libpthread-2.24.so
7f8a4bbc2000-7f8a4bbc3000 r--p 00017000 fe:00 3085                       /lib/x86_64-linux-gnu/libpthread-2.24.so
7f8a4bbc3000-7f8a4bbc4000 rw-p 00018000 fe:00 3085                       /lib/x86_64-linux-gnu/libpthread-2.24.so
7f8a4bbc4000-7f8a4bbc8000 rw-p 00000000 00:00 0 
7f8a4bbc8000-7f8a4bbca000 r-xp 00000000 fe:00 3157                       /lib/x86_64-linux-gnu/libutil-2.24.so
7f8a4bbca000-7f8a4bdc9000 ---p 00002000 fe:00 3157                       /lib/x86_64-linux-gnu/libutil-2.24.so
7f8a4bdc9000-7f8a4bdca000 r--p 00001000 fe:00 3157                       /lib/x86_64-linux-gnu/libutil-2.24.so
7f8a4bdca000-7f8a4bdcb000 rw-p 00002000 fe:00 3157                       /lib/x86_64-linux-gnu/libutil-2.24.so
7f8a4bdcb000-7f8a4bdf9000 r-xp 00000000 fe:00 4272                       /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
7f8a4bdf9000-7f8a4bff9000 ---p 0002e000 fe:00 4272                       /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
7f8a4bff9000-7f8a4c00f000 r--p 0002e000 fe:00 4272                       /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
7f8a4c00f000-7f8a4c010000 rw-p 00044000 fe:00 4272                       /lib/x86_64-linux-gnu/libseccomp.so.2.3.1
7f8a4c010000-7f8a4c035000 r-xp 00000000 fe:00 733                        /lib/x86_64-linux-gnu/libselinux.so.1
7f8a4c035000-7f8a4c234000 ---p 00025000 fe:00 733                        /lib/x86_64-linux-gnu/libselinux.so.1
7f8a4c234000-7f8a4c235000 r--p 00024000 fe:00 733                        /lib/x86_64-linux-gnu/libselinux.so.1
7f8a4c235000-7f8a4c236000 rw-p 00025000 fe:00 733                        /lib/x86_64-linux-gnu/libselinux.so.1
7f8a4c236000-7f8a4c238000 rw-p 00000000 00:00 0 
7f8a4c238000-7f8a4c23c000 r-xp 00000000 fe:00 1525                       /lib/x86_64-linux-gnu/libcap.so.2.25
7f8a4c23c000-7f8a4c43c000 ---p 00004000 fe:00 1525                       /lib/x86_64-linux-gnu/libcap.so.2.25
7f8a4c43c000-7f8a4c43d000 r--p 00004000 fe:00 1525                       /lib/x86_64-linux-gnu/libcap.so.2.25
7f8a4c43d000-7f8a4c43e000 rw-p 00005000 fe:00 1525                       /lib/x86_64-linux-gnu/libcap.so.2.25
7f8a4c43e000-7f8a4c4cc000 r-xp 00000000 fe:01 787209                     /usr/lib/x86_64-linux-gnu/liblxc.so.1.2.0
7f8a4c4cc000-7f8a4c6cb000 ---p 0008e000 fe:01 787209                     /usr/lib/x86_64-linux-gnu/liblxc.so.1.2.0
7f8a4c6cb000-7f8a4c6ce000 r--p 0008d000 fe:01 787209                     /usr/lib/x86_64-linux-gnu/liblxc.so.1.2.0
7f8a4c6ce000-7f8a4c6d0000 rw-p 00090000 fe:01 787209                     /usr/lib/x86_64-linux-gnu/liblxc.so.1.2.0
7f8a4c6d0000-7f8a4c6f3000 r-xp 00000000 fe:00 2296                       /lib/x86_64-linux-gnu/ld-2.24.so
7f8a4c8c7000-7f8a4c8cd000 rw-p 00000000 00:00 0 
7f8a4c8ee000-7f8a4c8f2000 rw-p 00000000 00:00 0 
7f8a4c8f2000-7f8a4c8f3000 r--p 00022000 fe:00 2296                       /lib/x86_64-linux-gnu/ld-2.24.so
7f8a4c8f3000-7f8a4c8f4000 rw-p 00023000 fe:00 2296                       /lib/x86_64-linux-gnu/ld-2.24.so
7f8a4c8f4000-7f8a4c8f5000 rw-p 00000000 00:00 0 
7ffd0ccf5000-7ffd0cd16000 rw-p 00000000 00:00 0                          [stack]
7ffd0cd8f000-7ffd0cd91000 r--p 00000000 00:00 0                          [vvar]
7ffd0cd91000-7ffd0cd93000 r-xp 00000000 00:00 0                          [vdso]
Aborted

zero@debian:~$ cat /tmp/lxc_test.log
lxc-start 20161106183416.369 INFO lxc_start_ui - tools/lxc_start.c:main:264 - using rcfile /home/zero/.local/share/lxc/ubuntu/config
lxc-start 20161106183416.369 WARN lxc_confile - confile.c:config_pivotdir:1879 - lxc.pivotdir is ignored. It will soon become an error.
lxc-start 20161106183416.370 INFO lxc_confile - confile.c:config_idmap:1500 - read uid map: type u nsid 0 hostid 1214112 range 1000
lxc-start 20161106183416.370 INFO lxc_confile - confile.c:config_idmap:1500 - read uid map: type g nsid 0 hostid 1214112 range 1000
lxc-start 20161106183416.370 INFO lxc_confile - confile.c:config_idmap:1500 - read uid map: type u nsid 1000 hostid 1000 range 1
lxc-start 20161106183416.370 INFO lxc_confile - confile.c:config_idmap:1500 - read uid map: type g nsid 1000 hostid 1000 range 1
lxc-start 20161106183416.370 INFO lxc_confile - confile.c:config_idmap:1500 - read uid map: type u nsid 1001 hostid 1215113 range 64535
lxc-start 20161106183416.370 INFO lxc_confile - confile.c:config_idmap:1500 - read uid map: type g nsid 1001 hostid 1215113 range 64535

brauner · Nov 6, 2016

Apart from the segfault that is fixed by a recent pr that yet has to be merged I mentioned you are not placed in a writeable cgroup for crucial subsystems such as systemd and freezer. This still points to a busted cgroup setup, so again: remove cgmanager and you probably also want to install libpam-cgfs and reboot and try again.

ne0zer0 · Nov 6, 2016

Already done:

I removed cgmanager
reboot,
and gave you the outputs…

but the issue still persists :s

brauner · Nov 6, 2016

Did you install libpam-cgfs and reboot?

ne0zer0 · Nov 6, 2016

Of course:
dpkg -l libpam-cgfs
ii libpam-cgfs 2.0.4-1 amd64 PAM module for managing cgroups f

brauner · Nov 6, 2016

So libpam-cgfs should place you in writeable cgroups for all relevant subsystems. Can you please show the output of:

/etc/pam.d/common-session
/etc/pam.d/common-session-noninteractive

ne0zer0 · Nov 6, 2016

/etc/pam.d/common-session

#
# /etc/pam.d/common-session - session-related modules common to all services
#
# This file is included from other service-specific PAM config files,
# and should contain a list of modules that define tasks to be performed
# at the start and end of sessions of *any* kind (both interactive and
# non-interactive).
#
# As of pam 1.0.1-6, this file is managed by pam-auth-update by default.
# To take advantage of this, it is recommended that you configure any
# local modules either before or after the default block, and use
# pam-auth-update to manage selection of other modules.  See
# pam-auth-update(8) for details.

# here are the per-package modules (the "Primary" block)
session [default=1]         pam_permit.so
# here's the fallback if no module succeeds
session requisite           pam_deny.so
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
session required            pam_permit.so
# and here are more per-package modules (the "Additional" block)
session required    pam_unix.so 
session optional    pam_systemd.so 
session optional            pam_ck_connector.so nox11
session optional    pam_cgfs.so -c freezer,memory,name=systemd
# end of pam-auth-update config
# ADDED
session optional pam_gnome_keyring.so auto_start

/etc/pam.d/common-session-noninteractive

#
# /etc/pam.d/common-session-noninteractive - session-related modules
# common to all non-interactive services
#
# This file is included from other service-specific PAM config files,
# and should contain a list of modules that define tasks to be performed
# at the start and end of all non-interactive sessions.
#
# As of pam 1.0.1-6, this file is managed by pam-auth-update by default.
# To take advantage of this, it is recommended that you configure any
# local modules either before or after the default block, and use
# pam-auth-update to manage selection of other modules.  See
# pam-auth-update(8) for details.

# here are the per-package modules (the "Primary" block)
session [default=1]         pam_permit.so
# here's the fallback if no module succeeds
session requisite           pam_deny.so
# prime the stack with a positive return value if there isn't one already;
# this avoids us returning an error just because nothing sets a success code
# since the modules above will each just jump around
session required            pam_permit.so
# and here are more per-package modules (the "Additional" block)
session required    pam_unix.so 
session optional    pam_cgfs.so -c freezer,memory,name=systemd
# end of pam-auth-update config

brauner · Nov 6, 2016

Looking at the output of your /proc/1/mountinfo, you realize that

27 26 0:22 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:11 - cgroup2 cgroup rw

means that the systemd controller is mounted into cgroup v2 hierarchy, right? That is not something we support right now as there is no feature parity between cgroup v1 and cgroup v2. I also do not know what can possibly cause systemd to mount itself into a cgroup v2 hierarchy, especially if you, as you said multiple times. rebooted. There is nothing LXC can currently do about this. So closing this. You should make sure that systemd is mounted into a cgroup v1 hierarchy.

ne0zer0 · Nov 7, 2016

Is there any roadmap for cgroup v2?

Because since Linux kernel 4.5 and Systemd 232, cgroup v2 are set by default.

I thought that was since libvirt0 update, that lxc stopped to work, but the problem came with the update of systemd 231-10>232-1.

evgeni · Nov 7, 2016

It seems the current workaround is to boot with systemd.legacy_systemd_cgroup_controller.

ne0zer0 · Nov 7, 2016

Thanks, I will try this.

And, can you give me a link?

evgeni · Nov 7, 2016

So far the only documentation on that is the PR which implemented it: systemd/systemd#3965

ne0zer0 · Nov 7, 2016

Super!
Thanks evgeni!
Already said but, LXC and Systemd suck! Their documentation are very sketchy!

phedders · Nov 8, 2016

systemd 232 broke lxc, docker and rkt for me. Not a pleasant surprise.

stgraber · Nov 8, 2016

Sounds like someone jump the gun here, distros very much shouldn't be switching to cgroupv2 until it's at parity with cgroupv1, and that won't be true until it's got a cpu controller...

Hopefully distros will be fixing this quickly by just forcing systemd in cgroupv1 mode until that happens...

brauner · Nov 8, 2016

This systemd/systemd#3965 really is a bit premature.

stgraber · Nov 8, 2016

At least it's limited to the systemd hierarchy, but that's still going to be a big problem for anyone who cares about running containers that use systemd as their init systems...

martinpitt · Nov 10, 2016

systemd/systemd#4628 reverted this for now, but you can still boot with systemd.legacy_systemd_cgroup_controller=0 to get the unified behaviour; that is useful for developers to make runc/docker/lxc/etc. work with the unified hierarchy eventually.

phedders · Nov 10, 2016

Many thanks.

On 10 Nov 2016, 07:11, at 07:11, Martin Pitt notifications@github.com wrote:

systemd/systemd#4628 reverted this for now, but
you can still boot with systemd.legacy_systemd_cgroup_controller=0 to
get the unified behaviour; that is useful for developers to make
runc/docker/lxc/etc. work with the unified hierarchy eventually.

You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
#1280 (comment)

brauner · Nov 23, 2016

Closing this issue and opening #1316 to track cgroupfs v2 support.

brauner closed this Nov 6, 2016

evverx referenced this issue in systemd/systemd Nov 8, 2016
Merged
Using the unified hierarchy for /sys/fs/cgroup/systemd when legacy hierarchies are being used #3965

brauner reopened this Nov 9, 2016

martinpitt referenced this issue in systemd/systemd Nov 9, 2016
Merged
core: don't use the unified hierarchy for the systemd cgroup yet #4628

brauner referenced this issue Nov 23, 2016
Open
cgroupfs v2 support #1316

brauner closed this Nov 23, 2016

lxc/lxc

Join GitHub today

lxc-start: segfault #1280

Comments

ne0zer0 commented Nov 6, 2016 • edited Edited 1 time ne0zer0 Nov 6, 2016

evgeni commented Nov 6, 2016

ne0zer0 commented Nov 6, 2016

ne0zer0 commented Nov 6, 2016

evgeni commented Nov 6, 2016

ne0zer0 commented Nov 6, 2016

brauner commented Nov 6, 2016

ne0zer0 commented Nov 6, 2016

brauner commented Nov 6, 2016

ne0zer0 commented Nov 6, 2016

brauner commented Nov 6, 2016

ne0zer0 commented Nov 6, 2016

brauner commented Nov 6, 2016

ne0zer0 commented Nov 6, 2016

brauner commented Nov 6, 2016

brauner closed this Nov 6, 2016

ne0zer0 commented Nov 7, 2016

evgeni commented Nov 7, 2016

ne0zer0 commented Nov 7, 2016

evgeni commented Nov 7, 2016

ne0zer0 commented Nov 7, 2016

phedders commented Nov 8, 2016

stgraber commented Nov 8, 2016

brauner commented Nov 8, 2016

stgraber commented Nov 8, 2016

evverx referenced this issue in systemd/systemd Nov 8, 2016

Using the unified hierarchy for /sys/fs/cgroup/systemd when legacy hierarchies are being used #3965

brauner reopened this Nov 9, 2016

martinpitt added a commit to martinpitt/systemd that referenced this issue Nov 9, 2016

martinpitt referenced this issue in systemd/systemd Nov 9, 2016

core: don't use the unified hierarchy for the systemd cgroup yet #4628

keszybz added a commit to systemd/systemd that referenced this issue Nov 10, 2016

martinpitt commented Nov 10, 2016

phedders commented Nov 10, 2016

fbuihuu added a commit to fbuihuu/systemd-opensuse-next that referenced this issue Nov 21, 2016

brauner referenced this issue Nov 23, 2016

cgroupfs v2 support #1316

brauner commented Nov 23, 2016

brauner closed this Nov 23, 2016

keszybz added a commit to systemd/systemd-stable that referenced this issue Jan 31, 2017

globin added a commit to NixOS/systemd that referenced this issue Feb 16, 2017

Yamakuzure added a commit to elogind/elogind that referenced this issue Jul 17, 2017

Yamakuzure added a commit to elogind/elogind that referenced this issue Jul 18, 2017

ne0zer0 commented Nov 6, 2016 •

Edited 1 time

ne0zer0
Nov 6, 2016