Cannot manage services inside chroot

[OnGle]

In SystemD services do not inherit the environment underwhich they were called unlike sysvinit. This means starting/stopping services within a chroot will fail.

Possible Solutions:

1. Use systemd-nspawn or some other container software rather than chroots.
2. Use systemd unit files to setup the environment so that it does work (related information here http://0pointer.de/blog/projects/changing-roots)

[OnGle]

Possibly related to #954

[JedMeister]

After having a bit of a think about this and a (really) quick test, I think option 1 may be the easiest way to go. FWIW, here's what I just tested (which appeared to work, at least on face value):

apt-get install dbus # required on host to launch nspawn containers
cat > /usr/local/bin/fab-chroot <<EOF
#!/bin/bash -e
systemd-nspawn -D \$@
EOF
chmod +x /usr/local/bin/fab-chroot

I have only tested for Core and the few services I checked (stunnel, webmin & shellinabox), actually seem to start ok within a chroot?! However, this does appear to work:

root@tkldev products/core# which fab-chroot 
/usr/local/bin/fab-chroot
root@tkldev products/core# fab-chroot build/root.patched/
Spawning container root.patched on /turnkey/fab/products/core/build/root.patched.
Press ^] three times within 1s to kill container.
/etc/localtime is not a symlink, not updating container timezone.
root@root ~# 

However, there isn't any networking by default, so obviously more work is needed. I have done some very quick and dirty testing and here's the results:

root@root ~# service networking start
[....] Configuring network interfaces...RTNETLINK answers: Operation not permitted
ifup: failed to bring up lo
udhcpc (v1.22.1) started
mkdir: cannot create directory '/etc/resolvconf/run': File exists
SIOCSIFADDR: Operation not permitted
SIOCSIFFLAGS: Operation not permitted
Sending discover...
Sending select for 192.168.1.84...
Lease of 192.168.1.84 obtained, lease time 86400
SIOCSIFADDR: Operation not permitted
SIOCSIFFLAGS: Operation not permitted
SIOCSIFBRDADDR: Operation not permitted
SIOCSIFFLAGS: Operation not permitted
SIOCSIFNETMASK: Operation not permitted
/etc/udhcpc/default.script: Resetting default routes
SIOCDELRT: Operation not permitted
RTNETLINK answers: Operation not permitted
/etc/udhcpc/default.script: Adding DNS 192.168.1.1
mkdir: cannot create directory '/etc/resolvconf/run': File exists
mkdir: cannot create directory '/etc/resolvconf/run': File exists
failed.
root@root ~# ping 192.168.1.151
PING 192.168.1.151 (192.168.1.151) 56(84) bytes of data.
64 bytes from 192.168.1.151: icmp_seq=1 ttl=64 time=0.028 ms
64 bytes from 192.168.1.151: icmp_seq=2 ttl=64 time=0.033 ms
64 bytes from 192.168.1.151: icmp_seq=3 ttl=64 time=0.034 ms
64 bytes from 192.168.1.151: icmp_seq=4 ttl=64 time=0.037 ms
^C
--- 192.168.1.151 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 2997ms
rtt min/avg/max/mdev = 0.028/0.033/0.037/0.003 ms
root@root ~# ping 192.168.1.1  
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
64 bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=5.25 ms
64 bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=0.362 ms
64 bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=0.340 ms
^C
--- 192.168.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.340/1.985/5.254/2.311 ms
root@root ~# ping google.com
ping: google.com: Temporary failure in name resolution
root@root ~# ping 8.8.8.8    
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=27.6 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=27.5 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=56 time=27.5 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 27.511/27.565/27.655/0.202 ms
root@root ~# cat /etc/resolv.conf 
cat: /etc/resolv.conf: No such file or directory
root@root ~# ls -l /etc/resolv.conf 
lrwxrwxrwx 1 root root 31 Oct 18 06:15 /etc/resolv.conf -> /etc/resolvconf/run/resolv.conf
root@root ~# ls -l /etc/resolvconf/run 
lrwxrwxrwx 1 root root 15 Oct 18 06:13 /etc/resolvconf/run -> /run/resolvconf
root@root ~# ls -l /run/resolvconf
ls: cannot access '/run/resolvconf': No such file or directory
root@root ~# mv /etc/resolv.conf /etc/resolv.conf.orig
root@root ~# echo "nameserver 8.8.8.8" > /etc/resolv.conf
root@root ~# ping google.com
PING google.com (172.217.25.174) 56(84) bytes of data.
64 bytes from sin01s16-in-f14.1e100.net (172.217.25.174): icmp_seq=1 ttl=54 time=27.1 ms
64 bytes from sin01s16-in-f14.1e100.net (172.217.25.174): icmp_seq=2 ttl=54 time=27.3 ms
64 bytes from sin01s16-in-f14.1e100.net (172.217.25.174): icmp_seq=3 ttl=54 time=26.8 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 26.865/27.140/27.389/0.253 ms

So networking appears to work ok (once a proper resolv.conf has been created)?!

[vondrt4]

That's exactly what they write in the referenced article. The nspawn shares networking with the host but cannot make any changes. There is a command-line argument to create a network namespace. But then you would have to create a /etc/network/interfaces entry for the host0 interface. And I think I also had to run dnsmasq on the host to serve DHCP.

[JedMeister]

We've discovered that this mainly affects Apache so far. We've added a service wrapper script (/usr/local/bin/service) to catch calls to Apache and start it using apachectl. Everything with a proper SysvInit script seems to work ok.

We just need to make sure that we clean that up afterwards, definitely prior to release!

[JedMeister]

We still need to finalise this and clean up afterwards, I'll assign myself for now...

[JedMeister]

I'm closing this issue. Whilst it still remains as something of an issue, we've worked around it ok for now. We'll need to figure out something better at some point in the future, but hopefully by the time we're getting close to a release based on Debian buster, we'll have TKLX up and running and won't need to worry about this so much...