Inconsistent autostart behavior

[lathiat]

Required information

• Distribution: Ubuntu
• Distribution version: 16.04
• The output of "lxc info" or if that fails:
  • Kernel version: 4.4.0-38-generic
  • LXC version: 2.0.4
  • LXD version: 2.0.4
  • Storage backend in use: zfs

Issue description

LXD autostart behavior is inconsistent and can be confusing.

A container is automatically restarted if it was last running, regardless of the autostart setting.

If you shutdown an LXD host with "poweroff", the container will store it's lastState of RUNNING and start again next boot. If you hard shutdown the host (e.g. echo b > /proc/sysrq-trigger), they will not restart - seems the state is never stored.

This behavior can lead you to make the wrong assumption about the autostart behavior, and it is likely to fail you at the worst time (when a host crashes or loses power). I believe this behavior contributed to the following juju bug not being discovered earlier:
https://bugs.launchpad.net/juju/+bug/1631254

I have two suggestions
(1) The autostart configuration option should allow you to set "NEVER", "LAST STATE" or "ALWAYS" - this kind of behavior is seen in the same option on physical servers. The current default (LAST STATE) is probably OK but would make this behavior more obvious
(2) The lastRunning state should be stored at runtime while the container is running, rather than at shutdown. So that an unclean or clean shutdown causes the same restart behavior.

These two items probably could be split, depending on feedback as to whether either option is considered a good idea.

[stgraber]

2. certainly sounds reasonable. I'd have to re-check exactly why we made it so that the state would only be stored when "shutdown" is requested through the internal LXD API rather than having it stored at startup time.

For 1), we may be able to differentiate between having boot.autostart set to false and not having it set at all. So if set to false, we could have LXD never restart the container, if unset we keep the current behavior (last state) and if set to true, we always start it.