Upgrading Salt via Salt results in dying minions and broken dpkg

[syphernl]

Description of Issue/Question

We use pkg.upgrade dist_upgrade=True to upgrade our servers semi-unattended.

They were running 2017.7.0 prior to issuing the upgrade command and were supposed to be upgraded to 2017.7.1.

However, a majority of the machines had issues where the salt-minion would not be running anymore afterwards and shows the following issue when doing anything with apt afterwards on the machine itself:

E: dpkg was interrupted, you must manually run 'dpkg —configure -a' to correct the problem.

Doing so shows the following issue:

dpkg: error processing package salt-minion (--configure):
 package is in a very bad inconsistent state; you should
 reinstall it before attempting configuration

After running the commands above, this

ISTM that this issue may be caused by the fact that the systemd config has changed, since it seems to only affect our Ubuntu 16.04 machines and works fine on 14.04.

Versions Report

Salt Version:
           Salt: 2017.7.1

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.3
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.12 (default, Nov 19 2016, 06:48:10)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.2.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: Ubuntu 16.04 xenial
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 4.4.59-1-pve
         system: Linux
        version: Ubuntu 16.04 xenial

[gtmanfred]

I have replicated this issue.

@terminalmage shouldn't this have been solved by using systemd-run --scope to move the apt-get and dpkg processes out of the same tree as the salt minion?

@syphernl a quick fix is to add KillMode=Process in a drop-in file for the salt-minion process. This will allow service to be restarted, but not kill off the minion process that is running the command.

Thanks,
Daniel

[terminalmage]

@gtmanfred Using systemd-run --scope will place the apt-get install command in a different cgroup from the salt-minion process. The question here though is how dpkg manages restarting the service. As I understand it, it will perform a systemctl try-restart salt-minion.service, which should SIGTERM all of the PIDs in the same cgroup as salt-minion.service. Given that the apt-get install command is in a different cgroup, it should survive the service restart. This was actually the reason we made this change in the first place.

[syphernl]

@gtmanfred Using KillMode=process (which appears to be case sensitive) it seems to work fine now.
It properly returns back to the master and the fresh processes (with the new version) are up and running afterwards.

[wwentland]

It appears that there are multiple PRs and bugs that either add or remove KillMode settings from all unit files, while we want to keep it for salt-minion, but not for salt-master.

Please see #33792 for a discussion concerning the salt-master and #29295 for the one concerning salt-minion.

#36806 is the one that triggered this round of removal.