Debian: upgrading from 2015.8.3 with pkg.install fails

[anlutro]

andeas@orch:~$ salt 'elk*' pkg.install salt-minion refresh=True
elk.test.infra.vagrant.local:
    Minion did not return. [No response]

root@elk:/var/log/apt# apt-get install
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.

root@elk:/var/log/apt# dpkg --configure -a
Setting up python-tz (2012c+dfsg-0.1) ...
dpkg: error processing package salt-minion (--configure):
 package is in a very bad inconsistent state; you should
 reinstall it before attempting configuration
Setting up python-croniter (0.3.4-1) ...
Errors were encountered while processing:
 salt-minion

root@elk:/var/log/apt# apt-get install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
You might want to run 'apt-get -f install' to correct these.
The following packages have unmet dependencies:
E: Unmet dependencies. Try using -f.

root@elk:/var/log/apt# apt-get install -f
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  python-mako
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  salt-common salt-minion
Suggested packages:
  python-augeas
The following packages will be upgraded:
  salt-common salt-minion
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 0 B/3,132 kB of archives.
After this operation, 569 kB of additional disk space will be used.
Do you want to continue? [Y/n] n
Abort.

Oddly enough, if I SSH into the box and manually run the exact same apt-get install command that salt does (based on the minion logs), the package installs just fine.

$ apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install salt-minion

This happens when upgrading from 2015.8.3 to .4 or .5. Upgrading from .4 to .5 went fine.

dpkg.log:

2016-02-05 14:40:42 startup archives unpack
2016-02-05 14:40:42 upgrade salt-minion:all 2015.8.3+ds-1 2015.8.5+ds-1
2016-02-05 14:40:42 status half-configured salt-minion:all 2015.8.3+ds-1

apt/history.log:

Start-Date: 2016-02-05  14:40:42
Commandline: apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install salt-minion
Upgrade: salt-minion:amd64 (2015.8.3+ds-1, 2015.8.5+ds-1), salt-common:amd64 (2015.8.3+ds-1, 2015.8.5+ds-1)

apt/term.log:

Log started: 2016-02-05  14:40:42
(Reading database ... 55552 files and directories currently installed.)
Preparing to unpack .../salt-minion_2015.8.5+ds-1_all.deb ...

[wwentland]

We have seen various reports of this in #salt and it looks as if something messed up the upgrade path.

[anlutro]

And of course, if I run the pkg.install on the salt minion with salt-call -l debug, it works fine... Incredibly annoying.

[wwentland]

I take it that we see the same behaviour if you run pkg.upgrade or pkg.upgrade dist_upgrade=True ?

[anlutro]

@BABILEN correct.

[anlutro]

I also tried upgrading with cmd.run out of curiosity, but that also failed.

andreas@orch:~$ salt 'elk*' cmd.run 'apt-get -q update && apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install salt-minion'
elk.test.infra.vagrant.local:
    Minion did not return. [No response]

root@elk:~# apt-get install
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.

[cmarzullo]

I've solved this issue by scheduling the upgrade with at. Suboptimal, but it does work. It won't fix your broken installs though.

# jobs/schedule_salt_upgrade.sls
jobs_salt_upgrade_script:
  file.managed:
    - name: '/tmp/salt_upgrade.sh'
    - source: 'salt://jobs/files/salt_upgrade.sh'
    - mode: 0700
    - user: root
    - group: root

jobs_salt_upgrade_schedule:
  at.present:
    - job: '/tmp/salt_upgrade.sh'
    - timespec: 'now +5 min'
    - user: root

#jobs/files/salt_upgrade.sh
#!/bin/bash
# Find salt packages to upgrade
salt_pkgs=$(dpkg-query -l 'salt*' | egrep '^ii' | awk  '{print $2}' | tr '\n' ' ')

# Upgrade salt packages
apt-get install -o Dpkg::Options::="--force-confold" $salt_pkgs -y

# remove any unused packages
apt-get autoremove -y

# clean up
rm /tmp/salt_upgrade.sh

[dmurphy18]

Thinking this may be related to the incorrect reliance on python-systemd in 2015.8.4 which has since been removed in 2015.8.5.

[anlutro]

If that was the case, I don't think running the apt-get install command manually would work, either. It seems the thing triggering the fail is if the apt-get install is a subprocess of the salt minion.

[dmurphy18]

@anlutro trying your commands now

[dmurphy18]

Looks like this is due to salt.loaded.int.module.nova.virtual() is wrongly returning None. #29445

Seeing the loader ERROR out due to exception virtual function for nova

[anlutro]

I've seen that error in my logs for a long time and it hasn't caused any real problems. Why does it cause the package upgrade to fail? And why only this particular upgrade path? Like I said, upgrading from 2015.8.4 to .5 worked fine.

[terminalmage]

The error referenced by @dmurphy18 would not cause this problem.

@anlutro What version was your master running when you performed this upgrade?

[anlutro]

2015.8.3

On Fri, 12 Feb 2016 18:22 Erik Johnson notifications@github.com wrote:

The error referenced by @dmurphy18 https://github.com/dmurphy18 would
not cause this problem.

@anlutro https://github.com/anlutro What version was your master
running when you performed this upgrade?

—
Reply to this email directly or view it on GitHub
#30937 (comment).

[terminalmage]

OK. It is discouraged to update the minion before the master, because sometimes changes in Salt require the minion's responses to the master to change, and an older master doesn't know how to deal with those changes in the minion's communication payloads.

I'm trying to reproduce this right now, but if you have any 2015.8.3 minions remaining, could you try upgrading the master first and then upgrade the minion?

[anlutro]

Right... I will try in our test environment on Monday. Though a pillar
regression is keeping us from upgrading in production.

On Fri, 12 Feb 2016 18:25 Erik Johnson notifications@github.com wrote:

OK. It is discouraged to update the minion before the master, because
sometimes changes in Salt require the minion's responses to the master to
change, and an older master doesn't know how to deal with those changes in
the minion's communication payloads.

I'm trying to reproduce this right now, but if you have any 2015.8.3
minions remaining, could you try upgrading the master first and then
upgrade the minion?

—
Reply to this email directly or view it on GitHub
#30937 (comment).

[anlutro]

@terminalmage: I tried upgrading the salt-master first, and the exact same problem occurs.

root@orch:~# salt '*' test.version
fullstack-mage.test.testcustomer.vagrant.local:
    2015.8.3
orch.test.infra.vagrant.local:
    2015.8.5
root@orch:~# salt 'full*' pkg.install salt-minion refresh=True
fullstack-mage.test.testcustomer.vagrant.local:
    Minion did not return. [No response]

root@fullstack-mage:~# apt-get install
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.

[wwentland]

Maybe closing this was a bit premature?

[Ch3LL]

I was able to replicate this upgrading from 2015.8.3 to 2015.8.7:

SALTMASTER:

ch3ll@ch3ll-centos:~$ date; sudo salt 'debian' test.version
Fri Feb 19 09:29:19 MST 2016
debian:
    2015.8.3

ch3ll@ch3ll-centos:~$ date; sudo salt 'debian' pkg.install salt-minion refresh=True                                                                                                     
Fri Feb 19 09:30:50 MST 2016
debian:
    Minion did not return. [Not connected]

DEBIAN8 VM:

root@localhost:~# apt-get install
E: dpkg was interrupted, you must manually run 'dpkg --configure -a' to correct the problem. 

root@localhost:~# salt-minion --version                                                                                                                               
salt-minion 2015.8.3 (Beryllium)

Also to note I did not see this behavior when upgrading from 2015.8.4 to 2015.8.7

Ping @terminalmage ^ my test case

[terminalmage]

After doing a number of tests (including replicating the results that @Ch3LL saw), all signs point to the problem being with the creation of the 2015.8.3 packages for Debian 8. I could not reproduce this with the Ubuntu 14.04 packages for 2015.8.3, and while testing with Debian, upgrading from any 2015.8.x release to any other 2015.8.x release worked fine, with the exception of upgrading from 2015.8.3 to anything else.

[anlutro]

As long as it's not an issue with new releases, I guess there's not much to do about this. Though it would be reassuring to find the root cause so that whatever mistake was made doesn't happen again.

[terminalmage]

Yeah, we're still doing investigation internally to see if we can narrow this down. If possible, we will update the 2015.8.3 archived packages to correct this.

[dmurphy18]

Just tried upgrading from 2015.8.3 to 2015.8.7 from the command line with apt-get and had no issues, apart from a screen pop-up informing me of new updates holding a lock, once that was 'remind me later' the upgrade was fine. Had installed 2015.8.3 and upgraded by the following steps:

edited sources.lst for 2015.8.7
apt-get update
apt-ger install salt-minion

everything worked fine.

Investigating further, getting with Ch3ll

[anlutro]

Yes, that worked fine for me as well. As did salt-call pkg.upgrade salt-minion.

It seems like the pkg.upgrade only fails when it's done remotely through a master/minion connection - so possibly, the error only happens when the pkg.upgrade function call happens as a subthread/subprocess of the salt-minion process.

[dmurphy18]

The salt-minion.service should contain KillMode=process but doews not.
Present in the code but not in the built salt-minion .deb,, rebuilding it is there correctly, so need to think about viable solution.

[dmurphy18]

The missing KillMode=process affects 2015.5.5 through 2015.5.8, and 20915.8.0 through 2015.8.3
noting the issue propagates from Debian build of 2015.5.3, salt-minion.service in that code is also missing KillMode=process

[dmurphy18]

@anlutro A Warning notice for this issue will be made in the release notes for the next point release (which is extremely immanent). With that I would like to close this issue, since it is a 'will not fix'

[anlutro]

Sounds good to me.