Proper way to upgrade salt-minions / salt-master packages without losing minion connectivity

[shantanub]

We ran through the right way of doing this in salt training with Seth but I think I'm still missing something. I'm not sure if this is a bug or if I've missed something. I tried to run through the upgrade the master first / use salt to upgrade the minion service steps to upgrade from v.17 to v.17.1 of salt and ended up with losing access to most of my minions.

Long story short, I need a reliable way of upgrading all of the salt-minions and salt-master packages without losing access to the minions. From what I can tell, every time I perform such an upgrade I lose access to some if not all of my minions and need to login to each host/VM and restart the salt-minion package. This is doable in test/dev where we have 30 nodes being managed but not when I move this infrastructure to prod where I have over 200 nodes to manage. I need the upgrade path not to break the remote execution framework established between minions and master.

So without further ado here's what I did:

Update the master.

[root@salt-master ~]# yum list updates
Loaded plugins: security 
epel                                                                                                | 3.0 kB     00:00
epel/primary_db                                                                                     | 6.2 MB     00:00
epel-testing                                                                                        | 2.9 kB     00:00
epel-testing/primary_db                                                                             | 2.2 MB     00:00
rhel-localrepo                                                                                      | 3.0 kB     00:00
rhel-localrepo/primary_db                                                                           |  26 MB     00:00
Updated Packages         
glibc.x86_64                                         2.12-1.107.el6_4.5                                      rhel-localrepo
glibc-common.x86_64                                  2.12-1.107.el6_4.5                                      rhel-localrepo
glibc-devel.x86_64                                   2.12-1.107.el6_4.5                                      rhel-localrepo
glibc-headers.x86_64                                 2.12-1.107.el6_4.5                                      rhel-localrepo
java-1.6.0-openjdk.x86_64                            1:1.6.0.0-1.65.1.11.13.el6_4                            rhel-localrepo
kernel.x86_64                                        2.6.32-358.23.2.el6                                     rhel-localrepo
kernel-firmware.noarch                               2.6.32-358.23.2.el6                                     rhel-localrepo
kernel-headers.x86_64                                2.6.32-358.23.2.el6                                     rhel-localrepo
libtar.x86_64                                        1.2.11-17.el6_4.1                                       rhel-localrepo
nscd.x86_64                                          2.12-1.107.el6_4.5                                      rhel-localrepo
perf.x86_64                                          2.6.32-358.23.2.el6                                     rhel-localrepo
salt.noarch                                          0.17.1-1.el6                                            epel-testing
salt-master.noarch                                   0.17.1-1.el6                                            epel-testing
salt-minion.noarch                                   0.17.1-1.el6                                            epel-testing
setup.noarch                                         2.8.14-20.el6_4.1                                       rhel-localrepo
tzdata.noarch                                        2013g-1.el6                                             rhel-localrepo
tzdata-java.noarch                                   2013g-1.el6                                             rhel-localrepo
You have new mail in /var/spool/mail/root
[root@salt-master ~]# yum update -y

I restart the master and minion on my master VM.

[root@salt-master ~]# service salt-master restart
Stopping salt-master daemon:                               [  OK  ]
Starting salt-master daemon:                               [  OK  ]
[root@salt-master ~]# service salt-minion restart
Stopping salt-minion daemon:                               [  OK  ]
Starting salt-minion daemon:                               [  OK  ]

Try to upgrade some of my test minion VMs.

[root@salt-master ~]# salt 'salt-minion*' pkg.upgrade
[root@salt-master ~]# salt 'salt-minion*' pkg.list_upgrades

[root@salt-master ~]# salt -v 'salt-minion*' test.ping
Executing job with jid 20131021102016190263
-------------------------------------------

salt-minion-00:
    Minion did not return
salt-minion-01:
    Minion did not return

I login to each minion VM and restart the salt-minion service.

[root@salt-minion-01 ~]# service salt-minion restart
Stopping salt-minion daemon:                               [FAILED]
Starting salt-minion daemon:                               [  OK  ]
[root@salt-minion-01 ~]# chkconfig --list | grep salt-minion
salt-minion     0:off   1:off   2:on    3:on    4:on    5:on    6:off

Now I can ping the VMs again.

[root@salt-master ~]# salt -v 'salt-minion*' test.ping
Executing job with jid 20131021102229314417
-------------------------------------------

salt-minion-01:
    True                 
salt-minion-00:
    True  

Versions reports:

[root@salt-master ~]# salt --versions-report
           Salt: 0.17.1
         Python: 2.6.6 (r266:84292, May 27 2013, 05:35:12)
         Jinja2: 2.2.1
       M2Crypto: 0.20.2
 msgpack-python: 0.1.13
   msgpack-pure: Not Installed
       pycrypto: 2.0.1
         PyYAML: 3.10
          PyZMQ: 2.2.0.1
            ZMQ: 3.2.4

[root@salt-minion-00 ~]# salt-call --versions-report
           Salt: 0.17.1
         Python: 2.6.6 (r266:84292, May 27 2013, 05:35:12)
         Jinja2: 2.2.1
       M2Crypto: 0.20.2
 msgpack-python: 0.1.13
   msgpack-pure: Not Installed
       pycrypto: 2.0.1
         PyYAML: 3.10
          PyZMQ: 2.2.0.1
            ZMQ: 3.2.4

[root@salt-minion-01 ~]# salt-call --versions-report
           Salt: 0.17.1
         Python: 2.6.8 (unknown, Nov  7 2012, 14:47:45)
         Jinja2: unknown
       M2Crypto: 0.21.1
 msgpack-python: 0.1.12
   msgpack-pure: Not Installed
       pycrypto: 2.3
         PyYAML: 3.08
          PyZMQ: 2.1.9
            ZMQ: 2.2.0

You'll notice that the upgrade proceeded correctly. The packages were upgraded, but the salt-minion services were not restarted as a part of the upgrade process (for both minion VMs - one is RHEL5 and the other is RHEL6). Unfortunately, I didn't think to run the upgrade packages command in verbose mode at the time.

Do I need to find some external remote-execution method to restart all of the minions post-upgrade (mussh/omnitty, etc...)? This is probably not a bug but it's still very frustrating... I'm unlikely to upgrade again until I can figure out how to do this properly.

[kiorky]

you may need my fix (already on git/develop) for #7987

[shantanub]

Was this always a problem or just something specific to the v.17 -> v.17.1 upgrade? I've personally never got it to work reliably across all minion upgrades with past version upgrades but I had assumed I was going about it the wrong way.

I just tried an alternative upgrade approach. Unfortunately, it didn't work either.

[root@salt-master ~]# salt 'test*' cmd.run "yum update -y"
[root@salt-master ~]# salt 'test*' service.restart salt-minion
[root@salt-master ~]# salt 'test*' -v test.ping
Executing job with jid 20131021110335741192
-------------------------------------------

test:
    Minion did not return

[root@test ~]# service salt-minion status
salt-minion is stopped
[root@test ~]# service salt-minion start
Starting salt-minion daemon:                               [  OK  ]

[root@salt-master ~]# salt 'test*' -v test.ping
Executing job with jid 20131021110634961121
-------------------------------------------

test:
    True

The package upgrade seems to stop the running service.

[basepi]

This process has never been as consistent or stable as we would like it to be. But the first thing you need to do is make sure all of your minions have ZMQ 3.2 or higher. That minion that you listed above with ZMQ 2 is definitely going to cause problems with keeping the connection alive or reconnecting.

The rest of the process tends to depend on the init system in question and a lot of other factors. As soon as we get the general bug count under control, we want to dedicate some resources to solving this upgrade problem for good.

[shantanub]

I don't have much of a choice here unless I deviate from the repos. ZMQ3 is only available in epel for RHEL6. I have the latest version of ZMQ offered for RHEL5 which happens to be the 2.2 version listed above.

Minion01 replicates our legacy RHEL5 nodes (which we have a lot of) and I'm using it as a control for testing issues.

Minion00 is running RHEL6 and mimics the configuration of our latest app/service deployments (and where we'd like to transition everything once I have enough cycles to complete migrations).

[shantanub]

I should also say, you're right. Most of my upgrade and losing minions when the salt-master reboots woes have been with RHEL5 and the legacy ZMQ. That said, this latest upgrade to v.17.1 hasn't worked cleanly for any client. Every single upgrade has stopped the salt-minion service and requires me to login and reboot the services.

Can I get around this by using salt-ssh to restart the minion services? I could use an example of how to use salt-ssh. I'm not entirely clear what user / keys / password it uses (or if this is all hidden under the covers with salt's key management system) or how to issue commands. The docs I've run across don't have too many examples of executing shell commands. I'm presuming using the -r flag (raw) works much like "ssh -t"?

[equinoxefr]

Hi,

I have a workaround for this:

For my Linux minion i set a crontab to restart minions every night.

{% if grains['kernel'] == 'Linux' %}
/etc/cron.daily/restart-minion.sh:
  file:                                     # state declaration
    - managed                               # function
    - mode: 755
    - source: salt://tools/restart-minion/restart-minion.sh   # function arg
{% endif %}

On Windows, same thing, i add a task to restart minions.

{% if grains['os'] == 'Windows' %}
addtask:
  cmd.run:
     - name: schtasks /f /create /tn "restart_salt" /ru System /tr "c:\salt\salt-call.exe service.restart salt-minion"  /sc daily
 /st 02:00

Regards

[shantanub]

That's actually quite clever and I'm going to steal that idea.

That said, it would not have helped with the v.17 to v.17.1 upgrade via epel-testing packages. From what I'm seeing, the package upgrade itself stops the running minion service (whether via yum or from within salt's framework -- which makes sense since salt calls yum's methods) which is very peculiar behavior (I don't recall seeing this happen before with any daemon installed from rpm packages). This seems to be a new upgrade artifact that I don't remember seeing before but I've only done 4-5 upgrades so far.

I'm going to run the upgrade in verbose mode and see if I can find any other artifacts but need to troubleshoot some 10GbE networking problems we're having first.

[shantanub]

Maybe I dismissed the idea too early... I could extend your example and poll to see if the service is running (every 5 minutes or something) and if it is not, start the salt-minion.

That seems a little overkill but would resolve this specific issue.

I should be able to use salt-ssh to login to all of minions and start the minion service manually however right?

[equinoxefr]

@shantanub, I look at the spec file. There is some code to stop salt-minion before upgrade and if it's an upgrade to restart the minion with service salt-minion condrestart

I don't know why this part of code doesn't work. Before 0.17 version, the minion wasn't stopped during upgrade.

[basepi]

Strange. I wonder if we changed something between 0.17.0 and 0.17.1 that would have caused this change.

@equinoxefr I can't see any recent changes that changed whether the salt process was stopped or not. Looks like the stop of the salt-minion has been in there for awhile (at least for the 0.16 release, I didn't go earlier). Just wondering where you were looking to see that change.

@shantanub You've had successful upgrades from epel before, then? But not 0.17.0-0.17.1?

[equinoxefr]

@basepi i didn't see any change in the code i have seen a change in the functioning of minion upgrade (on linux RPM).
Before 0.17.1, salt-minion process wasn't restarted after upgrade. If you upgraded from 0.16.3 to 0.17 and launched a test.version just after, salt-minion will say 0.16.3 not 0.17. After a manual restart it works and says 0.17...

Now with 0.17.1 salt-minion is stopped and not restarted. Perhaps the piece of code that use the state of RPM operation doesn't work.

0.16.0 -> 0.16.3 = Minion not restarted
0.16.3 -> 0.17 = Minion not restarted
0.17 -> 0.17.1 = Minion stopped

I did my tests on centos 6.4.

I don't know why but something has changed ;-)

[basepi]

Hrm, well, it doesn't appear to be in the spec file, so it must be somewhere else. Maybe the init script?

[shantanub]

@basepi That's correct. I've always upgraded and used the epel/epel-testing rpms to install salt so far. This is the first time I've noticed the minion stopped after/during the upgrade (this is something that would have been obvious since I would have to login to every minion and restart the service). The upgrade itself seems to have executed fine in every other regard that I can tell (no errors, etc...).

Now, I have in the past lost all of the rhel5 minions when the salt-master service is restarted. I haven't had a problem with that in a few versions, but as I mentioned above, I very much would like to depart from rhel5 as soon as possible.

[shantanub]

Upon upgrading to v.17.2, it looks like the minion restarts as a part of the upgrade just fine.

Has this issue with the package upgrade been resolved?

As a fail-safe, I "start" my minions every 5 minutes via cron just in case they're down for some reason or another. I'll need to add the windows specific scheduler task as well.

I would still like a definitive guide for how-to-upgrade minions and the salt-master. We're moving salt to production once v.17 is available in epel (as opposed to epel-testing), and I'd very much like upgrades to go smoothly.

[basepi]

In general, the upgrades themselves tend to go swimmingly. The problem is the restart after the upgrade. We have an open issue specifically for the restarting of the minion: #5721

The issue also varies in severity from system to system (specifically between different init systems). Making it so the salt minion can restart itself consistently is high on our priority list.