Salt minion bombs out during start-up when private key changes in open mode

[mbirtwell]

Most of the salt-minions in my system are pxe booting vms without hard disks. This means I can't store the minion private key between boots, so I've been running in open mode. After upgrading to 0.17.1 from 0.15.3 the minion is crashing during start-up with the following exception:

Nov 11 12:21:52 db-1 <U+FEFF>salt-minion[1549]: [salt ][INFO ] Setting up the Salt Minion "db-1.dev.dev
-colo"
Nov 11 12:21:53 db-1 <U+FEFF>salt-minion[1549]: [salt.crypt ][INFO ] Generating keys: /etc/salt/pki/minion
Nov 11 12:21:53 db-1 <U+FEFF>salt-minion[1549]: [salt.log.setup ][ERROR ] An un-handled exception was caught by salt's global exception handler:
Nov 11 12:21:53 db-1 salt-minion[1549]: Traceback (most recent call last):
Nov 11 12:21:53 db-1 salt-minion[1549]: File "/usr/bin/salt-minion", line 14, in
Nov 11 12:21:53 db-1 salt-minion[1549]: salt_minion()
Nov 11 12:21:53 db-1 salt-minion[1549]: File "/usr/lib/python2.7/site-packages/salt/scripts.py", line 30, in salt_minion
Nov 11 12:21:53 db-1 salt-minion[1549]: minion.start()
Nov 11 12:21:53 db-1 salt-minion[1549]: File "/usr/lib/python2.7/site-packages/salt/init.py", line 212, in start
Nov 11 12:21:53 db-1 salt-minion[1549]: self.prepare()
Nov 11 12:21:53 db-1 salt-minion[1549]: File "/usr/lib/python2.7/site-packages/salt/init.py", line 200, in prepare
Nov 11 12:21:53 db-1 salt-minion[1549]: self.minion = salt.minion.Minion(self.config)
Nov 11 12:21:53 db-1 salt-minion[1549]: File "/usr/lib/python2.7/site-packages/salt/minion.py", line 474, in init
Nov 11 12:21:53 db-1 salt-minion[1549]: self.authenticate(timeout, safe)
Nov 11 12:21:53 db-1 salt-minion[1549]: File "/usr/lib/python2.7/site-packages/salt/minion.py", line 886, in authenticate
Nov 11 12:21:53 db-1 salt-minion[1549]: creds = auth.sign_in(timeout, safe)
Nov 11 12:21:53 db-1 salt-minion[1549]: File "/usr/lib/python2.7/site-packages/salt/crypt.py", line 320, in sign_in
Nov 11 12:21:53 db-1 salt-minion[1549]: auth['aes'] = self.verify_master(payload)
Nov 11 12:21:53 db-1 salt-minion[1549]: File "/usr/lib/python2.7/site-packages/salt/crypt.py", line 265, in verify_master
Nov 11 12:21:53 db-1 salt-minion[1549]: aes, token = self.decrypt_aes(payload, False)
Nov 11 12:21:53 db-1 salt-minion[1549]: File "/usr/lib/python2.7/site-packages/salt/crypt.py", line 213, in decrypt_aes
Nov 11 12:21:53 db-1 salt-minion[1549]: key_str = key.private_decrypt(payload['aes'], RSA.pkcs1_oaep_padding)
Nov 11 12:21:53 db-1 salt-minion[1549]: File "/usr/lib64/python2.7/site-packages/M2Crypto/RSA.py", line 63, in private_decrypt
Nov 11 12:21:53 db-1 salt-minion[1549]: return m2.rsa_private_decrypt(self.rsa, data, padding)
Nov 11 12:21:53 db-1 salt-minion[1549]: M2Crypto.RSA.RSAError: oaep decoding error

The logs on the salt master do not show any errors.
If I delete the the minion's key from the salt master or save and restore the minion's key before running the minion it works.

[basepi]

Hrm, thanks for the report. You say you can resolve it by deleting the key on the master, correct? No change required on the minion?

[mbirtwell]

Yes well except for restarting the minion. I'm pretty sure it's the change
in key that triggers it I've since also tried deleting the keys and
restarting the service. That also triggers the problem.

On Monday, 11 November 2013, Colton Myers wrote:

Hrm, thanks for the report. You say you can resolve it by deleting the key
on the master, correct? No change required on the minion?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/8402#issuecomment-28247444
.

[basepi]

Right, I think what's going on here is that even though the master is in open mode, it has a key for a given minion ID, and if the same minion ID tries to auth with a different key, it refuses because it expects it may be an imposter minion trying to masquerade as the proper minion. Open mode doesn't get rid of key comparisons, but rather just automatically accepts keys (or so I'm guessing).

I want to say this is actually a duplicate issue, but I can't find it off hand, so we'll leave this open. I think we just need to get rid of that key comparison in open mode (because it's already not secure, so I'm not sure it's gaining us anything).

[basepi]

This should be fixed as of the above pull req. Can someone please verify?

[mbirtwell]

It looks good to me. I've only tested the change cherry-picked on to the 0.17 branch though.

[basepi]

Awesome, thanks.