Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] salt-master not running, unable to read key master.pem #57710

Closed
rj-dsl opened this issue Jun 18, 2020 · 13 comments
Closed

[BUG] salt-master not running, unable to read key master.pem #57710

rj-dsl opened this issue Jun 18, 2020 · 13 comments
Labels
Bug broken, incorrect, or confusing behavior
Milestone

Comments

@rj-dsl
Copy link

rj-dsl commented Jun 18, 2020

Description
Salt-master refused to start after reboot, citing Unable to read key: /etc/salt/pki/master/master.pem; passphrase may be incorrect as the issue

Setup
Running on Ubuntu 16.04 that has been updated from 14.04 quite a while ago.. Upstart is installed along systemd, service as been running fine for a few years like this.

Steps to Reproduce the behavior
salt-master debug:

salt-master -l debug
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Including configuration from '/etc/salt/master.d/nodegroups.conf'
[DEBUG   ] Reading configuration from /etc/salt/master.d/nodegroups.conf
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: salt
[DEBUG   ] Configuration file path: /etc/salt/master
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO    ] Setting up the Salt Master
[DEBUG   ] Marking 'base64_encode' as a jinja filter
[DEBUG   ] Marking 'base64_decode' as a jinja filter
[DEBUG   ] Marking 'md5' as a jinja filter
[DEBUG   ] Marking 'sha1' as a jinja filter
[DEBUG   ] Marking 'sha256' as a jinja filter
[DEBUG   ] Marking 'sha512' as a jinja filter
[DEBUG   ] Marking 'hmac' as a jinja filter
[DEBUG   ] Marking 'hmac_compute' as a jinja filter
[DEBUG   ] Marking 'random_hash' as a jinja filter
[DEBUG   ] Marking 'rand_str' as a jinja filter
[DEBUG   ] Marking 'file_hashsum' as a jinja filter
[DEBUG   ] Marking 'http_query' as a jinja filter
[DEBUG   ] Marking 'strftime' as a jinja filter
[DEBUG   ] Marking 'date_format' as a jinja filter
[DEBUG   ] Marking 'raise' as a jinja global
[DEBUG   ] Marking 'match' as a jinja test
[DEBUG   ] Marking 'equalto' as a jinja test
[DEBUG   ] Marking 'skip' as a jinja filter
[DEBUG   ] Marking 'sequence' as a jinja filter
[DEBUG   ] Marking 'to_bool' as a jinja filter
[DEBUG   ] Marking 'indent' as a jinja filter
[DEBUG   ] Marking 'tojson' as a jinja filter
[DEBUG   ] Marking 'quote' as a jinja filter
[DEBUG   ] Marking 'regex_escape' as a jinja filter
[DEBUG   ] Marking 'regex_search' as a jinja filter
[DEBUG   ] Marking 'regex_match' as a jinja filter
[DEBUG   ] Marking 'regex_replace' as a jinja filter
[DEBUG   ] Marking 'uuid' as a jinja filter
[DEBUG   ] Marking 'unique' as a jinja filter
[DEBUG   ] Marking 'min' as a jinja filter
[DEBUG   ] Marking 'max' as a jinja filter
[DEBUG   ] Marking 'avg' as a jinja filter
[DEBUG   ] Marking 'union' as a jinja filter
[DEBUG   ] Marking 'intersect' as a jinja filter
[DEBUG   ] Marking 'difference' as a jinja filter
[DEBUG   ] Marking 'symmetric_difference' as a jinja filter
[DEBUG   ] Marking 'method_call' as a jinja filter
[DEBUG   ] Marking 'yaml_dquote' as a jinja filter
[DEBUG   ] Marking 'yaml_squote' as a jinja filter
[DEBUG   ] Marking 'yaml_encode' as a jinja filter
[DEBUG   ] salt.crypt.get_rsa_key: Loading private key
[DEBUG   ] salt.crypt._get_key_with_evict: Loading private key
[ERROR   ] Unable to read key: /etc/salt/pki/master/master.pem; passphrase may be incorrect
Unable to read key: /etc/salt/pki/master/master.pem; passphrase may be incorrect

Expected behavior
That it works :/

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
           Salt: 3001
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
         Jinja2: 2.8
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: 3.4.7
         pygit2: Not Installed
         Python: 3.5.2 (default, Apr 16 2020, 17:47:17)
   python-gnupg: 0.3.8
         PyYAML: 3.11
          PyZMQ: 17.1.2
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.1.4
 
System Versions:
           dist: ubuntu 16.04 Xenial Xerus
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-184-generic
         system: Linux
        version: Ubuntu 16.04 Xenial Xerus

Additional context
Add any other context about the problem here.

@rj-dsl rj-dsl added the Bug broken, incorrect, or confusing behavior label Jun 18, 2020
@rj-dsl rj-dsl changed the title [BUG] [BUG] salt-master not running, unable to read key master.pem Jun 18, 2020
@krionbsd
Copy link
Contributor

Just for my understanding: did you upgrade to 3001 and see this error or you did upgrade to Ubuntu 16 from 14? If it's upgrade to 16, did you reinstall all Ubuntu packages afterwards?

@rj-dsl
Copy link
Author

rj-dsl commented Jun 18, 2020

The server has been upgraded to 16.04 for some time now.. upgrade to 3001 happened while on 16.04.. As far as I can see in my apt log, the following tried to update last night, and failed:

Start-Date: 2020-06-18  04:11:45
Commandline: /usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef dist-upgrade
Install: python3-pycryptodome:amd64 (3.4.7-2+ds, automatic), python3-distro:amd64 (1.0.1-2, automatic)
Upgrade: salt-master:amd64 (3000.3+ds-1, 3001+ds-1), python-zmq:amd64 (15.2.0-0ubuntu4, 17.1.2-3ubuntu2+ds), salt-minion:amd64 (3000.3+ds-1, 3001+ds-1), salt-common:amd64 (3000.3+ds-1, 3001+ds-1), python3-zmq:amd64 (15.2.0-0ubuntu4, 17.1.2-3ubuntu2+ds)
Error: Sub-process /usr/bin/dpkg returned an error code (1)
End-Date: 2020-06-18  04:13:58

However there nothing pending when I run apt update, apt upgrade or apt dist-upgrade now

@rj-dsl
Copy link
Author

rj-dsl commented Jun 18, 2020

Checking some of the failed packages tells me that the are indeed installed, and with correct versions:

apt list --installed | grep salt-master

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

salt-master/unknown,unknown,now 3001+ds-1 all [installed]

apt list --installed | grep python3-pycryptodome

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

python3-pycryptodome/unknown,now 3.4.7-2+ds amd64 [installed,automatic]

@krionbsd
Copy link
Contributor

Could you backup /etc/salt/pki/master/master.pem, delete it and try to restart master, afterwards delete /etc/salt/pki/minion_master.pub on the minion, restart it and re-accept the key on master.

@rj-dsl
Copy link
Author

rj-dsl commented Jun 18, 2020

That did the trick.. I did however need to do this on around 70 minions.. But is up and running again

@xeacott
Copy link
Contributor

xeacott commented Jun 18, 2020

Hey @rj-dsl are we able to close this issue? It looks like you're good to go, but sorry about the manual effort required ...

@xeacott xeacott added this to the Blocked milestone Jun 18, 2020
@rj-dsl
Copy link
Author

rj-dsl commented Jun 18, 2020

yes, sorry :)

@rj-dsl rj-dsl closed this as completed Jun 18, 2020
@dumol
Copy link
Contributor

dumol commented Jun 23, 2020

Having the same issues after upgrading from 3000.3+ds-1 to 3001+ds-1 on Ubuntu Server 18.04 LTS. The master keys were not changed for 7 years.

Is there any other solution except regenerate the master keys? It's not always easy to copy the new public key over to all minions through a separate channel… Thanks!

@dumol
Copy link
Contributor

dumol commented Jun 23, 2020

For people trying to get back to previous versions… It's not enough to downgrade to 3000.3, e.g. with:

dpkg -i /var/cache/apt/archives/salt*3000.3*

You'll also have to purge python3-pycryptodome afterwards, e.g.:

apt-get autoremove --purge -V

@xcorvis
Copy link

xcorvis commented Aug 24, 2020

I have a similar setup - 16.04 with old master keys, going from 3000 to 3001. I am not looking forward to manually re-accepting keys on 2000 servers.

This should be in the release notes.

@jwolfe-ns
Copy link

jwolfe-ns commented Nov 5, 2020

Definitely not fun, but at least the master public key being a mismatch is not a hard error. Meaning the minion won't die and will retry, so you can do something like:

salt '*' cmd.run 'sleep 60; rm -f /etc/salt/pki/minion/minion_master.pub; systemctl restart salt-minion' bg=True
rm /etc/salt/pki/master/master.pem /etc/salt/pki/master/master.pub
systemctl restart salt-master

The issue will come from any minions that aren't connected or fail to restart, of which there will certainly be in big installs. At least you don't have to around restarting all minions OOB though.

@baby-gnu
Copy link

Hello.

This is only for the record since I hit the same issue.

My problem comes from pycrptodome consistency check:

>>> from Cryptodome.PublicKey import RSA
>>> path = '/etc/salt/pki/master/master.pem'
>>> key_fh = open(path)
>>> RSA.importKey(key_fh.read())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/Cryptodome/PublicKey/RSA.py", line 727, in import_key
    return _import_keyDER(der, passphrase)
  File "/usr/lib/python3/dist-packages/Cryptodome/PublicKey/RSA.py", line 670, in _import_keyDER
    raise ValueError("RSA key format is not supported")
ValueError: RSA key format is not supported

I bypass this issue by installing the preferred M2Crypto library which does not trigger this issue:

>>> from M2Crypto import RSA
>>> path = '/etc/salt/pki/master/master.pem'
>>> RSA.load_key(path)
<M2Crypto.RSA.RSA object at 0x7fd2a03f3c88>

@mgenti
Copy link

mgenti commented Mar 30, 2022

Thanks to @baby-gnu that also worked for me. Installed M2Crypto 0.38.0 and then upgraded to the latest salt

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug broken, incorrect, or confusing behavior
Projects
None yet
Development

No branches or pull requests

8 participants