Cryptodome breaks FIPS Compliance in 2016.11.4-1 #40890

Closed
robthralls opened this Issue Apr 26, 2017 · 9 comments

Comments

Projects
None yet
4 participants
@robthralls

Description of Issue/Question

Master and minions are running RHEL 6.9 in FIPS mode, just upgraded from 2016.11.3-1 packages to 2016.11.4-1 and ran into FIPS issues again, looks to be related to the new Cryptodome stuff. I do use 'hash_type: sha256' on the master and all minions.

This is what happens when trying to start the services:

# service salt-master start
Starting salt-master daemon:  salt-master[ERROR   ] error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/salt/utils/parsers.py", line 200, in parse_args
    process_option_func()
  File "/usr/lib/python2.6/site-packages/salt/utils/parsers.py", line 520, in process_config_dir
    self.config.update(self.setup_config())
  File "/usr/lib/python2.6/site-packages/salt/utils/parsers.py", line 1714, in setup_config
    return config.master_config(self.get_config_file_path())
  File "/usr/lib/python2.6/site-packages/salt/config/__init__.py", line 3299, in master_config
    opts = apply_master_config(overrides, defaults)
  File "/usr/lib/python2.6/site-packages/salt/config/__init__.py", line 3318, in apply_master_config
    import salt.crypt
  File "/usr/lib/python2.6/site-packages/salt/crypt.py", line 27, in <module>
    from Cryptodome.Cipher import AES, PKCS1_OAEP
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Cipher/__init__.py", line 86, in <module>
    from Cryptodome.Cipher._mode_siv import _create_siv_cipher
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Cipher/_mode_siv.py", line 42, in <module>
    from Cryptodome.Protocol.KDF import _S2V
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Protocol/KDF.py", line 39, in <module>
    from Cryptodome.Hash import SHA1, SHA256, HMAC, CMAC
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Hash/HMAC.py", line 81, in <module>
    import MD5, BLAKE2s
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Hash/MD5.py", line 79, in <module>
    new = __make_constructor()
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Hash/MD5.py", line 49, in __make_constructor
    h = _hash_new()
ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
Usage: salt-master [options]

salt-master: error: Error while processing <bound method Master.process_config_dir of <salt.cli.daemons.Master object at 0x2557a10>>: Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/salt/utils/parsers.py", line 200, in parse_args
    process_option_func()
  File "/usr/lib/python2.6/site-packages/salt/utils/parsers.py", line 520, in process_config_dir
    self.config.update(self.setup_config())
  File "/usr/lib/python2.6/site-packages/salt/utils/parsers.py", line 1714, in setup_config
    return config.master_config(self.get_config_file_path())
  File "/usr/lib/python2.6/site-packages/salt/config/__init__.py", line 3299, in master_config
    opts = apply_master_config(overrides, defaults)
  File "/usr/lib/python2.6/site-packages/salt/config/__init__.py", line 3318, in apply_master_config
    import salt.crypt
  File "/usr/lib/python2.6/site-packages/salt/crypt.py", line 27, in <module>
    from Cryptodome.Cipher import AES, PKCS1_OAEP
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Cipher/__init__.py", line 86, in <module>
    from Cryptodome.Cipher._mode_siv import _create_siv_cipher
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Cipher/_mode_siv.py", line 42, in <module>
    from Cryptodome.Protocol.KDF import _S2V
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Protocol/KDF.py", line 39, in <module>
    from Cryptodome.Hash import SHA1, SHA256, HMAC, CMAC
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Hash/HMAC.py", line 81, in <module>
    import MD5, BLAKE2s
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Hash/MD5.py", line 79, in <module>
    new = __make_constructor()
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Hash/MD5.py", line 49, in __make_constructor
    h = _hash_new()
ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips


# service salt-minion start
ERROR: Unable to look-up config values for /etc/salt

Nothing written to either /var/log/salt/master or /var/log/salt/minion.

Steps to Reproduce Issue

  1. Enable FIPS (install dracut-fips + fipscheck, set fips=1 and boot= kernel command line arguments in GRUB, dracut -f, reboot).
  1. Upgrade to 2016.11.4-1.el6 salt, salt-minion, salt-master packages. libtommath, libtomcrypt, and python2-pycryptodomex packages are installed as new dependencies.

Versions Report

# salt --versions-report
Traceback (most recent call last):
  File "/usr/bin/salt", line 10, in <module>
    salt_main()
  File "/usr/lib/python2.6/site-packages/salt/scripts.py", line 464, in salt_main
    client.run()
  File "/usr/lib/python2.6/site-packages/salt/cli/salt.py", line 33, in run
    import salt.client
  File "/usr/lib/python2.6/site-packages/salt/client/__init__.py", line 31, in <module>
    import salt.cache
  File "/usr/lib/python2.6/site-packages/salt/cache/__init__.py", line 15, in <module>
    from salt.payload import Serial
  File "/usr/lib/python2.6/site-packages/salt/payload.py", line 17, in <module>
    import salt.crypt
  File "/usr/lib/python2.6/site-packages/salt/crypt.py", line 27, in <module>
    from Cryptodome.Cipher import AES, PKCS1_OAEP
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Cipher/__init__.py", line 86, in <module>
    from Cryptodome.Cipher._mode_siv import _create_siv_cipher
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Cipher/_mode_siv.py", line 42, in <module>
    from Cryptodome.Protocol.KDF import _S2V
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Protocol/KDF.py", line 39, in <module>
    from Cryptodome.Hash import SHA1, SHA256, HMAC, CMAC
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Hash/HMAC.py", line 81, in <module>
    import MD5, BLAKE2s
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Hash/MD5.py", line 79, in <module>
    new = __make_constructor()
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Hash/MD5.py", line 49, in __make_constructor
    h = _hash_new()
ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips
@BernhardGruen

This comment has been minimized.

Show comment
Hide comment
@BernhardGruen

BernhardGruen Apr 26, 2017

Hi,
did you already try to forcefully uninstall the cryptodome package python2-pycryptodomex (it is a install dependency)?
I did that for a CentOS 7 install as described in the comment of issue #40889 .
Maybe this helps you too.

Hi,
did you already try to forcefully uninstall the cryptodome package python2-pycryptodomex (it is a install dependency)?
I did that for a CentOS 7 install as described in the comment of issue #40889 .
Maybe this helps you too.

@Ch3LL

This comment has been minimized.

Show comment
Hide comment
@Ch3LL

Ch3LL Apr 26, 2017

Contributor

Looks like i'm able to replicate this. We will need to get this fixed thanks

Here is a vagrant file for anyone to quickly replicate this. This installs the latest salt version and enables fips mode.

$script = <<SCRIPT
    yum install dracut-fips -y
    dracut -f
    test="$(blkid /dev/sda1 | awk {'print $2'})"
    echo $test
    sed -i 's,quiet\",quit boot='"${test}"' fips=1",g' /etc/default/grub
    grub2-mkconfig -o /boot/grub2/grub.cfg
    yum install https://repo.saltstack.com/yum/redhat/salt-repo-latest-1.el7.noarch.rpm -y
    yum install salt-master salt-minion -y
    echo "hash_type: sha256" >> /etc/salt/master
    echo "hash_type: sha256" >> /etc/salt/minion
SCRIPT

Vagrant.configure(2) do |config|
  config.vm.box = "hfm4/centos7"
  config.vm.provision "shell", inline: $script
  # reload after script
  config.vm.provision :reload
end
Contributor

Ch3LL commented Apr 26, 2017

Looks like i'm able to replicate this. We will need to get this fixed thanks

Here is a vagrant file for anyone to quickly replicate this. This installs the latest salt version and enables fips mode.

$script = <<SCRIPT
    yum install dracut-fips -y
    dracut -f
    test="$(blkid /dev/sda1 | awk {'print $2'})"
    echo $test
    sed -i 's,quiet\",quit boot='"${test}"' fips=1",g' /etc/default/grub
    grub2-mkconfig -o /boot/grub2/grub.cfg
    yum install https://repo.saltstack.com/yum/redhat/salt-repo-latest-1.el7.noarch.rpm -y
    yum install salt-master salt-minion -y
    echo "hash_type: sha256" >> /etc/salt/master
    echo "hash_type: sha256" >> /etc/salt/minion
SCRIPT

Vagrant.configure(2) do |config|
  config.vm.box = "hfm4/centos7"
  config.vm.provision "shell", inline: $script
  # reload after script
  config.vm.provision :reload
end

@Ch3LL Ch3LL added this to the Approved milestone Apr 26, 2017

@robthralls

This comment has been minimized.

Show comment
Hide comment
@robthralls

robthralls Apr 26, 2017

Updated to 2016.11.4-1 again, ran rpm -e --nodeps python2-pycryptodomex and it looks like that did resolve it for now. The services start, test.ping worked, as did a few simple formulas.

Thanks for the quick responses!

Edit: versions report, in case you want it:

# salt --versions-report
Salt Version:
           Salt: 2016.11.4
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: 1.4.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8.1
        libgit2: 0.20.0
        libnacl: Not Installed
       M2Crypto: 0.20.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.20.3
         Python: 2.6.6 (r266:84292, Aug  9 2016, 06:11:56)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
 
System Versions:
           dist: redhat 6.9 Santiago
        machine: x86_64
        release: 2.6.32-642.1.1.el6.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 6.9 Santiago

The python2-pycryptodomex package it pulls in as a dependency is 3.4.3-1, if that helps.

robthralls commented Apr 26, 2017

Updated to 2016.11.4-1 again, ran rpm -e --nodeps python2-pycryptodomex and it looks like that did resolve it for now. The services start, test.ping worked, as did a few simple formulas.

Thanks for the quick responses!

Edit: versions report, in case you want it:

# salt --versions-report
Salt Version:
           Salt: 2016.11.4
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: unknown
       dateutil: 1.4.1
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.8.1
        libgit2: 0.20.0
        libnacl: Not Installed
       M2Crypto: 0.20.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: 0.20.3
         Python: 2.6.6 (r266:84292, Aug  9 2016, 06:11:56)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
 
System Versions:
           dist: redhat 6.9 Santiago
        machine: x86_64
        release: 2.6.32-642.1.1.el6.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 6.9 Santiago

The python2-pycryptodomex package it pulls in as a dependency is 3.4.3-1, if that helps.

@Ch3LL Ch3LL added Packaging Platform and removed Core labels Apr 26, 2017

@BernhardGruen

This comment has been minimized.

Show comment
Hide comment
@BernhardGruen

BernhardGruen Apr 27, 2017

I found the real reason for that bug: The used public/private key most likely uses an (insecure) exponent of "1" (please use this command to verify: openssl rsa -text -in minion.pem).
The only real solution so far seems to be to regenerate those keys Also documented in issue #40889 ).

I found the real reason for that bug: The used public/private key most likely uses an (insecure) exponent of "1" (please use this command to verify: openssl rsa -text -in minion.pem).
The only real solution so far seems to be to regenerate those keys Also documented in issue #40889 ).

@robthralls

This comment has been minimized.

Show comment
Hide comment
@robthralls

robthralls Apr 27, 2017

You scared me for a minute, but no, that's not it. I've confirmed that my master and minion keys in each of my environments are all using 65537. I actually regenerated all our keys only a couple of months ago and it shouldn't have even been an issue by the time we started using SaltStack anyway.

For kicks, I tried to reproduce in a python shell like in #40889 and couldn't even import the RSA lib:

>>> from Cryptodome.PublicKey import RSA
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.6/site-packages/Cryptodome/PublicKey/RSA.py", line 79, in <module>
    from Cryptodome.IO import PKCS8, PEM
  File "/usr/lib64/python2.6/site-packages/Cryptodome/IO/PKCS8.py", line 73, in <module>
    from Cryptodome.IO._PBES import PBES1, PBES2, PbesError
  File "/usr/lib64/python2.6/site-packages/Cryptodome/IO/_PBES.py", line 43, in <module>
    from Cryptodome.Hash import MD5, SHA1
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Hash/MD5.py", line 79, in <module>
    new = __make_constructor()
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Hash/MD5.py", line 49, in __make_constructor
    h = _hash_new()
ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips

You scared me for a minute, but no, that's not it. I've confirmed that my master and minion keys in each of my environments are all using 65537. I actually regenerated all our keys only a couple of months ago and it shouldn't have even been an issue by the time we started using SaltStack anyway.

For kicks, I tried to reproduce in a python shell like in #40889 and couldn't even import the RSA lib:

>>> from Cryptodome.PublicKey import RSA
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python2.6/site-packages/Cryptodome/PublicKey/RSA.py", line 79, in <module>
    from Cryptodome.IO import PKCS8, PEM
  File "/usr/lib64/python2.6/site-packages/Cryptodome/IO/PKCS8.py", line 73, in <module>
    from Cryptodome.IO._PBES import PBES1, PBES2, PbesError
  File "/usr/lib64/python2.6/site-packages/Cryptodome/IO/_PBES.py", line 43, in <module>
    from Cryptodome.Hash import MD5, SHA1
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Hash/MD5.py", line 79, in <module>
    new = __make_constructor()
  File "/usr/lib64/python2.6/site-packages/Cryptodome/Hash/MD5.py", line 49, in __make_constructor
    h = _hash_new()
ValueError: error:060800A3:digital envelope routines:EVP_DigestInit_ex:disabled for fips

@dmurphy18 dmurphy18 self-assigned this Apr 27, 2017

@dmurphy18

This comment has been minimized.

Show comment
Hide comment
@dmurphy18

This comment has been minimized.

Show comment
Hide comment
@dmurphy18

dmurphy18 May 18, 2017

Contributor

@robthralls With the release of Salt 2016.11.5 issues with use of pycryptodome have been mitigated as far as possible. On the repo.saltstack.com landing page for Redhat, there is a work-around stated to use pycrypto when on a FIPS and 64-bit platform (if 32-bit platform then pycrypto is used automatically).

See saltstack/salt-pack#261 for details of issues with use of MD5 also causing problems in other files.
An issue on this matter has been logged upstream with the pycryptodomex team.

Once a fix becomes available from the pycryptodomex team becomes available, the pycryptodomex package shall be updated such that the work-around is no longer required.

If this is satisfactory, can you please consider closing this issue, or please give feedback on other possible methods of migrating this issue, noting that pycryptodomex is needed for some non-FIPS installations.

Contributor

dmurphy18 commented May 18, 2017

@robthralls With the release of Salt 2016.11.5 issues with use of pycryptodome have been mitigated as far as possible. On the repo.saltstack.com landing page for Redhat, there is a work-around stated to use pycrypto when on a FIPS and 64-bit platform (if 32-bit platform then pycrypto is used automatically).

See saltstack/salt-pack#261 for details of issues with use of MD5 also causing problems in other files.
An issue on this matter has been logged upstream with the pycryptodomex team.

Once a fix becomes available from the pycryptodomex team becomes available, the pycryptodomex package shall be updated such that the work-around is no longer required.

If this is satisfactory, can you please consider closing this issue, or please give feedback on other possible methods of migrating this issue, noting that pycryptodomex is needed for some non-FIPS installations.

@robthralls

This comment has been minimized.

Show comment
Hide comment
@robthralls

robthralls May 18, 2017

Yes, this issue can be closed. Sorry, I didn't realize you were waiting for me. I'll keep an eye on the upstream issue (Legrandin/pycryptodome#64).

Just out of curiosity, I did see mention that pycrypto is "outdated" and that everything will eventually be moved to m2Crypto (#40503). Is pycrypto insecure in any way, or just a pain to work with or something? Is there a public discussion on this somewhere?

Yes, this issue can be closed. Sorry, I didn't realize you were waiting for me. I'll keep an eye on the upstream issue (Legrandin/pycryptodome#64).

Just out of curiosity, I did see mention that pycrypto is "outdated" and that everything will eventually be moved to m2Crypto (#40503). Is pycrypto insecure in any way, or just a pain to work with or something? Is there a public discussion on this somewhere?

@robthralls robthralls closed this May 18, 2017

@dmurphy18

This comment has been minimized.

Show comment
Hide comment
@dmurphy18

dmurphy18 May 22, 2017

Contributor

@robthralls pycrypto on GitHub appears to have become dormant, hence the move to pycryptodomex, at least on Redhat, Debian/Ubuntu have some updates to their packages..
The M2Crypto was dormant, but has become active again, hence once full 64-bit is supported, and other items then SaltStack may revert to using it in the future, noting that it will require code changes etc.
However no plans to use it yet, but once Nitrogen is released, that may be revisited for some future major point release.

Contributor

dmurphy18 commented May 22, 2017

@robthralls pycrypto on GitHub appears to have become dormant, hence the move to pycryptodomex, at least on Redhat, Debian/Ubuntu have some updates to their packages..
The M2Crypto was dormant, but has become active again, hence once full 64-bit is supported, and other items then SaltStack may revert to using it in the future, noting that it will require code changes etc.
However no plans to use it yet, but once Nitrogen is released, that may be revisited for some future major point release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment