[BUG] Salt 3006.1 x509 module breaks on RHEL 9 with m2crypto

[danielbakken]

Description
Salt 3006.1 x509 module breaks on RHEL 9 with m2crypto

Setup
RHEL 9 minion with Salt 3006.1 onedir installed from yum package. x509_v2 is not enabled, and m2crypto is installed with salt-pip.

• vmware vm
• onedir packaging

Steps to Reproduce the behavior

1. Install salt-minion 3006.1 on RHEL 9, and don't enable x509_v2
2. salt-pip install m2crypto
3. attempt to read an x509 certificate

# salt-call x509.read_certificate /path/to/cert.crt
'x509' __virtual__ returned False: Could not load x509 module, m2crypto unavailable

Expected behavior
x509 certificate details are output as a dictionary, and no error is returned. This command works as expected on RHEL 8 with m2crypto on Salt 3006.1 and 3005.1

Versions Report

MINION:

salt-pip show m2crypto
Name: M2Crypto
Version: 0.38.0
Summary: M2Crypto: A Python crypto and SSL toolkit
Home-page: https://gitlab.com/m2crypto/m2crypto
Author: Ng Pheng Siong
Author-email: ngps@sandbox.rulemaker.net
License: MIT
Location: /opt/saltstack/salt/extras-3.10
Requires:
Required-by:

salt-call --versions-report
Salt Version:
Salt: 3006.1

Python Version:
        Python: 3.10.11 (main, May  5 2023, 02:31:54) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
        relenv: 0.12.3
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: rhel 9.2 Plow
        locale: utf-8
       machine: x86_64
       release: 5.14.0-162.23.1.el9_1.x86_64
        system: Linux
       version: Red Hat Enterprise Linux 9.2 Plow

MASTER

salt --versions-report
Salt Version:
Salt: 3006.1

Python Version:
        Python: 3.10.11 (main, May  5 2023, 02:31:54) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: 0.38.0
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
        relenv: 0.12.3
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
Salt Extensions:
        SSEAPE: 8.11.2.6
 
System Versions:
          dist: rhel 8.7 Ootpa
        locale: utf-8
       machine: x86_64
       release: 4.18.0-425.19.2.el8_7.x86_64
        system: Linux
       version: Red Hat Enterprise Linux 8.7 Ootpa

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

[OrangeDog]

Was there anything logged when you installed M2Crypto?
Is there anything in the minion logs?
Is there any output from /opt/saltstack/salt/bin/python3 -c 'import M2Crypto'?

[danielbakken]

Was there anything logged when you installed M2Crypto?

# salt-pip install m2crypto
Using pip 22.3.1 from /opt/saltstack/salt/lib/python3.10/site-packages/pip (python 3.10)
Collecting m2crypto
  Using cached M2Crypto-0.38.0-cp310-cp310-linux_x86_64.whl
Installing collected packages: m2crypto
  WARNING: In `rpath_only mode` but /lib64/libssl.so.3 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/libcrypto.so.3 is not in /opt/saltstack/salt
  Do not adjust rpath of /tmp/pip-target-behw1y45/M2Crypto/_m2crypto.cpython-310-x86_64-linux-gnu.so
Successfully installed m2crypto-0.38.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.3.1 -> 23.1.2
[notice] To update, run: /opt/saltstack/salt/bin/python3.10 -m pip install --upgrade pip

Is there anything in the minion logs?

2023-05-17 06:54:52,455 [salt.utils.parsers:1060][WARNING ][14561] Minion received a SIGTERM. Exiting.
2023-05-17 06:54:57,591 [tornado.general  :567 ][WARNING ][14672] Got events for closed stream <zmq.eventloop.zmqstream.ZMQStream object at 0x7ff62c9b6920>

Is there any output from /opt/saltstack/salt/bin/python3 -c 'import M2Crypto'?

# /opt/saltstack/salt/bin/python3 -c 'import M2Crypto'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/saltstack/salt/extras-3.10/M2Crypto/__init__.py", line 26, in <module>
    from M2Crypto import (ASN1, AuthCookie, BIO, BN, DH, DSA, EVP, Engine, Err,
  File "/opt/saltstack/salt/extras-3.10/M2Crypto/ASN1.py", line 15, in <module>
    from M2Crypto import BIO, m2, six
  File "/opt/saltstack/salt/extras-3.10/M2Crypto/BIO.py", line 11, in <module>
    from M2Crypto import m2, six
  File "/opt/saltstack/salt/extras-3.10/M2Crypto/m2.py", line 30, in <module>
    from M2Crypto.m2crypto import *
  File "/opt/saltstack/salt/extras-3.10/M2Crypto/m2crypto.py", line 13, in <module>
    from ._m2crypto import *
ImportError: /opt/saltstack/salt/extras-3.10/M2Crypto/_m2crypto.cpython-310-x86_64-linux-gnu.so: undefined symbol: ossl_check_OPENSSL_BLOCK_copyfunc_type

[OrangeDog]

This is the same issue as #64121

[danielbakken]

This isn't a showstopper for us, since we've switched to x509_v2. But since m2crypto is not installable with salt-pip, the default x509 module is broken on RHEL 9.

[OrangeDog]

N.B. x509_v2 currently has this problem: #64195

[dwoz]

I was able to reproduce this. Still looking into a root cause.

[dwoz]

I was able to get M2Crypto to work using relenv's toolchain.

/opt/saltstack/salt/bin/relenv toolchain fetch

followed by

RELENV_BUILDENV=yes \
CFLAGS="-I/opt/saltstack/salt/include" \
LDFLAGS="-L/opt/saltstack/salt/lib" \
SWIG_FEATURES="-I/opt/saltstack/salt/include" \
./salt-pip install m2crypto --no-cache -v

[dwoz]

After digging in a bit deeper; because we shouldn't need to compile using relenv's toolchain. I discovered an issue with relenv's runtime setup. We should be able to fix this so a normal pip install will work.

[lmf-mx]

As mentioned, it's a path issue and not system specific. Closed #64485 as a duplicate of this.

[Ch3LL]

Is this fixed on the latest version of Salt? 3006.3

[emalzer]

Hi! I'm running into the same issue with Ubuntu 22.04. and 3006.3. m2crypto is installed with salt-pip and it still fails to execute x509 states.

Any info when a fix will be availalble?

[dwoz]

This should be resolved when we upgrade to relenv 0.16.1.