Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] ldap module broken after upgrade to 3006.2 #64962

Open
gvalkov opened this issue Aug 11, 2023 · 11 comments
Open

[BUG] ldap module broken after upgrade to 3006.2 #64962

gvalkov opened this issue Aug 11, 2023 · 11 comments
Assignees
Labels
Bug broken, incorrect, or confusing behavior Regression The issue is a bug that breaks functionality known to work in previous releases. Sulfur v3006.2

Comments

@gvalkov
Copy link

gvalkov commented Aug 11, 2023

Description
On a clean install of 3006.2 on AlmaLinux 9, the ldap module fails to import with the following error:

$ /opt/saltstack/salt/bin/python3 -c 'import ldap'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/saltstack/salt/extras-3.10/ldap/__init__.py", line 34, in <module>
    import _ldap
ImportError: /lib64/libldap.so.2: undefined symbol: EVP_md2, version OPENSSL_3.0.0

It works with 3006.1, as well as the system Python.

Steps to Reproduce the behavior

$ podman run --rm -it almalinux:9
$ curl -o bootstrap-salt.sh -L https://bootstrap.saltproject.io
$ bash bootstrap-salt.sh onedir
$ rpm -qf `which salt-pip`
salt-3006.2-0.x86_64
$ dnf install gcc openldap-devel
$ rpm -q openldap-devel
openldap-devel-2.6.2-3.el9.x86_64
$ salt-pip install python-ldap
Collecting python-ldap
  Downloading python-ldap-3.4.3.tar.gz (377 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━ 377.4/377.4 kB 8.8 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting pyasn1-modules>=0.1.5
  Downloading pyasn1_modules-0.3.0-py2.py3-none-any.whl (181 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━ 181.3/181.3 kB 22.3 MB/s eta 0:00:00
Collecting pyasn1>=0.3.7
  Downloading pyasn1-0.5.0-py2.py3-none-any.whl (83 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.9/83.9 kB 64.4 MB/s eta 0:00:00
Building wheels for collected packages: python-ldap
  Building wheel for python-ldap (pyproject.toml) ... done
  Created wheel for python-ldap: filename=python_ldap-3.4.3-cp310-cp310-linux_x86_64.whl size=317527 sha256=caec96d436be13a253a9e328e3cafd4257429b96f17a44210e05df38d283f479
  Stored in directory: /root/.cache/pip/wheels/46/da/1e/560bb8f6a776fb7e42af5f32f40360f4bd4674710337e1b087
Successfully built python-ldap
Installing collected packages: pyasn1, pyasn1-modules, python-ldap
  WARNING: In `rpath_only mode` but /lib64/libldap.so.2 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/liblber.so.2 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/libevent-2.1.so.7 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/libsasl2.so.3 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/libcrypt.so.2 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/libcom_err.so.2 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/libkeyutils.so.1 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/libselinux.so.1 is not in /opt/saltstack/salt
  WARNING: In `rpath_only mode` but /lib64/libpcre2-8.so.0 is not in /opt/saltstack/salt
  Do not adjust rpath of /tmp/pip-target-zabty43p/_ldap.cpython-310-x86_64-linux-gnu.so
Successfully installed pyasn1-0.5.0 pyasn1-modules-0.3.0 python-ldap-3.4.3

$ /opt/saltstack/salt/bin/python3 -c 'import ldap'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/saltstack/salt/extras-3.10/ldap/__init__.py", line 34, in <module>
    import _ldap
ImportError: /lib64/libldap.so.2: undefined symbol: EVP_md2, version OPENSSL_3.0.0

Repeating the above with bootstrap-salt.sh stable 3006.1 works as expected. For now we're switching to the ldap3 module in our external pillars.

@gvalkov gvalkov added Bug broken, incorrect, or confusing behavior needs-triage labels Aug 11, 2023
@welcome
Copy link

welcome bot commented Aug 11, 2023

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

@OrangeDog OrangeDog added Regression The issue is a bug that breaks functionality known to work in previous releases. Sulfur v3006.2 labels Aug 14, 2023
@OrangeDog
Copy link
Contributor

Do you have the same versions of libldap and libssl installed on both systems?

Is it python_ldap==3.4.3 that you're trying to install on both?

@OrangeDog OrangeDog added the info-needed waiting for more info label Aug 14, 2023
@Ch3LL Ch3LL modified the milestones: Sulfur v3006.3, Sulfur v3006.2 Aug 14, 2023
@MKLeb MKLeb removed the info-needed waiting for more info label Aug 15, 2023
@MKLeb
Copy link
Contributor

MKLeb commented Aug 16, 2023

Hi @gvalkov, it seems that openldap is trying to use the md2 method of openssl. This is happening on 3006.2 because we switched to openssl 3.1.2, and md2 is seemingly not available normally or with the legacy provider. Oddly enough, older versions of openldap (debian 11, for instance, installs 2.4.2 I believe), don't run into this problem with relenv==0.13.4. I have looked at it, and conferred with @dwoz - He is out until friday but will look at this when he gets time. With that being said, we are in code-freeze for 3006.3 as of tomorrow, so I will be pushing to 3006.4.

@gvalkov
Copy link
Author

gvalkov commented Aug 19, 2023

Hi @MKLeb. Thank you for looking into this.

I already migrated our FreeIPA extpillar to use the pure-python ldap3 module (it's somewhat confusing that salt's ldap3 module is still based on the native bindings). Before that, I had this workaround that patched EVP_md2 out of libldap (with lief):

import lief

lib = lief.parse("/lib64/libldap.so.2")
sym = next(i for i in lib.imported_symbols if i.name == "EVP_md2")
lib.remove_dynamic_symbol(sym)
lib.write("/opt/saltstack/salt/lib//libldap.so.2")

@iMikeG6
Copy link

iMikeG6 commented Dec 8, 2023

As I also had this issue and based on @gvalkovr comment's, here is what I did to workaround this problem.

System is running salt version 3006.4 (Sulfur) on OS Red Hat Enterprise Linux release 9.3 (Plow)

To confirm the error, I ran the following command

/opt/saltstack/salt/bin/python3 -c 'import ldap'

As expected, it throw the following

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/saltstack/salt/extras-3.10/ldap/__init__.py", line 34, in <module>
    import _ldap
ImportError: /lib64/libldap.so.2: undefined symbol: EVP_md2, version OPENSSL_3.0.0

So, I nstalled lief python module

salt-pip install lief

Then edited file in /opt/saltstack/salt/extras-3.10/ldap/__init__.py or /opt/saltstack/salt/lib/python3.10/site-packages/ldap/__init__.py, depend on your installation

I'v added the following on top of the line 34 before import _ldap

import lief

lib = lief.parse("/lib64/libldap.so.2")
sym = next(i for i in lib.imported_symbols if i.name == "EVP_md2")
lib.remove_dynamic_symbol(sym)
lib.write("/opt/saltstack/salt/lib/libldap.so.2")

Restarted services

systemctl restart salt-master.service salt-minion.service salt-api.service

It generated segfault errors in syslog output

segfault at 15d90 ip 0000000000015d90 sp 00007ffdb8d040e8 error 14 in python3.10[400000+308000] likely on CPU

Again, edited file in /opt/saltstack/salt/extras-3.10/ldap/__init__.py or /opt/saltstack/salt/lib/python3.10/site-packages/ldap/__init__.py, depend on your installation

Removed the lines previously added near line 34 before import _ldap

And again, restarted services

systemctl restart salt-master.service salt-minion.service salt-api.service

Re-ran the following

/opt/saltstack/salt/bin/python3 -c 'import ldap'

Then voilà, this time no output nor errors and no more error such as LDAP authentication requires python-ldap module

@dwoz dwoz modified the milestones: Sulfur v3006.5, Sulfur v3006.9 May 1, 2024
@zleite
Copy link

zleite commented May 31, 2024

Also present in 3007.1

@nf-brentsaner
Copy link

nf-brentsaner commented Jun 15, 2024

Confirming 3007.1 is affected.

I need to roll back; LDAP eauth is the only enterprise-level auth option available and it is incredibly broken. My team cannot use the API whatsoever.

Note that the workaround above does not work in 3007.1 in my testing. (See next comment)

@nf-brentsaner
Copy link

nf-brentsaner commented Jun 16, 2024

Okay, I have this working in my lab environment. I've identified two things.

  1. I had a /opt/saltstack/salt/extras-3.10/_ldap.cpython-310-x86_64-linux-gnu.so file. This does not seem to be present in fresh installs, and may have been causing issues. I have removed it.
  2. The lief workaround above DOES work, but it does not need to be run from within Salt python. The below will also not segfault. It (should) stop and start Salt services. Works fine on Alma 9.4 once the lief python module is installed.
#!/usr/bin/env python3

import datetime
import grp
import os
import pathlib
import pwd
import shutil
##
import dbus  # dnf -y install python3-dbus
##
import lief  # https://pypi.org/project/lief/

# https://github.com/saltstack/salt/issues/64962

salt_root = pathlib.Path('/opt/saltstack')
src_lib = pathlib.Path('/lib64/libldap.so.2')
dst_lib = salt_root.joinpath('salt', 'lib', 'libldap.so.2')

uname = 'salt'
gname = 'salt'

lib = lief.parse(str(src_lib))
sym = next(i for i in lib.imported_symbols if i.name == 'EVP_md2')
if sym:
    # Get the Salt services from DBus.
    sysbus = dbus.SystemBus()
    sysd = sysbus.get_object('org.freedesktop.systemd1', '/org/freedesktop/systemd1')
    mgr = dbus.Interface(sysd, 'org.freedesktop.systemd1.Manager')
    svcs = []
    for i in mgr.ListUnits():
        # first element is unit name.
        if not str(i[0]).startswith('salt-'):
            continue
        svc = sysbus.get_object('org.freedesktop.systemd1', object_path = mgr.GetUnit(str(i[0])))
        props = dbus.Interface(svc, dbus_interface = 'org.freedesktop.DBus.Properties')
        state = props.Get('org.freedesktop.systemd1.Unit', 'ActiveState')
        if str(state) == 'active':
            svcs.append(i[0])
    # Get the user/group
    u = pwd.getpwnam(uname)
    g = grp.getgrnam(gname)
    # Modify
    print('Modifications necessary.')
    if svcs:
        # Stop the services first.
        for sn in svcs:
            mgr.StopUnit(sn, 'replace')
    if dst_lib.exists():
        # 3.10 deprecated .utcnow().
        #dst_lib_bak = pathlib.Path(str(dst_lib) + '.bak_{0}'.format(datetime.datetime.now(datetime.UTC).timestamp()))
        dst_lib_bak = pathlib.Path(str(dst_lib) + '.bak_{0}'.format(datetime.datetime.utcnow().timestamp()))
        os.rename(dst_lib, dst_lib_bak)
        print('Destination file {0} exists; backed up to {1}.'.format(dst_lib, dst_lib_bak))
    lib.remove_dynamic_symbol(sym)
    lib.write(str(dst_lib))
    os.chown(dst_lib, u.pw_uid, g.gr_gid)
    os.chmod(dst_lib, src_lib.stat().st_mode)
    # Before we restart services, we also want to remove any python caches.
    for root, dirs, files in os.walk(salt_root):
        for f in files:
            if f.lower().endswith('.pyc'):
                fpath = os.path.join(root, f)
                os.remove(fpath)
                print('Removed file {0}'.format(fpath))
        if '__pycache__' in dirs:
            dpath = os.path.join(root, '__pycache__')
            shutil.rmtree(dpath)
            print('Removed directory {0}'.format(dpath))
    # And then start the units that were started before.
    if svcs:
        for sn in svcs:
            mgr.RestartUnit(sn, 'replace')

print('Done.')

@dwoz dwoz unassigned MKLeb Jun 19, 2024
@dwoz
Copy link
Contributor

dwoz commented Jun 22, 2024

@nf-brentsaner Are you able to test this on a current nightly build?

https://repo.saltproject.io/salt-dev/3006.x/latest/

@Sxderp
Copy link

Sxderp commented Jul 11, 2024

I tried with latest 3007 on RHEL9 and it still fails.

Sxderp added a commit to Sxderp/relenv that referenced this issue Jul 11, 2024
md2 is required for the python-ldap module. Using the python-ldap module
is currently the only supported way to run ldap within Salt.

fixes saltstack/salt#64962
@dwoz dwoz modified the milestones: Sulfur v3006.9, Sulfur v3006.10 Jul 29, 2024
@m0duspwnens
Copy link

This appears to be having an impact on the libvirt module as well. This issue is not present in 3006.1.

Running the script from nf-brentsaner resolves the issue.

Prior to running the script:

[root@localhost ~]# /opt/saltstack/salt/bin/python3.10
Python 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvirt
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/libvirt.py", line 16, in <module>
    import cygvirtmod as libvirtmod  # type: ignore
ModuleNotFoundError: No module named 'cygvirtmod'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/saltstack/salt/lib/python3.10/site-packages/libvirt.py", line 19, in <module>
    raise lib_e
  File "/opt/saltstack/salt/lib/python3.10/site-packages/libvirt.py", line 13, in <module>
    import libvirtmod  # type: ignore
ImportError: /lib64/libldap.so.2: undefined symbol: EVP_md2, version OPENSSL_3.0.0

After running the script

[root@localhost ~]# /opt/saltstack/salt/bin/python3.10
Python 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvirt
>>>
[root@localhost ~]# salt --versions
Salt Version:
          Salt: 3006.9

Python Version:
        Python: 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.17.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: oracle 9.2
        locale: utf-8
       machine: x86_64
       release: 5.15.0-101.103.2.1.el9uek.x86_64
        system: Linux
       version: Oracle Linux Server 9.2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug broken, incorrect, or confusing behavior Regression The issue is a bug that breaks functionality known to work in previous releases. Sulfur v3006.2
Projects
None yet
Development

Successfully merging a pull request may close this issue.