importing rpm fails with python3.11-rpm

[ktdreyer]

What happened? What is the problem?

I cannot make this module work with the "python3.11-rpm" module in EPEL 9.

What did you expect to happen?

Python should import rpm correctly in an isolated virtualenv

Example URL(s)

No response

Steps to reproduce

In a ubi9 container, run the following:

dnf -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
dnf -y install python3.11 python3.11-rpm
python3.11 -m venv venv
. venv/bin/activate
pip install rpm
python -m rpm

Here is the full output:

python -m rpm
Traceback (most recent call last):
  File "/venv/lib64/python3.11/site-packages/rpm/__init__.py", line 102, in <module>
    _shim_module_initializing_
NameError: name '_shim_module_initializing_' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "/venv/lib64/python3.11/site-packages/rpm/__init__.py", line 105, in <module>
    initialize()
  File "/venv/lib64/python3.11/site-packages/rpm/__init__.py", line 94, in initialize
    raise ImportError(
ImportError: Failed to import system RPM module. Make sure RPM Python bindings are installed on your system.

Workaround

• There is an existing workaround that can be used until this issue is fixed.

Participation

• I am willing to submit a pull request for this issue. (Packit team is happy to help!)

[ktdreyer]

I haven't dug into the code for rpm-shim yet, but I'm interested in this project because I've written a library that operates in a similar way. Here is the code for it (system_site_packages.py)

import sys
import sysconfig


def site_packages_dirs():
    """
    Return a list of paths where RPMs can install Python libraries system-wide.

    For example, "dnf" can be in /usr/lib/python3.11/site-packages
    Or "rpm" or "gpg" can be in /usr/lib64/python3.11/site-packages
    """
    # (Inspired from sysconfig.get_paths())
    prefix = sysconfig.get_config_var('prefix')  # "/usr"
    libdir = sysconfig.get_config_var('LIBDIR')  # "/usr/lib64"
    py_version_short = f'{sys.version_info[0]}.{sys.version_info[1]}'  # "3.11"
    # /usr/lib/python3.11/site-packages
    purelib = f'{prefix}/lib/python{py_version_short}/site-packages'
    # /usr/lib64/python3.11/site-packages
    platlib = f'{libdir}/python{py_version_short}/site-packages'
    return (purelib, platlib)


class SystemSitePackages:
    """
    Context manager that temporarily adds the system-wide site-packages
    directories to sys.path, like virtualenv's --system-site-packages CLI
    option. Normally tox and virtualenv isolate us from the system-wide
    packages, but some Python packages (dnf, rpm, gpg) are not available this
    way.

    Examples:

    with SystemSitePackages():
        import rpm

    or:

    with SystemSitePackages():
        import dnf
    """
    def __init__(self):
        self.original_sys_path = sys.path.copy()

    def __enter__(self):
        for directory in site_packages_dirs():
            sys.path.insert(0, directory)

    def __exit__(self, exc_type, exc_value, traceback):
        sys.path = self.original_sys_path

This allows me to selectively import system-site modules like this:

with SystemSitePackage():
    import rpm

[nforro]

Hi, rpm-shim considers only platform-python and the interpreter with the same major version when looking for site-packages directories, the fix for your use case is simple:

--- a/rpm/__init__.py
+++ b/rpm/__init__.py
@@ -41,9 +41,13 @@ def get_system_sitepackages() -> List[str]:
         output = subprocess.check_output(command)
         return json.loads(output.decode())
 
-    majorver, *_ = platform.python_version_tuple()
+    majorver, minorver, _ = platform.python_version_tuple()
     # try platform-python first (it could be the only interpreter present on the system)
-    interpreters = ["/usr/libexec/platform-python", f"/usr/bin/python{majorver}"]
+    interpreters = [
+        "/usr/libexec/platform-python",
+        f"/usr/bin/python{majorver}",
+        f"/usr/bin/python{majorver}.{minorver}",
+    ]
     result = []
     for interpreter in interpreters:
         if not Path(interpreter).is_file():

Your code is very similar, except rpm-shim collects available python interpreters and then uses site.getsitepackages() to get a list of paths rather than constructing them manually, and with each path in sys.path[0] it tries to import the module and does a simple functionality check.