Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyKCS11 in error when installing on Ubuntu 24.04 (Noble) #107

Closed
SixK opened this issue Jan 31, 2024 · 9 comments
Closed

PyKCS11 in error when installing on Ubuntu 24.04 (Noble) #107

SixK opened this issue Jan 31, 2024 · 9 comments
Assignees

Comments

@SixK
Copy link

SixK commented Jan 31, 2024

  • Operating system used: Ubuntu 24.04 (Noble)
  • PyKCS11 version: 1.5.13
  • Python version: 3.11.7
  • PKCS#11 library used:

PyKCS11 seem's to not install on Ubuntu 24.04 (Noble) anymore (seem's it was working about a week ago)

The problem can be reproduced using Ubuntu 24.04 docker image like this:

docker pull ubuntu:24.04
docker run -it ubuntu:24.04 bash
apt update
apt install python3 python3-pip swig
pip install PyKCS11 --break-system-packages

Note that v1.5.0 also finish with the same error:
pip install PyKCS11==1.5.0 --break-system-packages --verbose
But this command work:
pip install PyKCS11==1.4.4 --break-system-packages --verbose

SWIG library version is 4.2.0 on Ubuntu 24.04.
PyKCS11 installation work with Ubuntu 22.04 (Swig 4.0.2) and Debian Bookworm (Swig 4.1.0)

Problem seem's to be related to this variable:
SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t

Corresponding error seem's to be in the following file near line 281:
https://github.com/LudovicRousseau/PyKCS11/blob/master/src/pykcs11.i

Find below full log:

root@be58da524838:/# pip install PyKCS11 --break-system-packages --verbose
Using pip 23.3 from /usr/lib/python3/dist-packages/pip (python 3.11)
Collecting PyKCS11
  Using cached PyKCS11-1.5.13.tar.gz (78 kB)
  Running command pip subprocess to install build dependencies
  Collecting setuptools
    Using cached setuptools-69.0.3-py3-none-any.whl.metadata (6.3 kB)
  Using cached setuptools-69.0.3-py3-none-any.whl (819 kB)
  Installing collected packages: setuptools
  Successfully installed setuptools-69.0.3
  WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  running egg_info
  writing PyKCS11.egg-info/PKG-INFO
  writing dependency_links to PyKCS11.egg-info/dependency_links.txt
  writing top-level names to PyKCS11.egg-info/top_level.txt
  file PyKCS11/LowLevel.py (for module PyKCS11.LowLevel) not found
  reading manifest file 'PyKCS11.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  adding license file 'COPYING'
  writing manifest file 'PyKCS11.egg-info/SOURCES.txt'
  Getting requirements to build wheel ... done
  Running command pip subprocess to install backend dependencies
  Collecting wheel
    Using cached wheel-0.42.0-py3-none-any.whl.metadata (2.2 kB)
  Using cached wheel-0.42.0-py3-none-any.whl (65 kB)
  Installing collected packages: wheel
  Successfully installed wheel-0.42.0
  WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
  Installing backend dependencies ... done
  Running command Preparing metadata (pyproject.toml)
  running dist_info
  creating /tmp/pip-modern-metadata-asi8l97q/PyKCS11.egg-info
  writing /tmp/pip-modern-metadata-asi8l97q/PyKCS11.egg-info/PKG-INFO
  writing dependency_links to /tmp/pip-modern-metadata-asi8l97q/PyKCS11.egg-info/dependency_links.txt
  writing top-level names to /tmp/pip-modern-metadata-asi8l97q/PyKCS11.egg-info/top_level.txt
  writing manifest file '/tmp/pip-modern-metadata-asi8l97q/PyKCS11.egg-info/SOURCES.txt'
  file PyKCS11/LowLevel.py (for module PyKCS11.LowLevel) not found
  reading manifest file '/tmp/pip-modern-metadata-asi8l97q/PyKCS11.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  adding license file 'COPYING'
  writing manifest file '/tmp/pip-modern-metadata-asi8l97q/PyKCS11.egg-info/SOURCES.txt'
  creating '/tmp/pip-modern-metadata-asi8l97q/PyKCS11-1.5.13.dist-info'
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: PyKCS11
  Running command Building wheel for PyKCS11 (pyproject.toml)
  running bdist_wheel
  running build
  running build_py
  running build_ext
  building 'PyKCS11._LowLevel' extension
  swigging src/pykcs11.i to src/pykcs11_wrap.cpp
  swig -python -c++ -o src/pykcs11_wrap.cpp src/pykcs11.i
  creating build
  creating build/temp.linux-x86_64-cpython-311
  creating build/temp.linux-x86_64-cpython-311/src
  x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O -Wall -g -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -Isrc -I/usr/local/incude -I/usr/include/python3.11 -c src/ck_attribute_smart.cpp -o build/temp.linux-x86_64-cpython-311/src/ck_attribute_smart.o
  x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g fwrapv -O2 -Wall -g -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -Isrc -I/usrlocal/include -I/usr/include/python3.11 -c src/dyn_unix.c -o build/temp.linux-x86_64-cpython-311/src/dyn_unix.o
  x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -wrapv -O2 -Wall -g -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -Isrc -I/usr/ocal/include -I/usr/include/python3.11 -c src/pkcs11lib.cpp -o build/temp.linux-x86_64-cpython-311/src/pkcs11lib.o
  x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g-fwrapv -O2 -Wall -g -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -Isrc -I/us/local/include -I/usr/include/python3.11 -c src/pykcs11.cpp -o build/temp.linux-x86_64-cpython-311/src/pykcs11.o
  x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrpv -O2 -Wall -g -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -Isrc -I/usr/locl/include -I/usr/include/python3.11 -c src/pykcs11_wrap.cpp -o build/temp.linux-x86_64-cpython-311/src/pykcs11_wrap.o
  src/pykcs11_wrap.cpp: In function ‘PyObject* _wrap_CK_MECHANISM_pParameter_set(PyObject*, PyObject*)’:
  src/pykcs11_wrap.cpp:15849:64: error: ‘SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t’ was not declared in this scope
  15849 |     if (SWIG_IsOK(SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0)))
        |                                                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp:343:37: note: in definition of macro ‘SWIG_IsOK’
    343 | #define SWIG_IsOK(r)               (r >= 0)
        |                                     ^
  src/pykcs11_wrap.cpp:1130:57: note: in expansion of macro ‘SWIG_Python_ConvertPtr’
   1130 | #define SWIG_ConvertPtr(obj, pptr, type, flags)         SWIG_Python_ConvertPtr(obj, pptr, type, flags)
        |                                                         ^~~~~~~~~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp:15849:19: note: in expansion of macro ‘SWIG_ConvertPtr’
  15849 |     if (SWIG_IsOK(SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0)))
        |                   ^~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp: In function ‘PyObject* _wrap_CK_GCM_PARAMS_pIv_set(PyObject*, PyObject*)’:
  src/pykcs11_wrap.cpp:16027:60: error: ‘SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t’ was not declared in this scope
  16027 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |                                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp:1129:97: note: in definition of macro ‘SWIG_Python_ConvertPtr’
   1129 | #define SWIG_Python_ConvertPtr(obj, pptr, type, flags)  SWIG_Python_ConvertPtrAndOwn(obj, pptr, type, flags, 0)
        |                                                                                                 ^~~~
  src/pykcs11_wrap.cpp:16027:15: note: in expansion of macro ‘SWIG_ConvertPtr’
  16027 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |               ^~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp: In function ‘PyObject* _wrap_CK_GCM_PARAMS_pAAD_set(PyObject*, PyObject*)’:
  src/pykcs11_wrap.cpp:16145:60: error: ‘SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t’ was not declared in this scope
  16145 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |                                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp:1129:97: note: in definition of macro ‘SWIG_Python_ConvertPtr’
   1129 | #define SWIG_Python_ConvertPtr(obj, pptr, type, flags)  SWIG_Python_ConvertPtrAndOwn(obj, pptr, type, flags, 0)
        |                                                                                                 ^~~~
  src/pykcs11_wrap.cpp:16145:15: note: in expansion of macro ‘SWIG_ConvertPtr’
  16145 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |               ^~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp: In function ‘PyObject* _wrap_CK_RSA_PKCS_OAEP_PARAMS_pSourceData_set(PyObject*, PyObject*)’:
  src/pykcs11_wrap.cpp:16527:60: error: ‘SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t’ was not declared in this scope
  16527 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |                                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp:1129:97: note: in definition of macro ‘SWIG_Python_ConvertPtr’
   1129 | #define SWIG_Python_ConvertPtr(obj, pptr, type, flags)  SWIG_Python_ConvertPtrAndOwn(obj, pptr, type, flags, 0)
        |                                                                                                 ^~~~
  src/pykcs11_wrap.cpp:16527:15: note: in expansion of macro ‘SWIG_ConvertPtr’
  16527 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |               ^~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp: In function ‘PyObject* _wrap_CK_ECDH1_DERIVE_PARAMS_pSharedData_set(PyObject*, PyObject*)’:
  src/pykcs11_wrap.cpp:17011:60: error: ‘SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t’ was not declared in this scope
  17011 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |                                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp:1129:97: note: in definition of macro ‘SWIG_Python_ConvertPtr’
   1129 | #define SWIG_Python_ConvertPtr(obj, pptr, type, flags)  SWIG_Python_ConvertPtrAndOwn(obj, pptr, type, flags, 0)
        |                                                                                                 ^~~~
  src/pykcs11_wrap.cpp:17011:15: note: in expansion of macro ‘SWIG_ConvertPtr’
  17011 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |               ^~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp: In function ‘PyObject* _wrap_CK_ECDH1_DERIVE_PARAMS_pPublicData_set(PyObject*, PyObject*)’:
  src/pykcs11_wrap.cpp:17129:60: error: ‘SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t’ was not declared in this scope
  17129 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |                                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  src/pykcs11_wrap.cpp:1129:97: note: in definition of macro ‘SWIG_Python_ConvertPtr’
   1129 | #define SWIG_Python_ConvertPtr(obj, pptr, type, flags)  SWIG_Python_ConvertPtrAndOwn(obj, pptr, type, flags, 0)
        |                                                                                                 ^~~~
  src/pykcs11_wrap.cpp:17129:15: note: in expansion of macro ‘SWIG_ConvertPtr’
  17129 |     int res = SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0);
        |               ^~~~~~~~~~~~~~~
  error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
  error: subprocess-exited-with-error
  
  × Building wheel for PyKCS11 (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /usr/bin/python3 /usr/lib/python3/dist-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpciub54do
  cwd: /tmp/pip-install-q1hwk9le/pykcs11_35eada89245c4008885373382292fd38
  Building wheel for PyKCS11 (pyproject.toml) ... error
  ERROR: Failed building wheel for PyKCS11
Failed to build PyKCS11
ERROR: Could not build wheels for PyKCS11, which is required to install pyproject.toml-based projects
@LudovicRousseau
Copy link
Owner

What happens if you follow the documentation to build PyKCS11?
https://github.com/LudovicRousseau/PyKCS11/blob/master/README.md#unix-build-howto

@LudovicRousseau LudovicRousseau self-assigned this Jan 31, 2024
@SixK
Copy link
Author

SixK commented Feb 1, 2024

Got the same error if building with this commands in a virtual python env (following the unix-build-howto).

If compairing generated pykcs11_wrap.cpp from ubuntu 22.04 and 24.04, I see the following difference:
Ubuntu 22.04 --> src/pykcs11_wrap.cpp:#define SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t swig_types[29]
Ubuntu 24.04 --> src/pykcs11_wrap.cpp:#define SWIGTYPE_p_std__vectorT_unsigned_char_t swig_types[29]

That can explain that SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t definition is not found.

At this state, I can't figure out if this is a swig 4.2.0 (or swig 4.1.1) change or something else.

In swig #defines seem's to be generated here:
https://github.com/swig/swig/blob/f762ab35b29b993e868ea4e0750bdb608e0084cd/Source/Swig/typesys.c#L2263

But this doesn't help much to understand

I replaced SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t by SWIGTYPE_p_std__vectorT_unsigned_char_t in pykcs11.i file.
Build is now working without any error, but I didn't have tested the generated package.
I have no idea if it's the right modification to do for Ubuntu 24.04 or Swig 4.2.0
Maybe it will be more obvious for you ?

@lmiguelmh
Copy link

lmiguelmh commented Feb 1, 2024

We had a similar error using swig==4.2.0 (latest) and pykcs==1.5.12

Workaround: use swig==4.1.1.post1

pip install swig==4.1.1.post1

@SixK
Copy link
Author

SixK commented Feb 1, 2024

If I follow the sequence below on an official ubuntu 24.04 image, I can confirm that your work-around is working to build PyKCS11 for Ubuntu 24.04:

    2  apt update
    3  apt install python3 python3-pip
    4  apt install git
    6  apt install python3-venv
    8  apt install build-essential
    9  python3 -m venv temp
   10  source temp/bin/activate
   11  pip3 install -r dev-requirements.txt
   12  git clone https://github.com/LudovicRousseau/PyKCS11
   13  cd PyKCS11/
   14  pip3 install -r dev-requirements.txt
   16  pip3 install swig==4.1.1.post1
   17  make

But If I simply want to install PyKCS11 to the system, I can't find a simple way :
pip3 install swig==4.1.1.post1 --break-system-packages
pip3 install PyKCS11 --break-system-packages


Collecting PyKCS11
  Using cached PyKCS11-1.5.13.tar.gz (78 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: PyKCS11
  Building wheel for PyKCS11 (pyproject.toml) ... error
  error: subprocess-exited-with-error
  × Building wheel for PyKCS11 (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [12 lines of output]
      running bdist_wheel
      running build
      running build_py
      running build_ext
      building 'PyKCS11._LowLevel' extension
      swigging src/pykcs11.i to src/pykcs11_wrap.cpp
      swig -python -c++ -o src/pykcs11_wrap.cpp src/pykcs11.i
      Traceback (most recent call last):
        File "/usr/local/bin/swig", line 5, in <module>
          from swig import swig
      ModuleNotFoundError: No module named 'swig'
      error: command '/usr/local/bin/swig' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for PyKCS11
Failed to build PyKCS11
ERROR: Could not build wheels for PyKCS11, which is required to install pyproject.toml-based projects

It's weird that swig module is not found, in this case.
If I install swig using apt, I will get v4.2.0 and will have the undefined variable error.

LudovicRousseau added a commit to LudovicRousseau/PyKCS11-debug that referenced this issue Feb 2, 2024
The error was:
src/pykcs11_wrap.cpp:15849:64: error: ‘SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t’ was not declared in this scope
15849 |     if (SWIG_IsOK(SWIG_ConvertPtr(swig_obj[1], (void **)&vect, SWIGTYPE_p_std__vectorT_unsigned_char_std__allocatorT_unsigned_char_t_t, 0)))
      |                                                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Thanks to SixK for the bug report
" PyKCS11 in error when installing on Ubuntu 24.04 (Noble) #107 "
Closes: LudovicRousseau/PyKCS11#107
@LudovicRousseau
Copy link
Owner

Thanks for the bug report.
I will make a new release of PyKCS11 "soon".

@LudovicRousseau
Copy link
Owner

@SixK
Copy link
Author

SixK commented Feb 5, 2024

Thank's for new release,
according to wsfulton on swig project, the right way to solve the error would be to use descriptor:
$descriptor(std::vector

corresponding swig issue:
swig/swig#2793 (comment)

@LudovicRousseau
Copy link
Owner

I tried with $descriptor(std::vector<unsigned char>) but I then have regressions in my tests.
It may be a problem in my code.
I will have a look. Thanks for the info @SixK.

@LudovicRousseau
Copy link
Owner

Fixed in 808199f

The correct fix is to use $descriptor(vector<unsigned char> *)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants