How to pre-test RasPiOS 12 [cffi, cryptography, lxml "wheels" for Python 3.11 now exist!] #3526

jvonau · 2023-04-04T21:44:13Z

Install the latest arm64 image of RasPiOS and boot

sudo apt update
sudo apt -y upgrade
sudo apt -y autoremove
sudo reboot

sudo sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list

sudo apt update
sudo apt -y dist-upgrade
# Don't go too far away there a couple of confirmations to confirm just press enter when prompted.
# Any changed files should be a result of the image creation process I would presume. 
# I hate suggesting that method but you can stop and read the actions and decide for yourself.

sudo apt -y autoremove
sudo reboot

sudo raspi-config and set the country code for the WiFi and other settings.
Might want to try the NetworkManager option at this time.
Edit: Make the required changes in local_facts.fact and provide a OS.yml file is covered with #3525 not needed now.
Install IIAB as per normal.

The text was updated successfully, but these errors were encountered:

jvonau · 2023-04-04T21:50:16Z

Just a how to note, not for general consumption but I thought I would share.

holta · 2023-04-04T22:21:02Z

Install IIAB as per normal.

Did an IIAB install largely complete?

(Either way, instructions like this are indeed extremely useful for newcomers.)

IIAB installs on Debian 12 Bookworm pre-releases IF (1) KA Lite bypassed (2) PHP 8.2 requires Nextcloud 26? (3) Munin bypassed? [Ubuntu 23.04 issues quite similar!] #3399

jvonau · 2023-04-04T23:02:30Z

I went with the 'none' version of local_vars as a starting point, installed clean.

jvonau · 2023-04-05T15:07:55Z

FYI

jerryv@box:~ $ apt list python2*
Listing... Done

jvonau · 2023-04-05T16:32:58Z

#3289 (comment)

perhaps using the last deb package(s) released from an older install.

http://ftp.debian.org/debian/pool/main/p/python2.7/?C=M;O=D

wget http://ftp.debian.org/debian/pool/main/p/python2.7/libpython2.7-minimal_2.7.18-8_arm64.deb
sudo apt install ./libpython2.7-minimal_2.7.18-8_arm64.deb
wget http://ftp.debian.org/debian/pool/main/p/python2.7/python2.7-minimal_2.7.18-8_arm64.deb
sudo apt install ./python2.7-minimal_2.7.18-8_arm64.deb

The order is important and are 'dpkg_arch' dependent, that was the reasoning behind my original #3518 having dpkg_arch available for other roles.

holta · 2023-04-11T16:56:01Z

@jvonau mentions at #3535 (comment) :

http://raspbian.raspberrypi.org/ has a new file snapshotindex.txt suggesting there will be a new bookworm release sooner or later when you look at the bottom of the file

jvonau · 2023-04-11T17:22:43Z

I don't have time to try this but should work:
Install the latest armhf image of RasPiOS and boot

sudo apt update
sudo apt -y upgrade
sudo apt -y autoremove

apt info libpython2.7-stdlib > ~/libpython.txt 
# attach or pastebin the file to PR #3535 please.

sudo reboot 

sudo sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list

sudo apt update
sudo apt -y dist-upgrade
# Don't go too far away there a couple of confirmations to confirm just press enter when prompted.
# Any changed files should be a result of the image creation process I would presume. 
# I hate suggesting that method but you can stop and read the actions and decide for yourself.

sudo apt -y autoremove
sudo reboot

Might not all be in place yet but post how far you get to.

holta · 2023-04-11T17:27:40Z

@EMG70 would you have time in the coming week, to try the instructions immediately above?

EMG70 · 2023-04-11T17:46:58Z

That’s OK, I will try it this week and let you know.

jvonau · 2023-04-11T17:48:22Z

Stepping through an upgrade of an older install

jerryv@box:~ $ sudo apt update
Hit:1 http://ppa.launchpad.net/learningequality/kolibri/ubuntu focal InRelease
Hit:2 https://deb.nodesource.com/node_18.x bullseye InRelease                                          
Hit:3 http://archive.raspberrypi.org/debian bullseye InRelease                                         
Get:4 http://raspbian.raspberrypi.org/raspbian bookworm InRelease [15.0 kB]
Get:5 http://raspbian.raspberrypi.org/raspbian bookworm/main armhf Packages [14.5 MB]
Get:6 http://raspbian.raspberrypi.org/raspbian bookworm/contrib armhf Packages [62.3 kB]
Get:7 http://raspbian.raspberrypi.org/raspbian bookworm/non-free armhf Packages [118 kB]
Get:8 http://raspbian.raspberrypi.org/raspbian bookworm/rpi armhf Packages [1,360 B]
Fetched 14.7 MB in 9s (1,689 kB/s)                                                                                                      
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
795 packages can be upgraded. Run 'apt list --upgradable' to see them.

That was with iiab already install should be a smaller set on a newly written image

jvonau · 2023-04-11T17:53:42Z

sudo apt -y dist-upgrade
The following packages will be REMOVED:
  fuse hardlink libnfsidmap2 libnginx-mod-http-lua libsemanage1 libsepol1-dev mariadb-client-10.5 mariadb-client-core-10.5
  mariadb-server-10.5 mariadb-server-core-10.5 nginx-extras php-pear php7.4-bcmath php7.4-cgi php7.4-cli php7.4-common php7.4-curl
  php7.4-fpm php7.4-gd php7.4-imap php7.4-json php7.4-mbstring php7.4-mysql php7.4-opcache php7.4-readline php7.4-snmp php7.4-xml
  php7.4-zip python-pip-whl python3-kms++ python3-libcamera python3-picamera2 python3-pidng python3-simplejpeg python3-spidev
  python3.9-venv
Get:917
Fetched 483 MB in 6min 25s (1,254 kB/s)

most of these will not be installed on a fresh image.

jvonau · 2023-04-11T18:19:13Z

This test box never had kalite installed so no python2 was ever installed http://sprunge.us/M2I5sQ

jerryv@box:~ $ apt list *python2*
Listing... Done
idle-python2.7/testing 2.7.18-13.2 all
libpython2-dbg/testing 2.7.18-3 armhf
libpython2-dev/testing 2.7.18-3 armhf
libpython2-stdlib/testing 2.7.18-3 armhf
libpython2.7-dbg/testing 2.7.18-13.2 armhf
libpython2.7-dev/testing 2.7.18-13.2 armhf
libpython2.7-minimal/testing 2.7.18-13.2 armhf
libpython2.7-stdlib/testing 2.7.18-13.2 armhf
libpython2.7-testsuite/testing 2.7.18-13.2 all
libpython2.7/testing 2.7.18-13.2 armhf
python2-dbg/testing 2.7.18-3 armhf
python2-dev/testing 2.7.18-3 armhf
python2-doc/testing 2.7.18-3 all
python2-minimal/testing 2.7.18-3 armhf
python2-setuptools-whl/testing 44.1.1-1.2 all
python2.7-dbg/testing 2.7.18-13.2 armhf
python2.7-dev/testing 2.7.18-13.2 armhf
python2.7-doc/testing 2.7.18-13.2 all
python2.7-examples/testing 2.7.18-13.2 all
python2.7-minimal/testing 2.7.18-13.2 armhf
python2.7/testing 2.7.18-13.2 armhf
python2/testing 2.7.18-3 armhf

Looks like they are going to ship python2

jvonau · 2023-04-11T18:28:53Z

Bingo libssl3 is being used

jerryv@box:~ $ apt info libpython2.7-stdlib
Package: libpython2.7-stdlib
Version: 2.7.18-13.2
Priority: optional
Section: python
Source: python2.7
Maintainer: Matthias Klose <doko@debian.org>
Installed-Size: 10.0 MB
Provides: python-argparse, python2.7-argparse, python2.7-celementtree, python2.7-cjkcodecs, python2.7-ctypes, python2.7-elementtree, python2.7-profiler, python2.7-wsgiref
Depends: libpython2.7-minimal (= 2.7.18-13.2), libssl3 (>= 3.0.0), media-types | mime-support, libbz2-1.0, libc6 (>= 2.33), libcrypt1 (>= 1:4.1.0), libdb5.3, libexpat1 (>= 2.1~beta3), libffi8 (>= 3.4), libncursesw6 (>= 6), libnsl2 (>= 1.0), libreadline8 (>= 7.0~beta), libsqlite3-0 (>= 3.5.9), libtinfo6 (>= 6), libtirpc3 (>= 1.0.2)
Breaks: libgcj-common (<< 1:7), libpython-dbg (<< 2.7.18), libpython-dev (<< 2.7.18), libpython-stdlib (<< 2.7.18), libvolk1-bin (<< 2), pysycache (<< 3.1-4~), python (<< 2.7.18), python-catkin-pkg (<< 0.4.14-2~), python-chardet (<< 3.0.4-6~), python-dbg (<< 2.7.18), python-dev (<< 2.7.18), python-extras (<< 1.0.0-4~), python-fixtures (<< 3.0.0-3~), python-genpy (<< 0.6.9-2~), python-iso8601 (<< 0.1.12-2~), python-linecache2 (<< 1.0.0-4~), python-minimal (<< 2.7.18), python-pbr (<< 5.4.5), python-pyasn1 (<< 0.4.2-4~), python-pygame (<< 1.9.6+dfsg-3~), python-pyicu (<< 2.2-3~), python-rospkg (<< 1.1.10-2~), python-std-msgs (<< 0.5.12-2~), python-testtools (<< 2.3.0-7~), python-traceback2 (<< 1.4.0-6~), python-traitlets (<< 4.3.3-3~), python-urllib3 (<< 1.25.8-2~), python-yaml (<< 5.3.1-2~)
Replaces: libpython2.7-testsuite (<< 2.7.13-4), python2.7 (<< 2.7.5-0~)
Download-Size: 1,797 kB
APT-Manual-Installed: no
APT-Sources: http://raspbian.raspberrypi.org/raspbian bookworm/main armhf Packages
Description: Interactive high-level object-oriented language (standard library, version 2.7)
 Python is a high-level, interactive, object-oriented language. Its 2.7 version
 includes an extensive class library with lots of goodies for
 network programming, system administration, sounds and graphics.
 .
 This package contains Python 2.7's standard library. It is normally not
 used on its own, but as a dependency of python2.7.

jvonau · 2023-04-11T18:29:04Z

And it installs too

jerryv@box:~ $ sudo apt install python2
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  libpython2-stdlib libpython2.7-minimal libpython2.7-stdlib python2-minimal python2.7 python2.7-minimal
Suggested packages:
  python2-doc python-tk python2.7-doc binfmt-support
The following NEW packages will be installed:
  libpython2-stdlib libpython2.7-minimal libpython2.7-stdlib python2 python2-minimal python2.7 python2.7-minimal
0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 3,631 kB of archives.
After this operation, 16.6 MB of additional disk space will be used.
Get:1 https://mirrors.switch.ca/raspbian/raspbian bookworm/main armhf libpython2.7-minimal armhf 2.7.18-13.2 [397 kB]
Get:2 https://mirrors.switch.ca/raspbian/raspbian bookworm/main armhf python2.7-minimal armhf 2.7.18-13.2 [1,046 kB]
Get:3 https://mirrors.switch.ca/raspbian/raspbian bookworm/main armhf python2-minimal armhf 2.7.18-3 [34.2 kB]
Get:4 https://mirrors.switch.ca/raspbian/raspbian bookworm/main armhf libpython2.7-stdlib armhf 2.7.18-13.2 [1,797 kB]
Get:5 https://mirrors.switch.ca/raspbian/raspbian bookworm/main armhf python2.7 armhf 2.7.18-13.2 [311 kB]
Get:6 https://mirrors.switch.ca/raspbian/raspbian bookworm/main armhf libpython2-stdlib armhf 2.7.18-3 [21.2 kB]
Get:7 https://mirrors.switch.ca/raspbian/raspbian bookworm/main armhf python2 armhf 2.7.18-3 [24.5 kB]
Fetched 3,631 kB in 3s (1,109 kB/s) 
Selecting previously unselected package libpython2.7-minimal:armhf.
(Reading database ... 82616 files and directories currently installed.)
Preparing to unpack .../0-libpython2.7-minimal_2.7.18-13.2_armhf.deb ...
Unpacking libpython2.7-minimal:armhf (2.7.18-13.2) ...
Selecting previously unselected package python2.7-minimal.
Preparing to unpack .../1-python2.7-minimal_2.7.18-13.2_armhf.deb ...
Unpacking python2.7-minimal (2.7.18-13.2) ...
Selecting previously unselected package python2-minimal.
Preparing to unpack .../2-python2-minimal_2.7.18-3_armhf.deb ...
Unpacking python2-minimal (2.7.18-3) ...
Selecting previously unselected package libpython2.7-stdlib:armhf.
Preparing to unpack .../3-libpython2.7-stdlib_2.7.18-13.2_armhf.deb ...
Unpacking libpython2.7-stdlib:armhf (2.7.18-13.2) ...
Selecting previously unselected package python2.7.
Preparing to unpack .../4-python2.7_2.7.18-13.2_armhf.deb ...
Unpacking python2.7 (2.7.18-13.2) ...
Selecting previously unselected package libpython2-stdlib:armhf.
Preparing to unpack .../5-libpython2-stdlib_2.7.18-3_armhf.deb ...
Unpacking libpython2-stdlib:armhf (2.7.18-3) ...
Setting up libpython2.7-minimal:armhf (2.7.18-13.2) ...
Setting up python2.7-minimal (2.7.18-13.2) ...
Linking and byte-compiling packages for runtime python2.7...
Setting up python2-minimal (2.7.18-3) ...
Selecting previously unselected package python2.
(Reading database ... 83363 files and directories currently installed.)
Preparing to unpack .../python2_2.7.18-3_armhf.deb ...
Unpacking python2 (2.7.18-3) ...
Setting up libpython2.7-stdlib:armhf (2.7.18-13.2) ...
Setting up python2.7 (2.7.18-13.2) ...
Setting up libpython2-stdlib:armhf (2.7.18-3) ...
Setting up python2 (2.7.18-3) ...
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for mailcap (3.70+nmu1) ...

jvonau · 2023-04-11T18:53:55Z

installing ansible might be a pain

Create virtual environment for Ansible
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting ansible-core
  Using cached https://www.piwheels.org/simple/ansible-core/ansible_core-2.14.4-py3-none-any.whl (2.2 MB)
Collecting jinja2>=3.0.0
  Using cached https://www.piwheels.org/simple/jinja2/Jinja2-3.1.2-py3-none-any.whl (133 kB)
Collecting PyYAML>=5.1
  Downloading PyYAML-6.0.tar.gz (124 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.0/125.0 kB 1.2 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting cryptography
  Downloading cryptography-40.0.1.tar.gz (624 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 625.0/625.0 kB 3.1 MB/s eta 0:00:00
  Installing build dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [77 lines of output]
      Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://www.piwheels.org/simple
      Collecting setuptools!=60.9.0,>=40.6.0
        Using cached https://www.piwheels.org/simple/setuptools/setuptools-67.6.1-py3-none-any.whl (1.1 MB)
      Collecting wheel
        Using cached https://www.piwheels.org/simple/wheel/wheel-0.40.0-py3-none-any.whl (64 kB)
      Collecting cffi>=1.12
        Downloading cffi-1.15.1.tar.gz (508 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 508.5/508.5 kB 3.1 MB/s eta 0:00:00
        Preparing metadata (setup.py): started
        Preparing metadata (setup.py): finished with status 'done'
      Collecting setuptools-rust>=0.11.4
        Downloading https://www.piwheels.org/simple/setuptools-rust/setuptools_rust-1.5.2-py3-none-any.whl (23 kB)
      Collecting pycparser
        Using cached https://www.piwheels.org/simple/pycparser/pycparser-2.21-py2.py3-none-any.whl (119 kB)
      Collecting semantic-version<3,>=2.8.2
        Downloading https://www.piwheels.org/simple/semantic-version/semantic_version-2.10.0-py2.py3-none-any.whl (19 kB)
      Collecting typing-extensions>=3.7.4.3
        Downloading https://www.piwheels.org/simple/typing-extensions/typing_extensions-4.5.0-py3-none-any.whl (27 kB)
      Installing collected packages: wheel, typing-extensions, setuptools, semantic-version, pycparser, setuptools-rust, cffi
        DEPRECATION: cffi is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
        Running setup.py install for cffi: started
        Running setup.py install for cffi: finished with status 'error'
        error: subprocess-exited-with-error
      
        × Running setup.py install for cffi did not run successfully.
        │ exit code: 1
        ╰─> [40 lines of output]
            /usr/local/ansible/lib/python3.11/site-packages/setuptools/config/setupcfg.py:515: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
              warnings.warn(msg, warning_class)
            running install
            /usr/local/ansible/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
              warnings.warn(
            running build
            running build_py
            creating build
            creating build/lib.linux-aarch64-cpython-311
            creating build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/lock.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/cparser.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/verifier.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/vengine_gen.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/backend_ctypes.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/vengine_cpy.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/setuptools_ext.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/error.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/cffi_opcode.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/__init__.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/model.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/pkgconfig.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/recompiler.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/api.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/ffiplatform.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/commontypes.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/_cffi_include.h -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/parse_c_type.h -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/_embedding.h -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/_cffi_errors.h -> build/lib.linux-aarch64-cpython-311/cffi
            running build_ext
            building '_cffi_backend' extension
            creating build/temp.linux-aarch64-cpython-311
            creating build/temp.linux-aarch64-cpython-311/c
            arm-linux-gnueabihf-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DFFI_BUILDING=1 -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/local/ansible/include -I/usr/include/python3.11 -c c/_cffi_backend.c -o build/temp.linux-aarch64-cpython-311/c/_cffi_backend.o
            c/_cffi_backend.c:2:10: fatal error: Python.h: No such file or directory
                2 | #include <Python.h>
                  |          ^~~~~~~~~~
            compilation terminated.
            error: command '/usr/bin/arm-linux-gnueabihf-gcc' failed with exit code 1
            [end of output]
      
        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: legacy-install-failure
      
      × Encountered error while trying to install package.
      ╰─> cffi
      
      note: This is an issue with the package mentioned above, not pip.
      hint: See above for output from the failure.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Note the -aarch64- in the above suggests that python's pip is using uname while the userspace is armhf

jerryv@box:/opt/iiab/iiab $ dpkg --print-architecture 
armhf

Lets see where is goes in a few days and see if a python3.11 (cp311) wheel is created for cffi on piwheels

holta · 2023-04-11T20:22:27Z

@jvonau should @EMG70 wait a few days before testing 32-bit RasPiOS ?

jvonau · 2023-04-11T20:58:32Z

Don't try until there is a new wheel with cp311 within the name in the above url.

EMG70 · 2023-04-12T10:00:08Z

Don't try until there is a new wheel with cp311 within the name in the above url.

Noted Jerry,Thanks

holta · 2023-07-30T20:14:39Z

holta · 2023-09-07T19:12:40Z

Don't try until there is a new wheel with cp311 within the name in the above url.

Done, 5 months later! 💯

Where CFFI = C Foreign Function Interface

https://www.piwheels.org/simple/cffi/cffi-1.15.1-cp311-cp311-linux_armv7l.whl#sha256=b7d4d03d3abebb8b6623c14c20e2f9defec95aabde9f5e8999af483eeab7ae18

https://www.piwheels.org/simple/cffi/cffi-1.15.1-cp311-cp311-linux_armv6l.whl#sha256=b7d4d03d3abebb8b6623c14c20e2f9defec95aabde9f5e8999af483eeab7ae18

@EMG70 let's do a LARGE-sized install onto the latest pre-release of "Raspberry Pi OS 12" (desktop version ideally? or Lite version?) if you have time in the coming 10 days!

holta · 2023-09-07T23:14:55Z

sudo sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list

@EMG70 in addition to the above, at that time I recommend you also run:

sed -i 's/bullseye/bookworm/g' /etc/apt/sources.list.d/raspi.list
After your OS upgrade from RasPiOS 11 Lite to RasPiOS 12 Lite is completed, try to install SMALL-sized IIAB with PR scripts/ansible: Evaluate upstream piwheels / cryptography fix on "32-bit" RasPiOS 12 Lite [pinning to cryptography 40.0.1 no longer nec!] [create /etc/pip.conf missing from RasPiOS as nec, to avoid cffi glitch] #3632 as follows:

curl iiab.io/install.txt | bash -s 3632
If your IIAB install on "32-bit" RasPiOS 12 Lite works, please also test your microSD card in a truly 32-bit Raspberry Pi Zero W if you have one (to check the Wi-Fi hotspot works well!)

Just FYI if you run apt update immediately after the OS upgrade to RasPiOS 12 (Bookworm), you'll likely see something like:

N: Repository 'Debian bookworm' changed its 'non-free component' value from 'non-free' to 'non-free non-free-firmware'
N: More information about this can be found online in the Release notes at: https://www.debian.org/releases/bookworm/arm64/release-notes/ch-information.html#non-free-split

holta · 2023-09-08T12:01:10Z

CFFI issues persist for @EMG70 on today's new test on "32-bit" (sort of!) RasPiOS 12 Lite:

(He should post iiab-diagnostics shortly to give a bit more context!)

holta · 2023-09-08T15:03:51Z

Same issue seems to occur with cryptography 40.0.1 too :/

And @EMG70 also tried booting "32-bit" RasPiOS 12 Lite (pre-release) on his RPi 400 in true 32-bit mode, by doing:

echo 'arm_64bit=0' >> /boot/config.txt
reboot

In short, he tried both kinds of boots (32-bit armv7l kernel and 64-bit aarch64 kernel, per #3516) each running on RPi 400, and no luck in either case, even when avoiding PR #3632. CLARIF: each time, he saved the "32-bit" RasPiOS 12 Lite pre-release test OS, using techniques like the following:

mv /etc/iiab /etc/iiab.old
mv /opt/iiab/iiab /opt/iiab/iiab.old
mv /usr/local/ansible /usr/local/ansible.old
sudo iiab

A bit more detail in this screenshot below, showing what appears almost immediately as/after scripts/ansible tries to install cryptography 40.0.1 to the new /usr/local/ansible venv:

holta · 2023-09-09T13:59:04Z

FYI @EMG70 and I intend to give "32-bit" RasPiOS 12 (Bookworm) another crack on Friday September 22:

Whether or not cffi wheel for Python 3.11 shows improvement here... ?

https://www.piwheels.org/project/cffi/
- "1.15.1 from 2022-06-30"
https://www.piwheels.org/simple/cffi/
- cffi-1.15.1-cp311-cp311-linux_armv7l.whl
- cffi-1.15.1-cp311-cp311-linux_armv6l.whl

holta · 2023-09-09T16:19:56Z

Full paste of the latest cffi error on a fresh "32-bit" RasPiOS 12 Lite below — running with a 64-bit kernel in this case — in the hope this helps us & all move the ball forward:

OS/Architecture/Prep Details

root@raspberrypi:~# cat /etc/issue
Raspbian GNU/Linux 12 \n \l

root@raspberrypi:~# cat /etc/rpi-issue
Raspberry Pi reference 2023-05-03
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 47eee1f0ddcf8811559d51eea1c1bb48335e3e88, stage2

root@raspberrypi:~# uname -a
Linux raspberrypi 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64 GNU/Linux

root@raspberrypi:~# tail -1 /proc/cpuinfo
Model           : Raspberry Pi 4 Model B Rev 1.1

root@raspberrypi:~# apt install python3-venv
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  python3-distutils python3-lib2to3 python3-pip-whl python3-setuptools-whl python3.11-venv
The following NEW packages will be installed:
  python3-distutils python3-lib2to3 python3-pip-whl python3-setuptools-whl python3-venv python3.11-venv
0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.
Need to get 3,043 kB of archives.
After this operation, 4,197 kB of additional disk space will be used.
Do you want to continue? [Y/n]
Get:1 http://raspbian.mirror.constant.com/raspbian bookworm/main armhf python3-lib2to3 all 3.11.2-3 [76.3 kB]
Get:2 http://raspbian.mirror.constant.com/raspbian bookworm/main armhf python3-distutils all 3.11.2-3 [131 kB]
Get:3 http://raspbian.raspberrypi.org/raspbian bookworm/main armhf python3-pip-whl all 23.0.1+dfsg-1 [1,717 kB]
Get:4 http://raspbian.mirror.constant.com/raspbian bookworm/main armhf python3-setuptools-whl all 66.1.1-1 [1,111 kB]
Get:6 http://raspbian.raspberrypi.org/raspbian bookworm/main armhf python3-venv armhf 3.11.2-1 [1,188 B]
Get:5 http://raspbian.mirror.constant.com/raspbian bookworm/main armhf python3.11-venv armhf 3.11.2-6 [5,892 B]
Fetched 3,043 kB in 2s (1,409 kB/s)
Selecting previously unselected package python3-lib2to3.
(Reading database ... 44873 files and directories currently installed.)
Preparing to unpack .../0-python3-lib2to3_3.11.2-3_all.deb ...
Unpacking python3-lib2to3 (3.11.2-3) ...
Selecting previously unselected package python3-distutils.
Preparing to unpack .../1-python3-distutils_3.11.2-3_all.deb ...
Unpacking python3-distutils (3.11.2-3) ...
Selecting previously unselected package python3-pip-whl.
Preparing to unpack .../2-python3-pip-whl_23.0.1+dfsg-1_all.deb ...
Unpacking python3-pip-whl (23.0.1+dfsg-1) ...
Selecting previously unselected package python3-setuptools-whl.
Preparing to unpack .../3-python3-setuptools-whl_66.1.1-1_all.deb ...
Unpacking python3-setuptools-whl (66.1.1-1) ...
Selecting previously unselected package python3.11-venv.
Preparing to unpack .../4-python3.11-venv_3.11.2-6_armhf.deb ...
Unpacking python3.11-venv (3.11.2-6) ...
Selecting previously unselected package python3-venv.
Preparing to unpack .../5-python3-venv_3.11.2-1_armhf.deb ...
Unpacking python3-venv (3.11.2-1) ...
Setting up python3-setuptools-whl (66.1.1-1) ...
Setting up python3-pip-whl (23.0.1+dfsg-1) ...
Setting up python3-lib2to3 (3.11.2-3) ...
Setting up python3-distutils (3.11.2-3) ...
Setting up python3.11-venv (3.11.2-6) ...
Setting up python3-venv (3.11.2-1) ...

root@raspberrypi:~# python3 -m venv /usr/local/ansible

root@raspberrypi:~# /usr/local/ansible/bin/python3 -m pip install --upgrade ansible-core
Collecting ansible-core
  Downloading ansible_core-2.15.3-py3-none-any.whl (2.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 5.8 MB/s eta 0:00:00
Collecting jinja2>=3.0.0
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 kB 2.6 MB/s eta 0:00:00
Collecting PyYAML>=5.1
  Downloading PyYAML-6.0.1.tar.gz (125 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.2/125.2 kB 3.7 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting cryptography
  Downloading cryptography-41.0.3.tar.gz (630 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 630.1/630.1 kB 6.7 MB/s eta 0:00:00
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [97 lines of output]
      Collecting setuptools>=61.0.0
        Using cached setuptools-68.2.0-py3-none-any.whl (807 kB)
      Collecting wheel
        Using cached wheel-0.41.2-py3-none-any.whl (64 kB)
      Collecting cffi>=1.12
        Downloading cffi-1.15.1.tar.gz (508 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 508.5/508.5 kB 4.2 MB/s eta 0:00:00
        Preparing metadata (setup.py): started
        Preparing metadata (setup.py): finished with status 'done'
      Collecting setuptools-rust>=0.11.4
        Downloading setuptools_rust-1.7.0-py3-none-any.whl (25 kB)
      Collecting pycparser
        Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 1.8 MB/s eta 0:00:00
      Collecting semantic-version<3,>=2.8.2
        Downloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
      Collecting typing-extensions>=3.7.4.3
        Downloading typing_extensions-4.7.1-py3-none-any.whl (33 kB)
      Installing collected packages: wheel, typing-extensions, setuptools, semantic-version, pycparser, setuptools-rust, cffi
        DEPRECATION: cffi is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
        Running setup.py install for cffi: started
        Running setup.py install for cffi: finished with status 'error'
        error: subprocess-exited-with-error

        × Running setup.py install for cffi did not run successfully.
        │ exit code: 1
        ╰─> [60 lines of output]
            Package libffi was not found in the pkg-config search path.
            Perhaps you should add the directory containing `libffi.pc'
            to the PKG_CONFIG_PATH environment variable
            Package 'libffi', required by 'virtual:world', not found
            Package libffi was not found in the pkg-config search path.
            Perhaps you should add the directory containing `libffi.pc'
            to the PKG_CONFIG_PATH environment variable
            Package 'libffi', required by 'virtual:world', not found
            Package libffi was not found in the pkg-config search path.
            Perhaps you should add the directory containing `libffi.pc'
            to the PKG_CONFIG_PATH environment variable
            Package 'libffi', required by 'virtual:world', not found
            Package libffi was not found in the pkg-config search path.
            Perhaps you should add the directory containing `libffi.pc'
            to the PKG_CONFIG_PATH environment variable
            Package 'libffi', required by 'virtual:world', not found
            Package libffi was not found in the pkg-config search path.
            Perhaps you should add the directory containing `libffi.pc'
            to the PKG_CONFIG_PATH environment variable
            Package 'libffi', required by 'virtual:world', not found
            /usr/local/ansible/lib/python3.11/site-packages/setuptools/config/setupcfg.py:515: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
              warnings.warn(msg, warning_class)
            running install
            /usr/local/ansible/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
              warnings.warn(
            running build
            running build_py
            creating build
            creating build/lib.linux-aarch64-cpython-311
            creating build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/error.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/__init__.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/commontypes.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/lock.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/pkgconfig.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/setuptools_ext.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/ffiplatform.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/backend_ctypes.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/vengine_gen.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/model.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/api.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/verifier.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/cffi_opcode.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/recompiler.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/vengine_cpy.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/cparser.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/_cffi_include.h -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/parse_c_type.h -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/_embedding.h -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/_cffi_errors.h -> build/lib.linux-aarch64-cpython-311/cffi
            running build_ext
            building '_cffi_backend' extension
            creating build/temp.linux-aarch64-cpython-311
            creating build/temp.linux-aarch64-cpython-311/c
            arm-linux-gnueabihf-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DFFI_BUILDING=1 -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/ansible/include -I/usr/include/python3.11 -c c/_cffi_backend.c -o build/temp.linux-aarch64-cpython-311/c/_cffi_backend.o
            c/_cffi_backend.c:2:10: fatal error: Python.h: No such file or directory
                2 | #include <Python.h>
                  |          ^~~~~~~~~~
            compilation terminated.
            error: command '/usr/bin/arm-linux-gnueabihf-gcc' failed with exit code 1
            [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: legacy-install-failure

      × Encountered error while trying to install package.
      ╰─> cffi

      note: This is an issue with the package mentioned above, not pip.
      hint: See above for output from the failure.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

holta · 2023-09-10T19:36:33Z

@EMG70 I've posted upstream to make sure we stay on track with RasPiOS12 — as it evolves towards final release / general availability — presumably very quickly now in coming weeks:

Package issue: cryptography 41.0.3 leads to cffi 1.15.1 failure on "32-bit" Raspberry Pi OS [REASON: /etc/pip.conf missing on some Bookworm pre-releases] piwheels/packages#390

holta · 2023-09-11T03:02:37Z

@EMG70: Great news, the piwheels folk helped clear up the main problem, so this PR clears up several nagging issues: ✅
- PR scripts/ansible: Evaluate upstream piwheels / cryptography fix on "32-bit" RasPiOS 12 Lite [pinning to cryptography 40.0.1 no longer nec!] [create /etc/pip.conf missing from RasPiOS as nec, to avoid cffi glitch] #3632
Main remaining issue (on "32-bit" RasPiOS 12) appears to be KA Lite not finding Python 2:
- KA Lite fails to install on 32-bit RasPiOS 12 Bookworm: Python 2 installed but not usable? ("failed to find interpreter for Builtin discover of python_spec='python2.7'") #3635

holta · 2023-09-26T18:51:30Z

@EMG70 just FYI the Calibre-Web install issue on RasPiOS Bookworm (#3642) appears under control now, but might still be further improved as it converges here:

Missing package: lxml piwheels/packages#391

So let's test everything we can carefully on/around Friday 2023-10-06 if you have time to help then, as RasPiOS 12 is making big strides behind the scenes, and presumably is now on final approach very soon!

EMG70 · 2023-09-26T19:24:13Z

@EMG70 just FYI the Calibre-Web install issue on RasPiOS Bookworm (#3642) appears under control now, but might still be further improved as it converges here:

Missing package: lxml piwheels/packages#391

So let's test everything we can carefully on/around Friday 2023-10-06 if you have time to help then, as RasPiOS 12 is making big strides behind the scenes, and presumably is now on final approach very soon!

No problem. I will get in touch a few days before 06/10/23.

holta · 2023-09-26T20:54:09Z

No problem. I will get in touch a few days before 06/10/23.

Awesome. ASIDE / lots of grassroots folk are now testing pre-release Raspberry Pi OS Bookworm, so hopefully that means it'll be quite solid when it's released in coming-weeks-or-so — many months after Debian 12 Bookworm itself was released on 2023-06-10:

holta · 2023-09-26T21:46:03Z

@EMG70 just FYI the Calibre-Web install issue on RasPiOS Bookworm (#3642) appears under control now, but might still be further improved as it converges here:

Missing package: lxml piwheels/packages#391

Awesome News: https://piwheels.org just fixed it (lxml 4.9.3 for Python 3.11) today after I pinged them about this failed build!

holta · 2023-10-05T22:26:35Z

Two generally very successful RasPiOS 12/Bookworm pre-release tests — huge thanks to @EMG70. FYI he used graphical desktop versions of RasPiOS in both cases — each tested with a LARGE-sized IIAB install:

Fully 32-bit OS (using arm_64bit=0 in /boot/config.txt per Strange switch to 64bit kernel on 32bit RasPiOS [on 64-bit CPU architectures] #3516) on RPi 400:
http://sprunge.us/H36mDb?en
32-bit OS with 64-bit kernel / user space, on RPi 4:
http://sprunge.us/oGYxI6?en

There were a few errors like Error: NetworkManager is not running. — in both cases above — when running sudo iiab-network as this fully 32-bit screenshot shows:

holta · 2023-10-11T22:37:07Z

Now that Raspberry Pi OS Bookworm is released (e.g. 64-bit release notes), follow-up is happening on tickets like:

On RasPiOS Bookworm, just released: "Could not find the requested service systemd-resolved: host" #3657
PR PR For Discussion: "Just install systemd-resolved at the same place as systemd-networkd is installed" #3658

jvonau closed this as completed Apr 4, 2023

holta added this to the 8.1 milestone Apr 4, 2023

holta added the good 1st issue label Apr 4, 2023

holta pinned this issue Apr 4, 2023

jvonau mentioned this issue Apr 5, 2023

PLANNING: Does Debian 'bookworm' nightlies or Ubuntu 23.04 have python2 available or not? [KA Lite -> Kolibri discussion] #3289

Closed

This was referenced Apr 6, 2023

Python2 fix [for KA Lite on Debian 12 or Ubuntu 23.04 / 23.10] #3527

Merged

ka-lite libssl1.1 use raspi repo #3529

Merged

mongodb libssl1.1 #3530

Closed

Python2 fix [for KA Lite on Debian 12 or Ubuntu 23.04 / 23.10] #3534

Merged

holta reopened this Apr 11, 2023

holta added the help wanted label Apr 11, 2023

jvonau mentioned this issue Apr 11, 2023

notes and dependencies [for Python 2 on new OS's on {amd64, arm64, armhf}. Tested on Debian 12 & Ubuntu 23.04] #3535

Merged

holta mentioned this issue Aug 3, 2023

Munin fails to start until reboot, on ALL RasPiOS 12 pre-release tests (32-bit or 64-bit, RPi 4 or 400) #3603

Open

holta changed the title ~~How to pre-test RasPiOS 12~~ How to pre-test RasPiOS 12 [cffi wheel for Python 3.11 now exists, but fails] Sep 9, 2023

holta changed the title ~~How to pre-test RasPiOS 12 [cffi wheel for Python 3.11 now exists, but fails]~~ How to pre-test RasPiOS 12 [cffi wheel for Python 3.11 now exists] Sep 11, 2023

holta mentioned this issue Sep 26, 2023

Add option to change hotspot password iiab/iiab-admin-console#546

Closed

holta changed the title ~~How to pre-test RasPiOS 12 [cffi wheel for Python 3.11 now exists]~~ How to pre-test RasPiOS 12 [cffi, cryptography, lxml "wheels" for Python 3.11 now exist!] Sep 26, 2023

holta mentioned this issue Oct 1, 2023

32-bit fixes (Calibre-Web, JuperterHub, Moodle) + defer iiab-install of Internet Archive if 32-bit #3653

Merged

holta mentioned this issue Oct 11, 2023

On RasPiOS Bookworm, just released: "Could not find the requested service systemd-resolved: host" #3657

Closed

holta mentioned this issue Oct 14, 2023

sysd-netd-debian.yml: Install apt package 'systemd-resolved' if OS offers it (RasPiOS Bookworm issue) #3660

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to pre-test RasPiOS 12 [cffi, cryptography, lxml "wheels" for Python 3.11 now exist!] #3526

How to pre-test RasPiOS 12 [cffi, cryptography, lxml "wheels" for Python 3.11 now exist!] #3526

jvonau commented Apr 4, 2023 •

edited

jvonau commented Apr 4, 2023

holta commented Apr 4, 2023 •

edited

jvonau commented Apr 4, 2023

jvonau commented Apr 5, 2023

jvonau commented Apr 5, 2023

holta commented Apr 11, 2023 •

edited

jvonau commented Apr 11, 2023 •

edited by holta

holta commented Apr 11, 2023

EMG70 commented Apr 11, 2023

jvonau commented Apr 11, 2023 •

edited

jvonau commented Apr 11, 2023 •

edited

jvonau commented Apr 11, 2023

jvonau commented Apr 11, 2023

jvonau commented Apr 11, 2023

jvonau commented Apr 11, 2023 •

edited

holta commented Apr 11, 2023

jvonau commented Apr 11, 2023

EMG70 commented Apr 12, 2023

holta commented Jul 30, 2023

holta commented Sep 7, 2023 •

edited

holta commented Sep 7, 2023 •

edited

holta commented Sep 8, 2023

holta commented Sep 8, 2023 •

edited

holta commented Sep 9, 2023 •

edited

holta commented Sep 9, 2023

holta commented Sep 10, 2023

holta commented Sep 11, 2023

holta commented Sep 26, 2023 •

edited

EMG70 commented Sep 26, 2023

holta commented Sep 26, 2023 •

edited

holta commented Sep 26, 2023 •

edited

holta commented Oct 5, 2023 •

edited

holta commented Oct 11, 2023 •

edited

How to pre-test RasPiOS 12 [cffi, cryptography, lxml "wheels" for Python 3.11 now exist!] #3526

How to pre-test RasPiOS 12 [cffi, cryptography, lxml "wheels" for Python 3.11 now exist!] #3526

Comments

jvonau commented Apr 4, 2023 • edited

jvonau commented Apr 4, 2023

holta commented Apr 4, 2023 • edited

jvonau commented Apr 4, 2023

jvonau commented Apr 5, 2023

jvonau commented Apr 5, 2023

holta commented Apr 11, 2023 • edited

jvonau commented Apr 11, 2023 • edited by holta

holta commented Apr 11, 2023

EMG70 commented Apr 11, 2023

jvonau commented Apr 11, 2023 • edited

jvonau commented Apr 11, 2023 • edited

jvonau commented Apr 11, 2023

jvonau commented Apr 11, 2023

jvonau commented Apr 11, 2023

jvonau commented Apr 11, 2023 • edited

holta commented Apr 11, 2023

jvonau commented Apr 11, 2023

EMG70 commented Apr 12, 2023

holta commented Jul 30, 2023

holta commented Sep 7, 2023 • edited

holta commented Sep 7, 2023 • edited

holta commented Sep 8, 2023

holta commented Sep 8, 2023 • edited

holta commented Sep 9, 2023 • edited

holta commented Sep 9, 2023

holta commented Sep 10, 2023

holta commented Sep 11, 2023

holta commented Sep 26, 2023 • edited

EMG70 commented Sep 26, 2023

holta commented Sep 26, 2023 • edited

holta commented Sep 26, 2023 • edited

holta commented Oct 5, 2023 • edited

holta commented Oct 11, 2023 • edited

jvonau commented Apr 4, 2023 •

edited

holta commented Apr 4, 2023 •

edited

holta commented Apr 11, 2023 •

edited

jvonau commented Apr 11, 2023 •

edited by holta

jvonau commented Apr 11, 2023 •

edited

jvonau commented Apr 11, 2023 •

edited

jvonau commented Apr 11, 2023 •

edited

holta commented Sep 7, 2023 •

edited

holta commented Sep 7, 2023 •

edited

holta commented Sep 8, 2023 •

edited

holta commented Sep 9, 2023 •

edited

holta commented Sep 26, 2023 •

edited

holta commented Sep 26, 2023 •

edited

holta commented Sep 26, 2023 •

edited

holta commented Oct 5, 2023 •

edited

holta commented Oct 11, 2023 •

edited