Developer Notes: Updating Python

X1 has a truly ancient userspace, and building for it gets very annoying very quickly. Here is how I (@jwise) built a Python-3.12.3.

Put the correct gcc version on your path.

$ CROSS=$HOME/bambu/RV1126_RV1109_SDK_2.2.2/prebuilts/gcc/linux-x86/arm/gcc-arm-8.3-2019.03-x86_64-arm-linux-gnueabihf/
$ PATH=$CROSS/bin/:$PATH

Grab and build libffi.

joshua@anima:~/bambu/osrel/pkgs$ md5sum libffi_3.3.orig.tar.gz
6313289e32f1d38a9df4770b014a2ca7  libffi_3.3.orig.tar.gz
joshua@anima:~/bambu/osrel/pkgs$ tar xzvf libffi_3.3.orig.tar.gz
[...]
joshua@anima:~/bambu/osrel/pkgs/libffi-3.3$ ./configure --host=arm-linux-gnueabihf --prefix=/opt/python CC=arm-linux-gnueabihf-gcc
[...]
joshua@anima:~/bambu/osrel/pkgs/libffi-3.3$ make
[...]
joshua@anima:~/bambu/osrel/pkgs/libffi-3.3$ make install DESTDIR=$HOME/bambu/osrel/root/
[...]

Grab and build OpenSSL.

joshua@anima:~/bambu/osrel/pkgs$ md5sum openssl-3.0.13.tar.gz
c15e53a62711002901d3515ac8b30b86  openssl-3.0.13.tar.gz
[...]
joshua@anima:~/bambu/osrel/pkgs/openssl-3.0.13$ ./Configure linux-armv4 shared -DL_ENDIAN --prefix=/opt/python --openssldir=/opt/python --cross-compile-prefix=arm-linux-gnueabihf-
[...]
joshua@anima:~/bambu/osrel/pkgs/openssl-3.0.13$ make -j4
[...]
joshua@anima:~/bambu/osrel/pkgs/openssl-3.0.13$ make install DESTDIR=$HOME/bambu/osrel/root/
[...]

Grab enough of a libz to build python.

joshua@anima:~/bambu/osrel$ cp ../rootfs/usr/lib/libz.so* root/opt/python/lib/
joshua@anima:~/bambu/osrel$ for i in zconf zdict zlib; do cp /usr/include/$i.h root/opt/python/include/; done

Grab and build Python.

joshua@anima:~/bambu/osrel/pkgs$ md5sum Python-3.12.3.tgz
3c5498a34d5226c9b746b1199f0bf2d9  Python-3.12.3.tgz
joshua@anima:~/bambu/osrel/pkgs$ tar xzvf Python-3.12.3.tgz
[...]

Build a bootstrap python.

$ mkdir bootstrap; cd bootstrap; ../configure; make -j4; make install DESTDIR=.

Build a target Python.

$ mkdir arm; cd arm
$ cat > config.site <<EOF
ac_cv_file__dev_ptmx=yes
ac_cv_file__dev_ptc=no
EOF
joshua@anima:~/bambu/osrel/pkgs/Python-3.12.3/arm$ ../configure --build=x86_64-pc-linux-gnu --with-build-python=../bootstrap/python --host=arm-linux-gnueabihf CC=arm-linux-gnueabihf-gcc CPP=arm-linux-gnueabihf-cpp --with-pkg-config=no LDFLAGS=-L$HOME/bambu/osrel/root/opt/python/lib CFLAGS=-I$HOME/bambu/osrel/root/opt/python/include --disable-ipv6 CONFIG_SITE=config.site --enable-optimizations --with-openssl=$HOME/bambu/osrel/root/opt/python/ --prefix=/opt/python
[...]
$ make -j4
[...]
$ make install DESTDIR=$HOME/bambu/osrel/root/
[...]
$ patchelf --add-rpath /opt/python/lib $HOME/bambu/osrel/root/opt/python/bin/python3.12
$ patchelf --add-rpath /opt/python/lib $HOME/bambu/osrel/root/opt/python/lib/python3.12/lib-dynload/*
$ patchelf --add-rpath /userdata/x1plus/lib $HOME/bambu/osrel/root/opt/python/bin/python3.12
$ patchelf --add-rpath /userdata/x1plus/lib $HOME/bambu/osrel/root/opt/python/lib/python3.12/lib-dynload/*

Build other things we need.

$ bootstrap/dest/usr/local/bin/python3 -m ensurepip
$ bootstrap/dest/usr/local/bin/python3 -m pip install crossenv
$ bootstrap/dest/usr/local/bin/python3 -m crossenv ../../root/opt/python/bin/python3 crossenv-python3
$ . crossenv-python3/bin/activate 
(cross) $ cd ..
(cross) $ git clone https://github.com/gvalkov/python-evdev; cd python-evdev
(cross) $ cross-pip3 install setuptools
(cross) $ cross-pip3 install requests urllib3
(cross) $ cross-python3 ./setup.py build build_ecodes --evdev-headers $CROSS/arm-linux-gnueabihf/libc/usr/include/linux/input.h:$CROSS/arm-linux-gnueabihf/libc/usr/include/linux/input-event-codes.h build_ext install
$ deactivate

Start cleaning things up...

joshua@anima:~/bambu/osrel/root/opt/python$ du -s .
376348	.
joshua@anima:~/bambu/osrel$ cp -R root dist/
joshua@anima:~/bambu/osrel/dist/opt/python$ rm bin/openssl bin/c_rehash
joshua@anima:~/bambu/osrel/dist/opt/python$ arm-linux-gnueabihf-strip bin/python3.12
joshua@anima:~/bambu/osrel/dist/opt/python$ rm lib/*.a
joshua@anima:~/bambu/osrel/dist/opt/python$ rm lib/libz.so*
joshua@anima:~/bambu/osrel/dist/opt/python$ rm -rf lib/pkgconfig/
joshua@anima:~/bambu/osrel/dist/opt/python$ arm-linux-gnueabihf-strip lib/*.so
joshua@anima:~/bambu/osrel/dist/opt/python$ rm -rf share private
joshua@anima:~/bambu/osrel/dist/opt/python$ rm openssl.cnf*
joshua@anima:~/bambu/osrel/dist/opt/python$ rm ct_log_list.cnf*
joshua@anima:~/bambu/osrel/dist/opt/python$ rm -rf misc certs include
joshua@anima:~/bambu/osrel/dist/opt/python$ rm -rf lib/python3.12/config-3.12-arm-linux-gnueabihf/
joshua@anima:~/bambu/osrel/dist/opt/python$ rm -rf lib/python3.12/test/
joshua@anima:~/bambu/osrel/dist/opt/python$ rm -rf ./lib/python3.12/pydoc_data  ./lib/python3.12/ensurepip ./lib/python3.12/lib2to3 ./lib/python3.12/turtledemo ./lib/python3.12/tkinter
joshua@anima:~/bambu/osrel/dist/opt/python$ arm-linux-gnueabihf-strip lib/python3.12/lib-dynload/*
joshua@anima:~/bambu/osrel/dist/opt/python$ rm lib/python3.12/lib-dynload/_test*
joshua@anima:~/bambu/osrel/dist/opt/python$ cp -R ../../../pkgs/Python-3.12.3/crossenv-python3/cross/lib/python3.12/site-packages/evdev-1.7.0-py3.12-linux-arm.egg/evdev/ lib/python3.12/site-packages/
joshua@anima:~/bambu/osrel/dist/opt/python$ cp -R ../../../pkgs/Python-3.12.3/crossenv-python3/cross/lib/python3.12/site-packages/{certifi,charset_normalizer,idna,requests,urllib3}* lib/python3.12/site-packages/
joshua@anima:~/bambu/osrel/dist/opt/python$ find ./lib/python3.12 -iname '*.opt-*.pyc' -exec 'rm' '{}' ';'
joshua@anima:~/bambu/osrel/dist/opt/python$ du -s .
42836	.

Almost an order of magnitude better!

Make sure your build works:

joshua@anima:~/bambu/osrel/dist/opt/python$ qemu-arm -L $ROOTFS bin/python3
Python 3.12.3 (main, May  4 2024, 03:55:51) [GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

Phew.

Build a prebuilt package.

joshua@anima:~/bambu/osrel/dist$ tar czvf ../../x1plus/prebuilt/python3.tar.gz -C opt/python/  .

Edit x1plus/images/cfw.py to install site-packages to the right place. Update python caches.

joshua@anima:~/bambu/x1plus/images$ find site-packages/jeepney -iname '*.cpython-310.pyc' -exec rm '{}' ';'

Build an image. Have (another?) drink. Go to bed.