Self installation of python modules

Prerequisites

To install python modules that have C extensions install gcc as described in Using-GCC-for-native-compilation. You do not need to assign any special environment variables by running /opt/bin/gcc_env.sh, all needed flags will be used automatically by python. If your Python module has C extension that depends on some library, install this library with opkg before python module building.

Using pip

Install python3-pip package opkg install python3-pip and run pip install --upgrade pip setuptools.

If your home directory point to ram (tmpfs), create a directory on your storage. pip creates big caches and will fill you ram quickly. To install a python module simply run HOME=<storage directory> pip install <module-name>:

# HOME=/share/HDA_DATA/Public/test-modules/python pip install regex
Collecting regex
  Downloading regex-2016.01.10.tar.gz (574kB)
    100% |--------------------------------| 577kB 108kB/s
Installing collected packages: regex
  Running setup.py install for regex ... done
Successfully installed regex-2016.1.10

pip will automatically download all dependent python modules and compile C extensions, if any:

# HOME=/share/HDA_DATA/Public/test-modules/python pip show -f regex
---
Metadata-Version: 1.1
Name: regex
Version: 2016.1.10
Summary: Alternative regular expression module, to replace re.
Home-page: https://bitbucket.org/mrabarnett/mrab-regex
Author: Matthew Barnett
Author-email: regex@mrabarnett.plus.com
License: Python Software Foundation License
Location: /share/HDA_DATA/.qpkg/Entware/lib/python2.7/site-packages
Requires:
Files:
  _regex.so
  _regex_core.py
  _regex_core.pyc
  regex-2016.1.10-py2.7.egg-info
  regex-2016.1.10-py2.7.egg-info/PKG-INFO
  regex-2016.1.10-py2.7.egg-info/SOURCES.txt
  regex-2016.1.10-py2.7.egg-info/dependency_links.txt
  regex-2016.1.10-py2.7.egg-info/top_level.txt
  regex.py
  regex.pyc
  test_regex.py
  test_regex.pyc

_regex.so file here was compiled automatically by pip.

Note: In some cases (armv5 architecture is an example) pip will reinstall already installed modules. In this case you can run PIP_IGNORE_INSTALLED=0 pip install <module_name> to skip modules reinstallation.

Using Python

Sometimes you may encounter some errors during compilation. For example when you use pip to install cffi the libffi library will not be found. Entware (Openwrt in fact) does not install libffi.so symlink in /opt/lib. There is only symlink libffi.so.6 there. You can create missing link manually or fix the installation files. This can be done as follows. First download cffi archive and unpack to some directory. Short investigation shows that setup.py has a line libraries = ['ffi'] with the missing library. Edit it, and change to libraries = [':libffi.so.6']. Now run python setup.py build:

# python setup.py build
running build
running build_py
creating build
creating build/lib.linux-armv5tel-2.7
creating build/lib.linux-armv5tel-2.7/cffi
copying cffi/__init__.py -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/api.py -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/lock.py -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/commontypes.py -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/ffiplatform.py -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/vengine_gen.py -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/gc_weakref.py -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/model.py -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/verifier.py -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/recompiler.py -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/cparser.py -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/_cffi_include.h -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/parse_c_type.h -> build/lib.linux-armv5tel-2.7/cffi
copying cffi/_embedding.h -> build/lib.linux-armv5tel-2.7/cffi
warning: build_py: byte-compiling is disabled, skipping.

running build_ext
building '_cffi_backend' extension
creating build/temp.linux-armv5tel-2.7
creating build/temp.linux-armv5tel-2.7/c
arm-openwrt-linux-gnueabi-gcc -fno-strict-aliasing -O2 -pipe -march=armv5te -mtune=xscale -fno-caller-saves -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -mfloat-abi=soft -DNDEBUG -fno-inline -DNDEBUG -O2 -pipe -march=armv5te -mtune=xscale -fno-caller-saves -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -mfloat-abi=soft -fPIC -DUSE__THREAD -I/usr/include/ffi -I/usr/include/libffi -I/opt/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-armv5tel-2.7/c/_cffi_backend.o
arm-openwrt-linux-gnueabi-gcc -shared -L/media/ware2/Entware.2016.02/staging_dir/target-arm_xscale_glibc-2.22_eabi/opt/lib -L/media/ware2/Entware.2016.02/staging_dir/target-arm_xscale_glibc-2.22_eabi/lib -Wl,-rpath,/opt/lib -Wl,-rpath-link=/media/ware2/Entware.2016.02/staging_dir/target-arm_xscale_glibc-2.22_eabi/opt/lib -Wl,--dynamic-linker=/opt/lib/ld-linux.so.3 -L/media/ware2/Entware.2016.02/staging_dir/toolchain-arm_xscale_gcc-4.8.5_glibc-2.22_eabi/usr/lib -L/media/ware2/Entware.2016.02/staging_dir/toolchain-arm_xscale_gcc-4.8.5_glibc-2.22_eabi/lib -L/media/ware2/Entware.2016.02/build_dir/target-arm_xscale_glibc-2.22_eabi/Python-2.7.11 build/temp.linux-armv5tel-2.7/c/_cffi_backend.o -L/opt/lib -l:libffi.so.6 -lpython2.7 -o build/lib.linux-armv5tel-2.7/_cffi_backend.so
.....

Finish installation with python setup.py install

# python setup.py install
running install
running bdist_egg
running egg_info
writing requirements to cffi.egg-info/requires.txt
writing cffi.egg-info/PKG-INFO
writing top-level names to cffi.egg-info/top_level.txt
writing dependency_links to cffi.egg-info/dependency_links.txt
writing entry points to cffi.egg-info/entry_points.txt
reading manifest file 'cffi.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'cffi.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-armv5tel/egg
running install_lib
.......