Skip to content
Permalink
Browse files

Statically compile psutil into python

Add usage notes to the README.md and build notes as well. There will be a separate repository (at a later time) with the examples baked in.
  • Loading branch information...
Lisa Seelye
Lisa Seelye committed May 23, 2018
1 parent 5ee5985 commit c1536cc8a80461c3f41538170a39da0ed5255535
@@ -0,0 +1,15 @@
--- Modules/makesetup.dist 2018-05-23 01:58:50.034767531 +0000
+++ Modules/makesetup 2018-05-23 01:59:36.207811984 +0000
@@ -128,7 +128,11 @@

# Output DEFS in reverse order so first definition overrides
case $line in
- *=*) DEFS="$line$NL$DEFS"; continue;;
+ *=*) if [ $(sed -e 's$::=\|:=\|+=\|?=$=$' -e 's$\s*=.*$$' <<< $line | wc -w) == 1 ]
+ then
+ DEFS="$line$NL$DEFS"; continue;
+ fi
+ ;;
'include '*) DEFS="$line$NL$DEFS"; continue;;
'*noobjects*')
case $noobjects in
@@ -16,6 +16,11 @@ FROM scratch
MAINTAINER Lisa Seelye <lisa@thedoh.com>
WORKDIR /
COPY --from=0 /output/python* /

# just in case things need them to exist, like flask
COPY passwd /etc/passwd
COPY group /etc/group

ENV \
PYTHONHOME=/python2.7.zip \
PYTHONPATH=/python2.7.zip
@@ -1,3 +1,7 @@
# Disclaimer

The code and Docker image provided by this repository is provided without any warranty by the author(s) of any components included herein. Use at your own risk.

# Building Python

This will build Python-2.7.15rc1 into a static container:
@@ -17,6 +21,44 @@ By default the image will run `python -sS /entrypoint.py` but if you would prefe
[GCC 5.3.0] on linux2
>>>
## Build Notes

The image includes the [psutil](https://github.com/giampaolo/psutil) package which is available under the following BSD license:

> psutil is distributed under BSD license reproduced below.
>
> Copyright (c) 2009, Jay Loden, Dave Daeschler, Giampaolo Rodola'
> All rights reserved.
>
> Redistribution and use in source and binary forms, with or without modification,
> are permitted provided that the following conditions are met:
>
> * Redistributions of source code must retain the above copyright notice, this
> list of conditions and the following disclaimer.
> * Redistributions in binary form must reproduce the above copyright notice,
> this list of conditions and the following disclaimer in the documentation
> and/or other materials provided with the distribution.
> * Neither the name of the psutil authors nor the names of its contributors
> may be used to endorse or promote products derived from this software without
> specific prior written permission.
>
> THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
> ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
> WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
> DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
> ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
> (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
> LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
> ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
> SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The package is included to illustrate the possibility of compiling third party modules into the Python static image. The `psuutil` package is patched with two patches (below) to force the `psutil` Python library code to _not_ look in its directory (`from . import`) and rather to anywhere the requirement can be met (such as from within the `python` binary).
* [psutil_import__init__.patch](psutil_import__init__.patch)
* [psutil_import_pslinux.patch](psutil_import_pslinux.patch)

A common use case will be to `pip install` some set of requirements for use. Since some of those requirements may have C code to compile it is thus a requirement to statically compile that code into the `python` binary because there are no shared Python objects to link against. The inclusion of `psutil` demonstrates this process.

## Entrypoint.py

A sample `entrypoint.py` to call an app using Factory pattern might be:
@@ -58,3 +100,5 @@ Or `subprocess`:

subprocess.call(["/python", "-s","-S","/myapp.py"])
# Code execution continues

Or a factory pattern.
@@ -10,8 +10,33 @@ TERMCAP_VERSION=1.3.1
READLINE_VERSION=6.3
OPENSSL_VERSION=1.0.2k
PYTHON_BASE_VERSION=2.7.15
PSUTIL_VERSION=5.4.5
PYTHON_RC=rc1

CPU_COUNT=$(nproc --all 2>/dev/null)
MAKE_J="-j${CPU_COUNT:-1}"

function fetch_psutil() {
cd /build
curl -L -o /build/psutil-${PSUTIL_VERSION}.zip https://github.com/giampaolo/psutil/archive/release-${PSUTIL_VERSION}.zip
unzip /build/psutil-${PSUTIL_VERSION}.zip
mv psutil-release-${PSUTIL_VERSION} psutil-${PSUTIL_VERSION}
# Prune out the Python files from the C files and then monkeypatch the
# Python files
mkdir -p psutil/{c_files,python_files}
cp -r /build/psutil-${PSUTIL_VERSION}/psutil psutil/c_files/
rm -rf psutil/c_files/psutil/*.py* psutil/c_files/psutil/DEVNOTES psutil/c_files/psutil/tests
# Python files
cp -r /build/psutil-${PSUTIL_VERSION}/psutil psutil/python_files/
rm -rf psutil/python_files/psutil/*.c psutil/python_files/psutil/*.h psutil/python_files/psutil/arch psutil/python_files/psutil/tests
# Patch now
cd psutil
patch --ignore-whitespace -p0 < /build/psutil_import__init__.patch
patch --ignore-whitespace -p0 < /build/psutil_import_pslinux.patch
cd ..
# Ready to be copied for Lib
}


function build_zlib() {
cd /build
@@ -25,7 +50,7 @@ function build_zlib() {
CC='/opt/cross/x86_64-linux-musl/bin/x86_64-linux-musl-gcc -static -fPIC' \
./configure \
--static
make -j4
make ${MAKE_J}
}

function build_termcap() {
@@ -41,7 +66,7 @@ function build_termcap() {
./configure \
--disable-shared \
--enable-static
make -j4
make ${MAKE_J}
}

function build_readline() {
@@ -57,7 +82,7 @@ function build_readline() {
./configure \
--disable-shared \
--enable-static
make -j4
make ${MAKE_J}

# Note that things look for readline in <readline/readline.h>, so we need
# that directory to exist.
@@ -89,6 +114,13 @@ function build_python() {
tar -xvf Python-${PYTHON_BASE_VERSION}${PYTHON_RC}.tar
cd Python-${PYTHON_BASE_VERSION}${PYTHON_RC}

# Copy psutil source code into place
mkdir Modules/psutil
cp -r /build/psutil/c_files/psutil/* Modules/psutil/
# Convert the version to what psutil expects. This should be stable until
# their setup.py changes.
psversion=$(echo $PSUTIL_VERSION | sed 's,\.,,g')

# Set up modules
cp Modules/Setup.dist Modules/Setup
MODULES="_bisect _collections _csv _datetime _elementtree _functools _heapq _io _md5 _posixsubprocese _random _sha _sha256 _sha512 _socket _struct _weakref array binascii cmath cStringIO cPickle datetime fcntl future_builtins grp itertools math mmap operator parser readline resource select spwd strop syslog termios time unicodedata zlib"
@@ -115,13 +147,20 @@ function build_python() {
cat << 'EOF' >>Modules/Setup
_lsprof rotatingtree.c _lsprof.c
pyexpat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DUSE_PYEXPAT_CAPI -DHAVE_SYSCALL_GETRANDOM
_psutil_linux psutil/_psutil_common.c psutil/_psutil_posix.c psutil/_psutil_linux.c -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=__PSVER__ -DPSUTIL_LINUX=1
_psutil_posix psutil/_psutil_common.c psutil/_psutil_posix.c -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=__PSVER__ -DPSUTIL_LINUX=1
EOF

# Enable OpenSSL support
patch --ignore-whitespace -p1 < /build/cpython-enable-openssl.patch

# Fix https://bugs.python.org/issue7938
echo "Patching for https://bugs.python.org/issue7938"
patch --ignore-whitespace -p0 < /build/BPO-7938_pr4338-fix-makesetup-script.patch

sed -i \
-e "s|^SSL=/build/openssl-TKTK|SSL=/build/openssl-${OPENSSL_VERSION}|" \
-e "s|__PSVER__|${psversion}|g" \
Modules/Setup

# Configure
@@ -132,29 +171,31 @@ EOF
--disable-shared

# Build
make --trace -j4 LDFLAGS="-static" LINKFORSHARED=" "
make --trace ${MAKE_J} LDFLAGS="-static" LINKFORSHARED=" " || true


/opt/cross/x86_64-linux-musl/bin/x86_64-linux-musl-strip python
# There may be a better way to ensure _sysconfigdata.py is included in the
# .zip file, but I am not sure how best to do it, so this is a bit of a
# hack, banking on the contents of this container staying around for it
# to matter.
cp $(find . -name _sysconfigdata.py -print) Lib
# Copy the patched psutil Python files into place
cp -r /build/psutil/python_files/psutil Lib
cd Lib && zip -r ../python2.7.zip .

}

function doit() {
fetch_psutil
build_zlib
build_termcap
build_readline
build_openssl
build_python


mkdir -p /output
cp /build/Python-${PYTHON_BASE_VERSION}${PYTHON_RC}/{python,python2.7.zip} /output


}

doit
@@ -0,0 +1 @@
root:x:0:root
@@ -0,0 +1 @@
root:x:0:0:root:/root:/bin/bash
@@ -0,0 +1,14 @@
--- python_files/psutil/__init__.py.orig 2018-05-23 02:56:12.239724882 +0000
+++ python_files/psutil/__init__.py 2018-05-23 02:56:31.089602431 +0000
@@ -96,7 +96,10 @@
# via sys.modules.
PROCFS_PATH = "/proc"

- from . import _pslinux as _psplatform
+ try:
+ from . import _pslinux as _psplatform
+ except ImportError:
+ import _pslinux as _psplatform

from ._pslinux import IOPRIO_CLASS_BE # NOQA
from ._pslinux import IOPRIO_CLASS_IDLE # NOQA
@@ -0,0 +1,19 @@
--- python_files/psutil/_pslinux.py.orig 2018-05-23 02:56:42.366195862 +0000
+++ python_files/psutil/_pslinux.py 2018-05-23 02:56:58.489424487 +0000
@@ -23,8 +23,14 @@

from . import _common
from . import _psposix
-from . import _psutil_linux as cext
-from . import _psutil_posix as cext_posix
+try:
+ from . import _psutil_linux as cext
+except ImportError:
+ import _psutil_linux as cext
+try:
+ from . import _psutil_posix as cext_posix
+except ImportError:
+ import _psutil_posix as cext_posix
from ._common import ENCODING
from ._common import ENCODING_ERRS
from ._common import isfile_strict

0 comments on commit c1536cc

Please sign in to comment.
You can’t perform that action at this time.