Skip to content

Commit

Permalink
Statically compile psutil into python
Browse files Browse the repository at this point in the history
Add usage notes to the README.md and build notes as well. There will be a separate repository (at a later time) with the examples baked in.
  • Loading branch information
Lisa Seelye committed May 23, 2018
1 parent 5ee5985 commit c1536cc
Show file tree
Hide file tree
Showing 8 changed files with 147 additions and 7 deletions.
15 changes: 15 additions & 0 deletions python/BPO-7938_pr4338-fix-makesetup-script.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
--- Modules/makesetup.dist 2018-05-23 01:58:50.034767531 +0000
+++ Modules/makesetup 2018-05-23 01:59:36.207811984 +0000
@@ -128,7 +128,11 @@

# Output DEFS in reverse order so first definition overrides
case $line in
- *=*) DEFS="$line$NL$DEFS"; continue;;
+ *=*) if [ $(sed -e 's$::=\|:=\|+=\|?=$=$' -e 's$\s*=.*$$' <<< $line | wc -w) == 1 ]
+ then
+ DEFS="$line$NL$DEFS"; continue;
+ fi
+ ;;
'include '*) DEFS="$line$NL$DEFS"; continue;;
'*noobjects*')
case $noobjects in
5 changes: 5 additions & 0 deletions python/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,11 @@ FROM scratch
MAINTAINER Lisa Seelye <lisa@thedoh.com>
WORKDIR /
COPY --from=0 /output/python* /

# just in case things need them to exist, like flask
COPY passwd /etc/passwd
COPY group /etc/group

ENV \
PYTHONHOME=/python2.7.zip \
PYTHONPATH=/python2.7.zip
Expand Down
44 changes: 44 additions & 0 deletions python/README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# Disclaimer

The code and Docker image provided by this repository is provided without any warranty by the author(s) of any components included herein. Use at your own risk.

# Building Python

This will build Python-2.7.15rc1 into a static container:
Expand All @@ -17,6 +21,44 @@ By default the image will run `python -sS /entrypoint.py` but if you would prefe
[GCC 5.3.0] on linux2
>>>

## Build Notes

The image includes the [psutil](https://github.com/giampaolo/psutil) package which is available under the following BSD license:

> psutil is distributed under BSD license reproduced below.
>
> Copyright (c) 2009, Jay Loden, Dave Daeschler, Giampaolo Rodola'
> All rights reserved.
>
> Redistribution and use in source and binary forms, with or without modification,
> are permitted provided that the following conditions are met:
>
> * Redistributions of source code must retain the above copyright notice, this
> list of conditions and the following disclaimer.
> * Redistributions in binary form must reproduce the above copyright notice,
> this list of conditions and the following disclaimer in the documentation
> and/or other materials provided with the distribution.
> * Neither the name of the psutil authors nor the names of its contributors
> may be used to endorse or promote products derived from this software without
> specific prior written permission.
>
> THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
> ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
> WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
> DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR
> ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
> (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
> LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON
> ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
> (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
> SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
The package is included to illustrate the possibility of compiling third party modules into the Python static image. The `psuutil` package is patched with two patches (below) to force the `psutil` Python library code to _not_ look in its directory (`from . import`) and rather to anywhere the requirement can be met (such as from within the `python` binary).
* [psutil_import__init__.patch](psutil_import__init__.patch)
* [psutil_import_pslinux.patch](psutil_import_pslinux.patch)

A common use case will be to `pip install` some set of requirements for use. Since some of those requirements may have C code to compile it is thus a requirement to statically compile that code into the `python` binary because there are no shared Python objects to link against. The inclusion of `psutil` demonstrates this process.

## Entrypoint.py

A sample `entrypoint.py` to call an app using Factory pattern might be:
Expand Down Expand Up @@ -58,3 +100,5 @@ Or `subprocess`:

subprocess.call(["/python", "-s","-S","/myapp.py"])
# Code execution continues

Or a factory pattern.
55 changes: 48 additions & 7 deletions python/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,33 @@ TERMCAP_VERSION=1.3.1
READLINE_VERSION=6.3
OPENSSL_VERSION=1.0.2k
PYTHON_BASE_VERSION=2.7.15
PSUTIL_VERSION=5.4.5
PYTHON_RC=rc1

CPU_COUNT=$(nproc --all 2>/dev/null)
MAKE_J="-j${CPU_COUNT:-1}"

function fetch_psutil() {
cd /build
curl -L -o /build/psutil-${PSUTIL_VERSION}.zip https://github.com/giampaolo/psutil/archive/release-${PSUTIL_VERSION}.zip
unzip /build/psutil-${PSUTIL_VERSION}.zip
mv psutil-release-${PSUTIL_VERSION} psutil-${PSUTIL_VERSION}
# Prune out the Python files from the C files and then monkeypatch the
# Python files
mkdir -p psutil/{c_files,python_files}
cp -r /build/psutil-${PSUTIL_VERSION}/psutil psutil/c_files/
rm -rf psutil/c_files/psutil/*.py* psutil/c_files/psutil/DEVNOTES psutil/c_files/psutil/tests
# Python files
cp -r /build/psutil-${PSUTIL_VERSION}/psutil psutil/python_files/
rm -rf psutil/python_files/psutil/*.c psutil/python_files/psutil/*.h psutil/python_files/psutil/arch psutil/python_files/psutil/tests
# Patch now
cd psutil
patch --ignore-whitespace -p0 < /build/psutil_import__init__.patch
patch --ignore-whitespace -p0 < /build/psutil_import_pslinux.patch
cd ..
# Ready to be copied for Lib
}


function build_zlib() {
cd /build
Expand All @@ -25,7 +50,7 @@ function build_zlib() {
CC='/opt/cross/x86_64-linux-musl/bin/x86_64-linux-musl-gcc -static -fPIC' \
./configure \
--static
make -j4
make ${MAKE_J}
}

function build_termcap() {
Expand All @@ -41,7 +66,7 @@ function build_termcap() {
./configure \
--disable-shared \
--enable-static
make -j4
make ${MAKE_J}
}

function build_readline() {
Expand All @@ -57,7 +82,7 @@ function build_readline() {
./configure \
--disable-shared \
--enable-static
make -j4
make ${MAKE_J}

# Note that things look for readline in <readline/readline.h>, so we need
# that directory to exist.
Expand Down Expand Up @@ -89,6 +114,13 @@ function build_python() {
tar -xvf Python-${PYTHON_BASE_VERSION}${PYTHON_RC}.tar
cd Python-${PYTHON_BASE_VERSION}${PYTHON_RC}

# Copy psutil source code into place
mkdir Modules/psutil
cp -r /build/psutil/c_files/psutil/* Modules/psutil/
# Convert the version to what psutil expects. This should be stable until
# their setup.py changes.
psversion=$(echo $PSUTIL_VERSION | sed 's,\.,,g')

# Set up modules
cp Modules/Setup.dist Modules/Setup
MODULES="_bisect _collections _csv _datetime _elementtree _functools _heapq _io _md5 _posixsubprocese _random _sha _sha256 _sha512 _socket _struct _weakref array binascii cmath cStringIO cPickle datetime fcntl future_builtins grp itertools math mmap operator parser readline resource select spwd strop syslog termios time unicodedata zlib"
Expand All @@ -115,13 +147,20 @@ function build_python() {
cat << 'EOF' >>Modules/Setup
_lsprof rotatingtree.c _lsprof.c
pyexpat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DUSE_PYEXPAT_CAPI -DHAVE_SYSCALL_GETRANDOM
_psutil_linux psutil/_psutil_common.c psutil/_psutil_posix.c psutil/_psutil_linux.c -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=__PSVER__ -DPSUTIL_LINUX=1
_psutil_posix psutil/_psutil_common.c psutil/_psutil_posix.c -DPSUTIL_POSIX=1 -DPSUTIL_VERSION=__PSVER__ -DPSUTIL_LINUX=1
EOF

# Enable OpenSSL support
patch --ignore-whitespace -p1 < /build/cpython-enable-openssl.patch

# Fix https://bugs.python.org/issue7938
echo "Patching for https://bugs.python.org/issue7938"
patch --ignore-whitespace -p0 < /build/BPO-7938_pr4338-fix-makesetup-script.patch

sed -i \
-e "s|^SSL=/build/openssl-TKTK|SSL=/build/openssl-${OPENSSL_VERSION}|" \
-e "s|__PSVER__|${psversion}|g" \
Modules/Setup

# Configure
Expand All @@ -132,29 +171,31 @@ EOF
--disable-shared

# Build
make --trace -j4 LDFLAGS="-static" LINKFORSHARED=" "
make --trace ${MAKE_J} LDFLAGS="-static" LINKFORSHARED=" " || true


/opt/cross/x86_64-linux-musl/bin/x86_64-linux-musl-strip python
# There may be a better way to ensure _sysconfigdata.py is included in the
# .zip file, but I am not sure how best to do it, so this is a bit of a
# hack, banking on the contents of this container staying around for it
# to matter.
cp $(find . -name _sysconfigdata.py -print) Lib
# Copy the patched psutil Python files into place
cp -r /build/psutil/python_files/psutil Lib
cd Lib && zip -r ../python2.7.zip .

}

function doit() {
fetch_psutil
build_zlib
build_termcap
build_readline
build_openssl
build_python


mkdir -p /output
cp /build/Python-${PYTHON_BASE_VERSION}${PYTHON_RC}/{python,python2.7.zip} /output


}

doit
Expand Down
1 change: 1 addition & 0 deletions python/group
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
root:x:0:root
1 change: 1 addition & 0 deletions python/passwd
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
root:x:0:0:root:/root:/bin/bash
14 changes: 14 additions & 0 deletions python/psutil_import__init__.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
--- python_files/psutil/__init__.py.orig 2018-05-23 02:56:12.239724882 +0000
+++ python_files/psutil/__init__.py 2018-05-23 02:56:31.089602431 +0000
@@ -96,7 +96,10 @@
# via sys.modules.
PROCFS_PATH = "/proc"

- from . import _pslinux as _psplatform
+ try:
+ from . import _pslinux as _psplatform
+ except ImportError:
+ import _pslinux as _psplatform

from ._pslinux import IOPRIO_CLASS_BE # NOQA
from ._pslinux import IOPRIO_CLASS_IDLE # NOQA
19 changes: 19 additions & 0 deletions python/psutil_import_pslinux.patch
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
--- python_files/psutil/_pslinux.py.orig 2018-05-23 02:56:42.366195862 +0000
+++ python_files/psutil/_pslinux.py 2018-05-23 02:56:58.489424487 +0000
@@ -23,8 +23,14 @@

from . import _common
from . import _psposix
-from . import _psutil_linux as cext
-from . import _psutil_posix as cext_posix
+try:
+ from . import _psutil_linux as cext
+except ImportError:
+ import _psutil_linux as cext
+try:
+ from . import _psutil_posix as cext_posix
+except ImportError:
+ import _psutil_posix as cext_posix
from ._common import ENCODING
from ._common import ENCODING_ERRS
from ._common import isfile_strict

0 comments on commit c1536cc

Please sign in to comment.