UWSGI_AS_LIB

[lorencarvalho]

hey @kwlzn !

Firstly, I'm a big fan of this project -- but I recently discovered (accidentally) that when building uwsgi you can set an environment variable UWSGI_AS_LIB which the build system checks for. If supplied, it will compile uwsgi as a shared library!

This can be used with python's ctypes to invoke the uwsgi executable from within the python runtime. Here's an example (from this issue on unbit's page unbit/uwsgi#564):

import sys
import ctypes

def application(e, sr):
    sr('200 OK',[('Content-Type','text/html')])
    return ["Hello World"]

def uwsgi_run(uwsgi_args):
    # load the uwsgi library in the global namespace
    uwsgi = ctypes.CDLL('./libuwsgi.so',mode=ctypes.RTLD_GLOBAL)

    uwsgi_args.insert(0, sys.argv[0])
    uwsgi_args.insert(1, '--binary-path')
    uwsgi_args.insert(2, sys.argv[0])

    # build command line args
    argv = (ctypes.c_char_p * (len(uwsgi_args)+1))()
    for pos,arg in enumerate(uwsgi_args):
        argv[pos] = arg
    # inform the uwsgi engine, the passed environ is not safe to overwrite
    envs = (ctypes.c_char_p * 1)()
    # enter into uWSGI !!!
    uwsgi.uwsgi_init(len(uwsgi_args), argv, envs)

if __name__ == "__main__":
    uwsgi_run(['--http-socket', ':9090', '--master', '--processes', '8', '--wsgi-file', sys.argv[0]])

Since this is happening after the python runtime has initialized, I presume it would work with pex!

Lo and behold:

linux ~ ❯❯❯ pex requests -o poop.pex
linux ~ ❯❯❯ ./poop.pex u.py
*** Starting uWSGI 2.0.12 (64bit) on [Sun May 15 04:03:13 2016] ***
compiled with version: 4.4.7 20120313 (Red Hat 4.4.7-3) on 15 May 2016 04:00:01
os: Linux-2.6.32-431.17.1.el6.x86_64 #1 SMP Fri Apr 11 17:27:00 EDT 2014
nodename: lcarvalh-ld
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /home/lcarvalh
detected binary path: u.py
your processes number limit is 81920
your memory page size is 4096 bytes
detected max file descriptor number: 280000
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address :9090 fd 3
Python version: 2.6.6 (r266:84292, Oct 12 2012, 14:23:48)  [GCC 4.4.6 20120305 (Red Hat 4.4.6-4)]
--- Python VM already initialized ---
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x13290a0
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 654912 bytes (639 KB) for 8 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x13290a0 pid: 3749 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 3749)
spawned uWSGI worker 1 (pid: 3756, cores: 1)
spawned uWSGI worker 2 (pid: 3757, cores: 1)
spawned uWSGI worker 3 (pid: 3758, cores: 1)
spawned uWSGI worker 4 (pid: 3759, cores: 1)
spawned uWSGI worker 5 (pid: 3760, cores: 1)
spawned uWSGI worker 6 (pid: 3761, cores: 1)
spawned uWSGI worker 7 (pid: 3762, cores: 1)
spawned uWSGI worker 8 (pid: 3763, cores: 1)
[pid: 3756|app: 0|req: 1/1] 127.0.0.1 () {24 vars in 344 bytes} [Sun May 15 04:03:15 2016] GET / => generated 188 bytes in 37 msecs (HTTP/1.1 200) 1 headers in 44 bytes (1 switches on core 0)

linux ~ ❯❯❯ http localhost:9090
HTTP/1.1 200 OK
Content-Type: text/html

requests module found: /home/lcarvalh/.pex/install/requests-2.10.0-py2.py3-none-any.whl.671f167db5ff4b6702ea8ff9465a4059d31d99fd/requests-2.10.0-py2.py3-none-any.whl/requests/__init__.pyc

I thought this was pretty cool and I wanted to take advantage of it, but right now I'm stuck on how to reliably pip install uwsgi with this environment variable exported. Check it out:

linux ~ ❯❯❯ virtualenv foobar
New python executable in foobar/bin/python
Installing setuptools............done.
Installing pip...............done.
linux ~ ❯❯❯ . foobar/bin/activate
(foobar)linux ~ ❯❯❯ export UWSGI_AS_LIB=this_is_just_a_test.so
(foobar)linux ~ ❯❯❯ pip install uwsgi
Downloading/unpacking uwsgi
  Downloading uwsgi-2.0.13.1.tar.gz (784kB): 784kB downloaded
  Running setup.py egg_info for package uwsgi
Installing collected packages: uwsgi
  Running setup.py install for uwsgi
    using profile: buildconf/default.ini
    detected include path: ['/usr/local/include', '/usr/lib/gcc/x86_64-redhat-linux/4.4.7/include', '/usr/include']
    Patching "bin_name" to properly install_scripts dir
    detected CPU cores: 8
...
... 
<build output truncated >
    ################# uWSGI configuration #################

    pcre = True
    kernel = Linux
    malloc = libc
    execinfo = False
    ifaddrs = True
    ssl = True
    zlib = True
    locking = pthread_mutex
    plugin_dir = .
    timer = timerfd
    yaml = embedded
    json = False
    filemonitor = inotify
    routing = True
    debug = False
    ucontext = True
    capabilities = True
    xml = libxml2
    event = epoll

    ############## end of uWSGI configuration #############
    total build time: 5 seconds
    *** uWSGI shared library (./this_is_just_a_test.so) is ready, move it to a library directory ***
Successfully installed uwsgi
Cleaning up...

Clearly it says it compiled a shared library:

uWSGI shared library (./this_is_just_a_test.so) is ready, move it to a library directory

However, it doesn't appear to have 😞

(foobar)linux ~ ❯❯❯ ls|grep this_
(foobar)linux ~ ❯❯❯ find foobar/ -name 'this_is_just_a_test.so'

You may be wondering what's the point? idk, just wanted to share some findings. Thanks for reading 📖

-- loren

[kwlzn]

@sixninetynine very cool, thanks for sharing.

I also found this commit: unbit/uwsgi@5a17557

that eludes to uwsgi being installed as a proper console script as part of the uwsgi package install - and importable:

    # python -i -c \
        'import uwsgi as u; u.setup().worker_id()==1 or u.run()' \
        --master --workers=1 --enable-threads --honour-stdin \
        --auto-procname --http :8000

which would make it significantly easier to integrate into a pex - and could obsolete or vastly reduce the surface area needed in something like pyuwsgi_pex altogether (which is great!).

I'll have to dig more into this later. thanks for reaching out!

[lorencarvalho]

just to close the loop on this, specifying UWSGI_AS_LIB='src/foo/submodule/libuwsgi.so', .gitignore-ing the .so and having my uwsgi_run function in src/foo/submodule/__init__.py was essentially what I ended up doing. It works great!!

the actual implementation we are using in prod is somewhat more robust, but you get the idea