uwsgi: add package with modules (currently cgi and python plugin) [RFC] by peter-stadler · Pull Request #9855 · openwrt/packages

peter-stadler · 2019-08-28T08:16:41Z

Maintainer: Ansuel Smith ansuelsmth@gmail.com
Compile tested: MIPS 74K, Asus RT-N16, master snapshot
Run tested: MIPS 74K, Asus RT-N16, master snapshot, minimal app using uwsgi-python-plugin:

cat <<EOF >foobar.py
def application(env, start_response):
    start_response('200 OK', [('Content-Type','text/html')])
    return [b"Hello World"]
EOF
uwsgi --plugin /usr/lib/uwsgi/python_plugin.so --http-socket 127.0.0.1:9031 --wsgi-file foobar.py &
wget http://127.0.0.1:9031/; rm index.html foobar.py; fg

Description: This package is a modular build of uwsgi.

It superseeds the uwsgi-cgi package by splitting off the syslog and the cgi plugin. So luci on nginx could use uwsgi-cgi-plugin instead of uwsgi-cgi, see my other PR for nginx-luci and nginx-ssl-luci.

Additional there is the python plugin that would be needed for etesync: See my other PR for etesync-server, too, that will depend on uwsgi-python-plugin.

More plugins can be enabled if another package will use them.

Edit: Right now my other PRs refer to the old version, I will update them afterwards.

Edit 2: For now, include luci-support using the syslog plugin by default, and remove the uwsgi-cgi package entirely.

peter-stadler · 2019-10-28T08:36:54Z

I did update this PR to follow the changes of uwsgi-cgi. I will update the other two PRs later.

peter-stadler · 2019-10-28T11:49:29Z

(Fixed the dependencies for uwsgi-python3-plugin: it needs python3-light not python3-base.)

peter-stadler · 2019-10-28T11:52:21Z

Should I add a CONFLICTS:=uwsgi-cgi and how to make it clear that it would supersede it?

peter-stadler · 2019-10-31T09:06:25Z

(nothing changed)

neheb · 2019-11-10T02:53:43Z

ping @Ansuel

Ansuel · 2019-11-10T14:38:39Z

it's all good but we need to first update dependency of luci or we will create package with missing dependency.
(luci on nginx depend on uwsgi-cgi and this commits renames it to uwsgi-plugins-cgi)

Ansuel · 2019-11-10T14:42:18Z

@peter-stadler i think this should be squashed and also add the drop of the uwsgi-cgi package

Ansuel · 2019-11-10T15:02:33Z

@peter-stadler by testing this i have this error
unable to find logger syslog

so i think it should be compiled with syslog support by default ?

OK NO we just need to include the syslog plugin in the emperor

peter-stadler · 2019-11-10T15:16:10Z

The syslog plugin is installed by default, in the define Package/uwsgi/install there is
$(INSTALL_BIN) $(PKG_BUILD_DIR)/syslog_plugin.so $(1)/usr/lib/uwsgi/
But, it has to be loaded by plugin = syslog in the ini file.

Should the plugin be embedded in the binary /usr/sbin/uwsgi?

Ansuel · 2019-11-10T15:19:25Z

i think yes...

I don't know why but uwsgi complains if the emperor is loaded with no request plugin (with only the syslog plugin loaded)

So yes, I think we should embed the plugin

peter-stadler · 2019-11-10T15:48:03Z

Squashed the comits and:

removed uwsgi-cgi package
included luci-support
embeded the syslog plugin into the uwsgi binary
But, the warning is still there: The emperor has no request plugin intentionally.

Do we need yaml (I did remove it for now)?

Ansuel · 2019-11-10T15:54:08Z

LGTM but we need to wait for the nginx commit to be ready

peter-stadler · 2019-11-10T17:32:46Z

Got rid of the warning by not using the syslog plugin at all, procd allows to log the output to stderr …
I made another commit for comparing it easily and would squash it if you think it would be a good idea.
Edit: I think when using logger = syslog:… there will be spawned a master for the emperor complaining about no request plugin because it checks it before becoming a emperor, which would not need a request plugin. Without master there is not much logging.
Edit 2: Btw: There is still a lot of logging output from the vassals (with and without syslog). Should we add logto = /dev/null for the luci vassals?

Ansuel · 2019-11-10T18:10:01Z

Does logto option affect also output from the application itself ?

This output would be usefull

uwsgi-emperor: luci: accepted login on / for root from 192.168.3.12

peter-stadler · 2019-11-10T18:58:52Z

Yes, it affects it, I will look into a log-route.

peter-stadler · 2019-11-10T20:41:23Z

Should work now: All messages that contain "luci:" are logged in the webui vassal :-)

Ansuel · 2019-11-10T23:39:45Z

i notice that the package gets recompiled every time...

peter-stadler · 2019-12-31T15:36:03Z

I left out the static parts that are loaded in the same way in both cases.Tested it now also with a Guest window in Chrome -> The same.

peter-stadler · 2019-12-31T15:39:51Z

Hmm, did now a reinstall of the virtual machine and it is working. I do not know what is different now, I did a clean install half an hour ago, too.

peter-stadler · 2019-12-31T15:47:08Z

Nevermind, it was in an open browser window. After opening a new incognito window, I get the same error as before.

peter-stadler · 2019-12-31T16:18:40Z

So the Nginx version is working, when using the cached files of the uhttpd version and there is one more quirk on a clean install:
opkg update && opkg install luci && opkg install luci-nginx
Then it works in a new incognito window; Nginx logs:

error.log: bind() to 0.0.0.0:80 failed (98: Address in use)
access.log: GET /cgi-bin/luci/ -> 403
access.log: … (static files)
error.log: "/www/ubus/index.html" is not found
access.log: GET /ubus/?1577805659919 -> 404
access.log: … (static files)
access.log: POST /cgi-bin/luci/admin/ubus?1577805659971 -> 200

It still works after opening a new incognito window; Nginx logs additionally:

access.log: GET /cgi-bin/luci/ -> 403
access.log: … (static files)
error.log: "/www/ubus/index.html" is not found
access.log: GET /ubus/?1577805659919 -> 404
access.log: … (static files)
access.log: POST /cgi-bin/luci/admin/ubus?1577805659971 -> 200

It fails when opening a new incognito window after restarting Nginx; It adds the following lines to the access.log (nothing to the error.log):

GET /cgi-bin/luci/ -> 403
… (static files)
GET /ubus/?1577805288828 -> 400
… (static files)
POST /ubus/?1577805288966 -> 200
POST /ubus/?1577805288978 -> 200

peter-stadler · 2019-12-31T16:39:57Z

We could get a similar log by using location /cgi-bin/luci/admin/ubus in /etc/nginx/luci_uwsgi.conf. Then the difference is only in the size of the last response (to POST /cgi-bin/luci/admin/ubus?1577805659971 -> 200); it is 79 and should be 235.

I will leave it for this year and wish you a good start into the new year.

Ansuel · 2019-12-31T16:46:01Z

So the Nginx version is working, when using the cached files of the uhttpd version and there is one more quirk on a clean install:
opkg update && opkg install luci && opkg install luci-nginx
Then it works in a new incognito window; Nginx logs:
error.log: bind() to 0.0.0.0:80 failed (98: Address in use)
access.log: GET /cgi-bin/luci/ -> 403
access.log: … (static files)
error.log: "/www/ubus/index.html" is not found
access.log: GET /ubus/?1577805659919 -> 404
access.log: … (static files)
access.log: POST /cgi-bin/luci/admin/ubus?1577805659971 -> 200
It still works after opening a new incognito window; Nginx logs additionally:
access.log: GET /cgi-bin/luci/ -> 403
access.log: … (static files)
error.log: "/www/ubus/index.html" is not found
access.log: GET /ubus/?1577805659919 -> 404
access.log: … (static files)
access.log: POST /cgi-bin/luci/admin/ubus?1577805659971 -> 200
It fails when opening a new incognito window after restarting Nginx; It adds the following lines to the access.log (nothing to the error.log):
GET /cgi-bin/luci/ -> 403
… (static files)
GET /ubus/?1577805288828 -> 400
… (static files)
POST /ubus/?1577805288966 -> 200
POST /ubus/?1577805288978 -> 200

i can't get where is the problem in the last log...

the error 400 is intended (used by luci to check if /ubus can be used)
/cgi-bin/luci/admin/ubus
/ubus use the uhttpd ubus integrated module or the nginx ubus integrated module

is just the lua page that use the ubus lua lib

peter-stadler · 2019-12-31T17:24:48Z

It hangs with the message Session expired in the browser window. The problem is not shown in the log, maybe if the ubus module in Nginx would be more verbose … Nevertheless it is working if we omit the whole location /ubus section :-)

Ansuel · 2020-01-01T13:12:42Z

I tested with latest commit and nginx + /ubus works correctly o.O

Ansuel · 2020-01-01T13:38:24Z

Anyway there is a dependency recursion problem... make the uwsgi package dependent of the sub package
(+uwsgi -> uwsgi)

peter-stadler · 2020-01-01T14:21:08Z

I still have the problem. Can you reproduce it?

Start x86_64.img in qemu:

#!/bin/sh

IMG="openwrt-x86-64-combined-ext4.img"
QEMU="qemu-system-x86_64 -enable-kvm -nographic -m 1G -M q35 \
-drive file=$IMG,id=d0,if=none,bus=0,unit=0 -device ide-hd,drive=d0,bus=ide.0"
MSG="NB: The network needs a minute! tests: ifconfig; nslookup openwrt.org"
if [ $UID -gt 0 ] 
then
    echo To enable LAN and ssh from host run it as root: sudo $0
    $QEMU
else 
    echo Started as root, enabling LAN, connect to it via: ssh root@192.168.1.1
    echo $MSG
    LAN=openwrtx64tap0
    # create tap interface which will be connected to OpenWrt LAN NIC
    ip tuntap add mode tap $LAN && \
    ip link set dev $LAN up && \
    # configure interface with static ip to avoid overlapping routes                         
    ip addr add 192.168.1.123/24 dev $LAN && \
    $QEMU \
        -device virtio-net-pci,netdev=lan \
        -netdev tap,id=lan,ifname=$LAN,script=no,downscript=no \
        -device virtio-net-pci,netdev=wan \
        -netdev user,id=wan 
    # cleanup. delete tap interface created earlier
    ip addr flush dev $LAN
    ip link set dev $LAN down
    ip tuntap del mode tap dev $LAN
fi

Run opkg update && opkg install luci-nginx && service nginx restart in it.
Open a incognito window at http://192.168.1.1/cgi-bin/luci/

I get a message 'Session expired' that repeats when clicking on To login... (the same in Firefox).
Edit: It works before restarting Nginx but then I think it does not use its ubus module.

peter-stadler · 2020-01-01T14:45:05Z

I did fix the dependencies, changing also the DEPENDS for nginx-mod-luci/default. Thank you :-)

Ansuel · 2020-01-01T15:22:03Z

made a PR. #10923
This is ready to merge.

peter-stadler · 2020-01-01T16:42:58Z

Ah, I see. Sorry I forgot about that change. It is working.
I rebased this PR, tested it and consider it ready to merge :-)

Ansuel · 2020-01-01T16:46:02Z

mh can you remove the nginx changes ?

peter-stadler · 2020-01-01T16:52:53Z

Done that, I will make another small PR for Nginx.

Ansuel · 2020-01-01T17:10:19Z

do we really need that?

peter-stadler · 2020-01-01T17:14:37Z

I think else it will not be shown in make menuconfig. Or how should I change the uwsgi DEPENDS?

peter-stadler · 2020-01-01T17:20:08Z

The draft would be at #10925

Ansuel · 2020-01-01T17:22:16Z

Tested now and yes we need uwsgi in the nginx support package... as we need to change it anyway... At this point better make a mini pr also for nginx and rename the support package here to a better name. Sorry for all the mess.

After this and the mini pr get merged we can start working on nginx draft pr. Is it good for you?

peter-stadler · 2020-01-01T17:25:47Z

This is perfect :-)

Provide the minimal applications and plugins for: cgi, filelog, syslog and python3. More plugins can be added if needed by other packages. Autostart uwsgi in emperor mode loading vassals on demand. For now, include luci-support (maybe it will be moved to another package), which uses the syslog plugin by default. Signed-off-by: Peter Stadler <peter.stadler@student.uibk.ac.at>

peter-stadler · 2020-01-01T17:29:37Z

The above draft PR is a mini PR that can be merged after this one. I changed the name to uwsgi-luci-support ...

peter-stadler · 2020-01-01T17:38:08Z

I could also update the Nginx version later on if you think that would be better. And if you have another name for the uwsgi-luci-support I would change that, too ...

Ansuel · 2020-01-01T17:39:04Z

@hnyman good to merge
@peter-stadler convert this to a pr #10925 so it can be merged

peter-stadler changed the title ~~uwsgi: add package with modules (currently cgi and python plugin)~~ uwsgi: add package with modules (currently cgi and python plugin) [RFC] Aug 29, 2019

This was referenced Aug 29, 2019

etesync-server: add package and configuration for running with uwsgi on nginx [RFC] #9865

Merged

[RFC] luci-nginx and luci-ssl-nginx: collect files from nginx and uwsgi-cgi packages openwrt/luci#2981

Closed

peter-stadler mentioned this pull request Oct 10, 2019

uwsgi-cgi: adds patch to fix #10134 #10173

Merged

peter-stadler force-pushed the uwsgi branch from 81bc491 to 521cf4d Compare October 28, 2019 08:28

peter-stadler marked this pull request as ready for review October 28, 2019 08:37

peter-stadler force-pushed the uwsgi branch from 521cf4d to d45f249 Compare October 28, 2019 11:48

peter-stadler force-pushed the uwsgi branch from d45f249 to 773dce8 Compare October 31, 2019 09:02

peter-stadler mentioned this pull request Nov 5, 2019

django-etesync-journal: add python3 package #10469

Merged

peter-stadler force-pushed the uwsgi branch from 773dce8 to 69f550c Compare November 10, 2019 15:48

peter-stadler force-pushed the uwsgi branch from f78e022 to b37118a Compare November 10, 2019 17:30

peter-stadler force-pushed the uwsgi branch 2 times, most recently from be56313 to 53b14fd Compare November 10, 2019 20:16

peter-stadler force-pushed the uwsgi branch from c034fd2 to f76e1e9 Compare January 1, 2020 14:44

peter-stadler force-pushed the uwsgi branch from f76e1e9 to e0de548 Compare January 1, 2020 16:41

peter-stadler force-pushed the uwsgi branch from e0de548 to cf6a4d5 Compare January 1, 2020 16:51

peter-stadler force-pushed the uwsgi branch from cf6a4d5 to 96d1910 Compare January 1, 2020 17:29

hnyman merged commit e020c10 into openwrt:master Jan 1, 2020

peter-stadler deleted the uwsgi branch January 3, 2020 09:57

Conversation

peter-stadler commented Aug 28, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

peter-stadler commented Oct 28, 2019

Uh oh!

peter-stadler commented Oct 28, 2019

Uh oh!

peter-stadler commented Oct 28, 2019

Uh oh!

peter-stadler commented Oct 31, 2019

Uh oh!

neheb commented Nov 10, 2019

Uh oh!

Ansuel commented Nov 10, 2019

Uh oh!

Ansuel commented Nov 10, 2019

Uh oh!

Ansuel commented Nov 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

peter-stadler commented Nov 10, 2019

Uh oh!

Ansuel commented Nov 10, 2019

Uh oh!

peter-stadler commented Nov 10, 2019

Uh oh!

Ansuel commented Nov 10, 2019

Uh oh!

peter-stadler commented Nov 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ansuel commented Nov 10, 2019

Uh oh!

peter-stadler commented Nov 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

peter-stadler commented Nov 10, 2019

Uh oh!

Ansuel commented Nov 10, 2019

Uh oh!

peter-stadler commented Dec 31, 2019

Uh oh!

peter-stadler commented Dec 31, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

peter-stadler commented Dec 31, 2019

Uh oh!

peter-stadler commented Dec 31, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

peter-stadler commented Dec 31, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ansuel commented Dec 31, 2019

Uh oh!

peter-stadler commented Dec 31, 2019

Uh oh!

Ansuel commented Jan 1, 2020

Uh oh!

Ansuel commented Jan 1, 2020

Uh oh!

peter-stadler commented Jan 1, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

peter-stadler commented Jan 1, 2020

Uh oh!

Ansuel commented Jan 1, 2020

Uh oh!

peter-stadler commented Jan 1, 2020

Uh oh!

Ansuel commented Jan 1, 2020

Uh oh!

peter-stadler commented Jan 1, 2020

Uh oh!

Ansuel commented Jan 1, 2020

Uh oh!

peter-stadler commented Jan 1, 2020

Uh oh!

peter-stadler commented Jan 1, 2020

peter-stadler commented Aug 28, 2019 •

edited

Loading

Ansuel commented Nov 10, 2019 •

edited

Loading

peter-stadler commented Nov 10, 2019 •

edited

Loading

peter-stadler commented Nov 10, 2019 •

edited

Loading

peter-stadler commented Dec 31, 2019 •

edited

Loading

peter-stadler commented Dec 31, 2019 •

edited

Loading

peter-stadler commented Dec 31, 2019 •

edited

Loading

peter-stadler commented Jan 1, 2020 •

edited

Loading