(You can view this graph as time-for-50,000-requests here).
The benchmark results on a quad-core i7-8565 @1.8GHz are as follows, where 8081 is port serving OpenResty's Lua and 8082 is PUC Lua via FastCGI:
Benchmarking Redbean Lua 5.4
wrk2/wrk -d2 -c10 -t10 -R 1000000 "http://localhost:8085/multiply?a=2&b=3"
Requests/sec: 177811.23
Benchmarking openresty LuaJIT
wrk2/wrk -d2 -c10 -t10 -R 1000000 "http://localhost:8081/multiply?a=2&b=3"
Requests/sec: 113018.85
Benchmarking nginx-lws (Lua Web Services)
wrk2/wrk -d2 -c10 -t10 -R 1000000 "http://localhost:8084/multiply?a=2&b=3"
Requests/sec: 84172.28
Benchmarking apache mod-lua
wrk2/wrk -d2 -c10 -t10 -R 1000000 "http://localhost:8080/multiply?a=2&b=3"
Requests/sec: 51086.05
Benchmarking FastCGI Lua 5.4
wrk2/wrk -d2 -c10 -t10 -R 1000000 "http://localhost:8082/multiply?a=2&b=3"
Requests/sec: 18427.28
Benchmarking uwsgi/lua5.1
wrk2/wrk -d2 -c10 -t10 -R 1000000 "http://localhost:8083/multiply?a=2&b=3"
Requests/sec: 30894.79
Benchmarking uwsgi/lua5.4
wrk2/wrk -d2 -c10 -t10 -R 1000000 "http://localhost:8083/multiply?a=2&b=3"
Requests/sec: 30727.90
Benchmarking uwsgi/luajit
wrk2/wrk -d2 -c10 -t10 -R 1000000 "http://localhost:8083/multiply?a=2&b=3"
Requests/sec: 30579.44In short, LWS is as good as OpenResty for raw speed of requests. Since our Lua program is so small and simple, it makes no difference whether we use Lua 5.1, Lua 5.4 or LuaJIT:
- 1.6×: Redbean - impressive Lua server, but less generic (e.g. doesn't support Python apps)
- 1×: OpenResty - fastest general purpose server
- 0.8×: NGINX-LWS - close runner up
- 0.4×: Apache
- 0.3×: uWSGI - with WSAPI protocol
- 0.3×: FastCGI protocol
Notes:
- The uWSGI and FASTCGI options are slow because they use a protocol to serialize commands sent to Lua.
- It's possible that there is a way to double the speed of my FastCGI and WSAPI benchmarks, because those benchmarks only use about half of each CPU core (according to htop), whereas the other benchmarks use almost 100% of every core. I don't know why NGINX doesn't parallel those up sufficiently to use 100% CPU. There may be a better server config, but I've tried various ones and I can't find it.
- Not many benchmark tools are fast enough to produce good results. I have found only 3 that are fastest and produce very similar results:
wrk2,weighttp,httpress.
-
Install Lua 5.1 and 5.4 (for Lua version comparison).
-
Perform these instructions to install LuaRocks (so that you can get wsapi-fcgi below).
-
Install OpenResty and also standard NGINX
- This will install NGINX twice: one invoked as
openrestyand the other invoked withnginx. - Most of the benchmarks are performed with openresty, but standard nginx is also necessary to benchmark
nginx-lws, which doesn't work with OpenResty out of the box.
- This will install NGINX twice: one invoked as
-
Fetch and build
nginx-lwsagainst the Nginx source using:
make fetch buildNext install what we need for our specific benchmarks. These instructions work on Ubuntu (otherwise see troubleshooting note on missing packages below):
apt install apache2-utils # supplies ab, the apache benchmark tool
apt install libfcgi-dev # supplies fcgi_stdio.h
sudo luarocks install wsapi-fcgi
apt install lua5.1 liblua5.1-dev # required to build uWSGI with lua5.1 support
apt install lua5.4 liblua5.4-dev # required to build uWSGI with lua5.4 support
apt install luajit libluajit-5.1-dev # required to build uWSGI with luajit support
apt install apache2 # needed for benchmark with apache and LuaStart your nginx and fcgi servers:
make startTest that nginx is working:
$ make test
Testing resty server
curl -fsS "http://localhost:8081/multiply?a=2&b=3"
<html><body><p>Hello NGINX-Lua!</p><p>PATH=/multiply</p><p>RESULT: 2*3=6</p></body></html>
Testing fcgi server
curl -fsS "http://localhost:8082/multiply?a=2&b=3"
<html><body><p>Hello WSAPI!</p><p>PATH=/multiply</p><p>RESULT: 2*3=6</p></body></html>
...Run the benchmarks:
make summary # or try make benchmarks for more info
make stop # when you're done testing, stop the servers make startedPeruse the Makefile for other useful make targets if you want to test specific things.
If you want to experiment with different benchmark tools, I have left builders in the Makefile for weighttp and httpress which produce almost identical results. But you'll need:
apt install libev-dev libhttp-parser-dev libuchardet-dev gnutls-devPlease note that although uWSGI is used as a benchmark comparison point, it is already in maintenance mode and building it already has deprecation warnings for use of old ssl and python distutils functions. Because of this, it requires python < 3.12 and it is not clear what version of libssl-dev will drop support.
If your OS does have the specified Lua packages, you may need to build them from source. In that case, you will need to change the Makefile to specify locations to them. See uWSGI notes on using Lua.
If you get this error when you run make benchmark then the benchmarking is trying to make more simultaneous requests than your user allows. Check the number of requests your user is allowed as follows:
$ ulimit -Hn
1048576
$ ulimit -Sn
1024These numbers should be significantly greater than the -c<connections> parameter in the ab command run by make benchmark. If not, see how to increase your open file limit here or here for Ubuntu.