Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gunicorn plumbing #357

Merged
merged 8 commits into from
Sep 11, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 23 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,4 +16,26 @@ COPY . /src/ngsi-timeseries-api/
WORKDIR /src/ngsi-timeseries-api/src
ENV PYTHONPATH=$PWD:$PYTHONPATH

CMD python app.py
EXPOSE 8668
ENTRYPOINT ["python", "app.py"]
# NOTE.
# The above is basically the same as running:
#
# gunicorn server.wsgi --config server/gconfig.py
#
# You can also pass any valid Gunicorn option as container command arguments
# to add or override options in server/gconfig.py---see `server.grunner` for
# the details.
# In particular, a convenient way to reconfigure Gunicorn is to mount a config
# file on the container and then run the container with the following option
#
# --config /path/to/where/you/mounted/your/gunicorn.conf.py
#
# as in the below example
#
# $ echo 'workers = 2' > gunicorn.conf.py
# $ docker run -it --rm \
# -p 8668:8668 \
# -v $(pwd)/gunicorn.conf.py:/gunicorn.conf.py
# smartsdk/quantumleap --config /gunicorn.conf.py
#
1 change: 1 addition & 0 deletions Pipfile
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ flask = "~=1.1.0"
geocoder = "~=1.33"
geojson = "~=2.4"
geomet = "~=0.2"
gunicorn = "~=20.0.4"
influxdb = "~=4.0"
pg8000 = ">=1.15"
pymongo = "~=3.4"
Expand Down
262 changes: 136 additions & 126 deletions Pipfile.lock

Large diffs are not rendered by default.

21 changes: 12 additions & 9 deletions src/app.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
from utils.hosts import LOCAL
import server.wsgi as flask
import server.grunner as gunicorn
from utils.cfgreader import EnvReader, BoolVar


def use_flask() -> bool:
env_var = BoolVar('USE_FLASK', False)
return EnvReader().safe_read(env_var)


if __name__ == '__main__':
import connexion
app = connexion.FlaskApp(__name__, specification_dir='../specification/')
app.add_api('quantumleap.yml',
arguments={'title': 'QuantumLeap V2 API'},
pythonic_params=True,
# validate_responses=True, strict_validation=True
)
app.run(host=LOCAL, port=8668)
if use_flask(): # dev mode, run the WSGI app in Flask dev server
flask.run()
else: # prod mode, run the WSGI app in Gunicorn
gunicorn.run()
2 changes: 2 additions & 0 deletions src/geocoding/tests/test_geocoding.py
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,8 @@ def test_entity_add_street_line(air_quality_observed):
assert len(geo['coordinates']) > 1


# TODO: see #358
@pytest.mark.skip(reason="see #358")
def test_entity_add_city_shape(air_quality_observed):
air_quality_observed.pop('location')

Expand Down
2 changes: 1 addition & 1 deletion src/reporter/tests/run_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
docker build -t smartsdk/quantumleap ../../../

docker-compose up -d
sleep 12
sleep 20

cd ../../../
pytest src/reporter/ --cov-report= --cov-config=.coveragerc --cov-append --cov=src/
Expand Down
3 changes: 3 additions & 0 deletions src/server/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@

DEFAULT_HOST = '0.0.0.0' # bind to all available network interfaces
DEFAULT_PORT = 8668
74 changes: 74 additions & 0 deletions src/server/gconfig.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
#
# Gunicorn settings to run QuantumLeap.
# To make configuration more manageable, we keep all our Gunicorn settings
# in this file and start Gunincorn from the Docker container without any
# command line args except for the app module and the path to this config
# file, e.g.
#
# gunicorn server.wsgi --config server/gconfig.py
#
# Settings spec:
# - https://docs.gunicorn.org/en/stable/settings.html
#

import multiprocessing

import server


#
# Server config section.
#

bind = f"{server.DEFAULT_HOST}:{server.DEFAULT_PORT}"


#
# Worker processes config section.
# Read: https://docs.gunicorn.org/en/latest/design.html
#


# Number of worker processes for handling requests.
# We set it to the max Gunicorn recommends.
workers = multiprocessing.cpu_count() * 4 + 1

# QuantumLeap does alot of network IO, so we configure worker processes
# to use multi-threading (`gthread`) to improve performance. With this
# setting, each request gets handled in its own thread taken from a
# thread pool.
# In our tests, the `gthread` worker type had better throughput and
# latency than `gevent` but `gevent` used up less memory, most likely
# because of the difference in actual OS threads. So for now we go with
# `gthread` and a low number of threads. This has the advantage of better
# performance, reasonable memory consumption, and keeps us from accidentally
# falling into the `gevent` monkey patching rabbit hole. Also notice that
# according to Gunicorn docs, when using `gevent`, Psycopg (Timescale
# driver) needs psycogreen properly configured to take full advantage
# of async IO. (Not sure what to do for the Crate driver!)
worker_class = 'gthread'

# The size of each process's thread pool.
# So here's the surprise. In our tests, w/r/t to throughput `gthread`
# outperformed `gevent`---27% better. Latency was pretty much the same
# though. But the funny thing is that we used exactly the same number
# of worker processes and the default number of threads per process,
# which is, wait for it, 1. Yes, 1.
#
# TODO: proper benchmarking.
# We did some initial quick & dirty benchmarking to get these results.
# We'll likely have to measure better and also understand better the
# way the various Gunicorn worker types actually work. (Pun intended.)
threads = 1


#
# Logging config section.
#

loglevel = 'debug'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this logging for gunicorn or also for the app?

Copy link
Member Author

@c0c0n3 c0c0n3 Sep 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's Gunicorn own log level, it doesn't affect QuantumLeap. That's the log level you had in #354 and I suggest we keep it that way until we get to know Gunicorn better :-)



# TODO: other settings.
# Review gunicorn default settings with an eye on security and performance.
# We might need to set more options than the above.
81 changes: 81 additions & 0 deletions src/server/grunner.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
import sys
from typing import Any, Dict

from gunicorn.app.base import Application
from gunicorn.config import make_settings

import server.gconfig
from server.wsgi import application


def quantumleap_base_config() -> Dict[str, Any]:
"""
Read the base QuantumLeap configuration from the ``server.gconfig``
module.

:return: the dictionary with the base QuantumLeap settings.
"""
gunicorn_setting_names = [k for k, v in make_settings().items()]
server_config_vars = vars(server.gconfig).items()
return {
k: v
for k, v in server_config_vars if k in gunicorn_setting_names
}


class GuantumLeap(Application):
"""
Gunicorn server runner.

This class is a fully-fledged Gunicorn server WSGI runner, just like
``WSGIApplication`` from ``gunicorn.app.wsgiapp``, except the WSGI
app to run is fixed (QuantumLeap) and the content of ``server.gconfig``
is used as initial configuration. Notice you can override these base
config settings using CLI args or/and a config file as you'd normally
do with Gunicorn, but you can't run any WSGI app other than QuantumLeap.
"""

def init(self, parser, opts, args):
return quantumleap_base_config()

def load(self):
return application


def run():
"""
Start a fully-fledged Gunicorn server to run QuantumLeap.

If you pass no CLI args, this is the same as running

``$ gunicorn server.wsgi --config server/gconfig.py``

which starts Gunicorn to run the QuantumLeap WSGI Flask app with
the Gunicorn settings in our ``server.gconfig`` module.
You can specify any valid Gunicorn CLI option and it'll take
precedence over any setting with the same name in ``server.gconfig``.
Also you can specify a different config module and options in that
module will override those with the same name in ``server.gconfig``.
The only thing you won't be able to change is the WSGI app to run,
QuantumLeap.
"""
gunicorn = GuantumLeap('%(prog)s [OPTIONS] [APP_MODULE]') # (1)
sys.argv[0] = 'quantumleap' # (2)
sys.exit(gunicorn.run()) # (2)

# NOTE.
# 1. We keep the same usage as in `gunicorn.app.wsgiapp.run`.
# 2. We basically start Gunicorn in the same way as the Gunicorn launcher
# would:
#
# $ cat $(which gunicorn)
#
# #!/Users/andrea/.local/share/virtualenvs/ngsi-timeseries-api-MeJ80LMF/bin/python
# # -*- coding: utf-8 -*-
# import re
# import sys
# from gunicorn.app.wsgiapp import run
# if __name__ == '__main__':
# sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
# sys.exit(run())
#
54 changes: 54 additions & 0 deletions src/server/wsgi.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
from connexion import FlaskApp

import server


SPEC_DIR = '../../specification/'
SPEC = 'quantumleap.yml'


def new_wrapper() -> FlaskApp:
"""
Factory function to build a Connexion wrapper to manage the Flask
application in which QuantumLeap runs.

:return: the Connexion wrapper.
"""
wrapper = FlaskApp(__name__,
specification_dir=SPEC_DIR)
wrapper.add_api(SPEC,
arguments={'title': 'QuantumLeap V2 API'},
pythonic_params=True,
# validate_responses=True, strict_validation=True
)
return wrapper


quantumleap = new_wrapper()
"""
Singleton Connexion wrapper that manages the QuantumLeap Flask app.
"""

application = quantumleap.app
"""
The WSGI callable to run QuantumLeap in a WSGI container of your choice,
e.g. Gunicorn, uWSGI.
Notice that Gunicorn will look for a WSGI callable named `application` if
no variable name follows the module name given on the command line. So one
way to run QuantumLeap in Gunicorn would be

gunicorn server.wsgi --config server/gconfig.py

An even more convenient way is to use our Gunicorn standalone server,
see `server.grunner` module.
"""


def run():
"""
Runs the bare-bones QuantumLeap WSGI app.
Notice the app will run in the Flask dev server, so it's only good
for development, not prod!
"""
quantumleap.run(host=server.DEFAULT_HOST,
port=server.DEFAULT_PORT)
6 changes: 4 additions & 2 deletions src/tests/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ services:
- "27017:27017"

quantumleap:
image: ${QL_IMAGE:-smartsdk/quantumleap}
image: ${QL_IMAGE:-smartsdk/quantumleap:latest}
sysctls:
net.core.somaxconn: 4096
ports:
- "8668:8668"
depends_on:
Expand All @@ -33,7 +35,7 @@ services:
- USE_GEOCODING=True
- REDIS_HOST=redis
- REDIS_PORT=6379
- LOGLEVEL=DEBUG
- LOGLEVEL=ERROR

crate:
image: crate:${CRATE_VERSION:-4.1.4}
Expand Down
49 changes: 49 additions & 0 deletions src/tests/notify-load-test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
import http from 'k6/http';
import { check, sleep } from 'k6';

export default function() {
var url = 'http://192.0.0.1:8668/v2/notify';
const before = new Date().getTime();
const T = 30; // time needed to complete a VU iteration


for (var i = 0; i < 100; i++){
var data = {
"id": "Room:1",
"type": "Room",
"temperature": {
"value": 23,
"type": "Float"
},
"pressure": {
"value": 720,
"type": "Integer"
}
}
var array = [];
array.push(data);

var payload = {
"data" : array
}
var payload = JSON.stringify(payload);

var params = {
headers: {
'Content-Type': 'application/json',
}
};
let res = http.post(url, payload, params);
check(res, { 'status was 200': r => r.status == 200 });
}
const after = new Date().getTime();
const diff = (after - before) / 1000;
const remainder = T - diff;
if (remainder > 0) {
sleep(remainder);
} else {
console.warn(
`Timer exhausted! The execution time of the test took longer than ${T} seconds`
);
}
}
18 changes: 18 additions & 0 deletions src/tests/run_load_tests.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#!/usr/bin/env bash

docker build --cache-from smartsdk/quantumleap -t smartsdk/quantumleap ../../

docker-compose up -d
docker-compose stop orion
docker-compose stop mongo
sleep 10

docker run -i --rm loadimpact/k6 run --vus 10 --duration 60s - < notify-load-test.js

sleep 10

docker run -i --rm loadimpact/k6 run --vus 100 --duration 120s - < notify-load-test.js

sleep 10

docker-compose down
2 changes: 1 addition & 1 deletion src/tests/run_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ docker run -ti --rm --network tests_default \
# Restart QL on development version and CRATE on current version
docker-compose stop quantumleap
CRATE_VERSION=${CRATE_VERSION} QL_IMAGE=smartsdk/quantumleap docker-compose up -d
sleep 30
sleep 40

# Backwards Compatibility Test
cd ../../
Expand Down
1 change: 0 additions & 1 deletion src/utils/hosts.py

This file was deleted.