From 2d820c2371b2b64bd8318ed2df6960943a2481a1 Mon Sep 17 00:00:00 2001 From: Stefan Verhoeven Date: Mon, 9 Mar 2020 10:14:16 +0100 Subject: [PATCH 01/11] Added link to wikipedia article about algorithm --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index b9b74f2..7741ffb 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ - [Visualization](#visualization) Document describing a way that a scientist with a C++ algorithm can make it available as a web application. -Bubble algoritm will be the use case. +The [Newton raphson root finding algorithm](https://en.wikipedia.org/wiki/Newton%27s_method) will be the use case. ```{.cpp file=newtonraphson.hpp} #ifndef H_NEWTONRAPHSON_H @@ -112,11 +112,11 @@ A C++ algorithm is a collection of functions/classes that can perform a mathemat The web application is a set of web pages with a form to fill the input required for the algorithm, a submit button that will start the execution and a page that shows the output of the algorithm. The output should be presented in a usable format like a table, chart/plot and download. -The Bubble has the following characteristics: +The C++ code has the following characteristics: -- It is a collection of C++ algorithms which can be called as functions in a C++ library or command line executables. +- A C++ algorithm which can be called as function in a C++ library or command line executable - The input and output files of the command line executables adhere to a JSON schema -- Uses cmake as build tool +- Uses Makefile as build tool - Copies of C++ dependencies are in the git repository ## JSON schema From 1fbae0d20e47e27292e6f34e6789ea3a11df5789 Mon Sep 17 00:00:00 2001 From: Stefan Verhoeven Date: Mon, 9 Mar 2020 13:40:54 +0100 Subject: [PATCH 02/11] Added Flask example and Flask-Celery example --- INSTALL.md | 2 +- README.md | 205 ++++++++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 195 insertions(+), 12 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 33d8e29..86f942e 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -5,7 +5,7 @@ To run the commands in the README.md the following items are required 1. Apache httpd server 2.4 -1. Python libraries `pip install flask pybind11 celery connexion` +1. Python libraries `pip install flask pybind11 celery redis connexion` 1. Python devel with `sudo apt install python3-dev` 1. Emscriptem diff --git a/README.md b/README.md index 7741ffb..612eae8 100644 --- a/README.md +++ b/README.md @@ -307,9 +307,9 @@ print(root) ### Web application Now that the C++ functions can be called from Python it is time to call the function from a web page. -To assist in making a web application a web framework needs to be picked. For Bubble [flask](https://flask.palletsprojects.com/) was chosen as it minimalistic and has a large active community. +To assist in making a web application a web framework needs to be picked. The [Flask](https://flask.palletsprojects.com/) web framework was chosen as it minimalistic and has a large active community. -The Bubble web application has 3 kinds of pages: +The web application has 3 kinds of pages: 1. a page with form and submit button, 2. a page to show the progress of the calculation @@ -317,18 +317,25 @@ The Bubble web application has 3 kinds of pages: Each page is available on a different url. In flask the way urls are mapped to Python function is done by adding a route decorator to the function for example: -```python +```{.python file=src/py/hello.py} from flask import Flask app = Flask(__name__) @app.route("/") -def newtonraphson(): +def hello(): return "Hello World!" + +app.run() +``` + +Run with +```{.awk #py-hello} +python src/py/hello.py ``` The above route will just return the string "Hello World!" in the web browser when visiting [http://localhost:5000/](http://localhost:5000/). It is possible to return a html page aswell, but to make it dynamic it soon becomes a mess of string concatenations. Template languages help to avoid the concatination mess. Flask is configured with the [Jinja2](https://jinja.palletsprojects.com/). A template for the above route could look like: -```jinja +```{.html file=src/py/templates/hello.html} Hello from Flask {% if name %} @@ -340,19 +347,195 @@ The above route will just return the string "Hello World!" in the web browser wh and to render the template the function would look like: -```python -from flask import render_template +```{.python file=src/py/hello-templated.py} +from flask import Flask, render_template -@app.route("/") -def newtonraphson(): - return render_template('newtonraphson.html') +app = Flask(__name__) + +@app.route('/hello/') +def hello_name(name=None): + return render_template('hello.html', name=name) + +app.run() ``` Where `name` is a variable which gets combined with template to render into a html page. +The web application can be started with +```{.awk #py-hello-templated} +python src/py/hello-templated.py +``` + +In a web browser you can visit http://localhost:5000/hello/yourname to the web application. + +Let's make the web application for our Newton raphson algorithm. + +The first thing we want is the web page with the form, the template that renders the form looks like + +```{.html file=src/py/templates/form.html} + +
+ + + + + +
+``` + +The home page will render the form like so + +```{.python #py-form} +@app.route('/', methods=['GET']) +def form(): + return render_template('form.html') +``` + +The result will be displayed on a html page with the following template + +```{.html file=src/py/templates/result.html} + +

With epsilon of {{ epsilon }} and a guess of {{ guess }} the found root is {{ root }}.

+``` + +The form will be submitted to the '/' path with the POST method. In the handler of this route we want to perform the calculation and return the result html page. + +```{.python #py-calculate} +@app.route('/', methods=['POST']) +def calculate(): + epsilon = float(request.form['epsilon']) + guess = float(request.form['guess']) + + from newtonraphsonpy import NewtonRaphson + finder = NewtonRaphson(epsilon) + root = finder.find(guess) + + return render_template('result.html', epsilon=epsilon, guess=guess, root=root) +``` + +Putting it all together in + +```{.python file=src/py/webapp.py} +from flask import Flask, render_template, request +app = Flask(__name__) + +<> + +<> + +app.run() +``` + +And running it with + +```{.awk #py-webapp} +PYTHONPATH=$PWD python src/py/webapp.py +``` + +To test we can visit http://localhost:5000 fill the form and press submit to get the result. + When performing a long calculation (more than 30 seconds), the end-user requires feedback of the progress. In a normal request/response cycle, feedback is only returned in the response. To give feedback during the calculation, the computation must be offloaded to a task queue. In Python the most used task queue is [celery](http://www.celeryproject.org/). While the calculation is running on some worker it is possible to have a progress page which can check in the queue what the progress is of the calculation. -> TODO point celery example +Celery needs a broker to use a queue and store the results. +Will use [redis](https://redis.io/) in a Docker container as Celery broker. + +```{.awk #run-redis} +docker run -d -p 6379:6379 redis +``` + +Let's setup a method that can be submitted to the Celery task queue. +First configure Celery to use the Redis database. + +```{.python #celery-config} +from celery import Celery +capp = Celery('tasks', broker='redis://localhost:6379', backend='redis://localhost:6379') +``` + +When a method is decorated with the Celery task decorator then it can be submitted to the Celery task queue. +Will add some sleeps to demonstrate what would happen with a long running calculation. Will also tell Celery about in which step the calculation is, later we can display this step to the user. + +```{.python file=src/py/tasks.py} +import time + +<< celery-config>> + +@capp.task(bind=True) +def calculate(self, epsilon, guess): + if not self.request.called_directly: + self.update_state(state='INITIALIZING') + time.sleep(5) + from newtonraphsonpy import NewtonRaphson + finder = NewtonRaphson(epsilon) + if not self.request.called_directly: + self.update_state(state='FINDING') + time.sleep(5) + root = finder.find(guess) + return {'root': root, 'guess': guess, 'epsilon':epsilon} +``` + +Instead of running the calculation when the submit button is pressed. +We will submit the calculation task to the task queue by using the `.delay()` function. +The submission will return a job identifier we can use later to get the status and result of the job. The web browser will redirect to a url with the job identifier in it. + +```{.python #py-submit} +@app.route('/', methods=['POST']) +def submit(): + epsilon = float(request.form['epsilon']) + guess = float(request.form['guess']) + from tasks import calculate + job = calculate.delay(epsilon, guess) + return redirect(url_for('result', jobid=job.id)) +``` + +The last method is to ask the Celery task queue what the status is of the job and return the result when it is succesfull. + +```{.python #py-result} +@app.route('/result/') +def result(jobid): + from tasks import capp + job = capp.AsyncResult(jobid) + job.maybe_throw() + if job.successful(): + result = job.get() + return render_template('result.html', epsilon=result['epsilon'], guess=result['guess'], root=result['root']) + else: + return job.status +``` + +Putting it all together + +```{.python file=src/py/awebapp.py} +from flask import Flask, render_template, request, redirect, url_for + +app = Flask(__name__) + +<> + +<> + +<> + +if __name__ == '__main__': + app.run() +``` + +Start the web application like before with + +```{.awk #py-awebapp} +PYTHONPATH=$PWD python src/py/awebapp.py +``` + +Tasks will be run by the Celery worker. The worker can be started with + +```{.awk #py-awebapp} +cd src/py +PYTHONPATH=$PWD/../.. celery -A tasks worker +``` + +To test +1. Goto http://localhost:5000, +2. Submit form, +3. Refresh result page until progress states are replaced with result. ### Web service From 459329cc50fd64aacbc64d0ca0b2babd16643f00 Mon Sep 17 00:00:00 2001 From: Stefan Verhoeven Date: Mon, 9 Mar 2020 14:53:35 +0100 Subject: [PATCH 03/11] Fix code identifier --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 612eae8..2cc3395 100644 --- a/README.md +++ b/README.md @@ -527,7 +527,7 @@ PYTHONPATH=$PWD python src/py/awebapp.py Tasks will be run by the Celery worker. The worker can be started with -```{.awk #py-awebapp} +```{.awk #py-worker} cd src/py PYTHONPATH=$PWD/../.. celery -A tasks worker ``` From aab5424db6668e4e7ab1725c0b508bc62e184432 Mon Sep 17 00:00:00 2001 From: Stefan Verhoeven Date: Mon, 9 Mar 2020 14:56:06 +0100 Subject: [PATCH 04/11] Added Python OpenAPI example --- INSTALL.md | 2 +- README.md | 89 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 87 insertions(+), 4 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 86f942e..ba567a3 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -5,7 +5,7 @@ To run the commands in the README.md the following items are required 1. Apache httpd server 2.4 -1. Python libraries `pip install flask pybind11 celery redis connexion` +1. Python libraries `pip install flask pybind11 celery redis connexion[swagger-ui]` 1. Python devel with `sudo apt install python3-dev` 1. Emscriptem diff --git a/README.md b/README.md index 2cc3395..4c415dd 100644 --- a/README.md +++ b/README.md @@ -543,12 +543,95 @@ A web application is meant for consumption by humans and web service is meant fo So instead of returning HTML pages a web service will accept and return machine readable documents like JSON documents. A web service is an application programming interface (API) based on web technologies. A web service has a number of paths or urls to which a request can be sent and a response recieved. -The interface can be defined with [openapi specification](https://github.com/OAI/OpenAPI-Specification) (previously known as [Swagger](https://swagger.io/)). The openapi spec uses JSON schema to define request/response types. Making the JSON schema re-usable between the web service and command line interface. +The interface can be defined with [OpenAPI specification](https://github.com/OAI/OpenAPI-Specification) (previously known as [Swagger](https://swagger.io/)). The openapi spec uses JSON schema to define request/response types. Making the JSON schema re-usable between the web service and command line interface. The open api specifiation can either be generated by the web service provider or be a static document or contract. The contract first approach allows for both consumer and provider to come to an agreement on the contract and work more or less independently on implementation. The contract first approach was used for the Bubble web service. -To make a web service which adheres to the openapi specification contract, it is possible to generate a skeleton using the [generator](https://github.com/OpenAPITools/openapi-generator). +To make a web service which adheres to the OpenAPI specification contract, it is possible to generate a skeleton using the [generator](https://github.com/OpenAPITools/openapi-generator). Each time the contract changes the generator must be re-run. The generator uses [connexion](https://github.com/zalando/connexion) as a web framework when generating a Python based service. -For the Python based Bubble web service connexion was used as the web framework as it maps each path+method combination to a Python function and will handle the validation and serialization. +For the Python based root finding web service, connexion was used as the web framework as it maps each path+method combination to a Python function and will handle the validation and serialization. + +The OpenAPI specification for performing root finding would look like + +```{.yaml file=src/py/openapi.yaml} +openapi: 3.0.0 +info: + title: Root finder + license: + name: Apache-2.0 + url: https://www.apache.org/licenses/LICENSE-2.0.html + version: 0.1.0 +paths: + /api/newtonraphson: + post: + description: Perform root finding with the Newton Raphson algorithm + operationId: api.calculate + requestBody: + content: + 'application/json': + schema: + $ref: '#/components/schemas/NRRequest' + responses: + '200': + description: The found root + content: + application/json: + schema: + $ref: '#/components/schemas/NRResponse' +components: + schemas: + NRRequest: + type: object + properties: + epsilon: + type: number + minimum: 0 + guess: + type: number + required: + - epsilon + - guess + additionalProperties: false + NRResponse: + type: object + properties: + root: + type: number + required: + - root + additionalProperties: false +``` + +The request and response schemas are JSON schemas. + +The operation identifier (operationId) in the specification gets translated by Connexion to a Python method that will be called when the path is requested. Connexion call the functin with the JSON parsed POST request body. + +```{.python file=src/py/api.py} +def calculate(body): + epsilon = body['epsilon'] + guess = body['guess'] + from newtonraphsonpy import NewtonRaphson + finder = NewtonRaphson(epsilon) + root = finder.find(guess) + return {'root': root} +``` + +To run the web service we have to to tell Connexion which specificiation it should expose. + +```{.python file=src/py/webservice.py} +import connexion + +app = connexion.App(__name__) +app.add_api('openapi.yaml', validate_responses=True) +app.run(port=8080) +``` + +The web service can be started with + +```{.awk #run-webservice} +PYTHONPATH=$PWD python src/py/webservice.py +``` + +We can try out the webservice at http://localhost:8080/ui/, this user interface also shows the curl command which can be used to call the web service. ## Javascript From 6a08592ff5fee00ba5ef138eb11485a0cf277313 Mon Sep 17 00:00:00 2001 From: Stefan Verhoeven Date: Mon, 9 Mar 2020 16:05:16 +0100 Subject: [PATCH 05/11] More text --- README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 4c415dd..8a864f1 100644 --- a/README.md +++ b/README.md @@ -543,12 +543,12 @@ A web application is meant for consumption by humans and web service is meant fo So instead of returning HTML pages a web service will accept and return machine readable documents like JSON documents. A web service is an application programming interface (API) based on web technologies. A web service has a number of paths or urls to which a request can be sent and a response recieved. -The interface can be defined with [OpenAPI specification](https://github.com/OAI/OpenAPI-Specification) (previously known as [Swagger](https://swagger.io/)). The openapi spec uses JSON schema to define request/response types. Making the JSON schema re-usable between the web service and command line interface. -The open api specifiation can either be generated by the web service provider or be a static document or contract. The contract first approach allows for both consumer and provider to come to an agreement on the contract and work more or less independently on implementation. The contract first approach was used for the Bubble web service. +The interface can be defined with [OpenAPI specification](https://github.com/OAI/OpenAPI-Specification) (previously known as [Swagger](https://swagger.io/)). The OpenAPI spec uses JSON schema to define request/response types. Making the JSON schema re-usable between the web service and command line interface. +The OpenAPI specifiation can either be generated by the web service provider or be a static document or contract. The contract first approach allows for both consumer and provider to come to an agreement on the contract and work more or less independently on implementation. The contract first approach was used for the root finding web service. To make a web service which adheres to the OpenAPI specification contract, it is possible to generate a skeleton using the [generator](https://github.com/OpenAPITools/openapi-generator). -Each time the contract changes the generator must be re-run. The generator uses [connexion](https://github.com/zalando/connexion) as a web framework when generating a Python based service. -For the Python based root finding web service, connexion was used as the web framework as it maps each path+method combination to a Python function and will handle the validation and serialization. +Each time the contract changes the generator must be re-run. The generator uses the Python based web framework [Connexion](https://github.com/zalando/connexion). +For the Python based root finding web service, Connexion was used as the web framework as it maps each path+method combination in the contract to a Python function and will handle the validation and serialization. The OpenAPI specification for performing root finding would look like @@ -601,9 +601,10 @@ components: additionalProperties: false ``` -The request and response schemas are JSON schemas. +The webservice consists of a single path with the POST method which recieves a response and sends a response. +The request and response are in JSON format and adhere to their respective JSON schemas. -The operation identifier (operationId) in the specification gets translated by Connexion to a Python method that will be called when the path is requested. Connexion call the functin with the JSON parsed POST request body. +The operation identifier (`operationId`) in the specification gets translated by Connexion to a Python method that will be called when the path is requested. Connexion calls the function with the JSON parsed request body. ```{.python file=src/py/api.py} def calculate(body): From 3317e8a1c5d80926be3eeb6f809d4352fba3dc0c Mon Sep 17 00:00:00 2001 From: Stefan Verhoeven Date: Mon, 9 Mar 2020 16:25:21 +0100 Subject: [PATCH 06/11] Remove space --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 8a864f1..40c75ea 100644 --- a/README.md +++ b/README.md @@ -457,7 +457,7 @@ Will add some sleeps to demonstrate what would happen with a long running calcul ```{.python file=src/py/tasks.py} import time -<< celery-config>> +<> @capp.task(bind=True) def calculate(self, epsilon, guess): @@ -532,7 +532,7 @@ cd src/py PYTHONPATH=$PWD/../.. celery -A tasks worker ``` -To test +To test web service 1. Goto http://localhost:5000, 2. Submit form, 3. Refresh result page until progress states are replaced with result. From a49fee34e8d88dc2f9583ef67ce6cbc2d2b73595 Mon Sep 17 00:00:00 2001 From: Stefan Verhoeven Date: Tue, 10 Mar 2020 09:45:41 +0100 Subject: [PATCH 07/11] More text, less bubbles and spelling --- README.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 40c75ea..d0ec3fd 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,8 @@ Document describing a way that a scientist with a C++ algorithm can make it available as a web application. The [Newton raphson root finding algorithm](https://en.wikipedia.org/wiki/Newton%27s_method) will be the use case. +The interface would like + ```{.cpp file=newtonraphson.hpp} #ifndef H_NEWTONRAPHSON_H #define H_NEWTONRAPHSON_H @@ -37,6 +39,8 @@ namespace rootfinding { #endif ``` +The implementation would look like + ```{.cpp #algorithm} #include "newtonraphson.hpp" @@ -77,6 +81,8 @@ double NewtonRaphson::find(double xin) } // namespace rootfinding ``` +A simple CLI program would look like + ```{.cpp file=cli-newtonraphson.cpp} #include @@ -123,7 +129,7 @@ The C++ code has the following characteristics: To make the same input and output reusable from either the command line or web service, the [JSON format](http://json.org/) was chosen. As the JSON format is easy read and write by human and machines. -Compare with a binary format or comma seperated file it is more verbose, but is more self documenting. It is less verbose than [XML](https://en.wikipedia.org/wiki/XML) and like there is a [XML schema definition](https://en.wikipedia.org/wiki/XML_Schema_(W3C)) for validation of XML there is an equivalent for JSON called [JSON schema](https://json-schema.org/). JSON schema is used to describe the shape of a JSON document and make sure Bubble consumers know how to provide the input and what to expect from the output. [YAML format](https://yaml.org/) was not chosen, because it is a superset of JSON and JSON is all expressiveness Bubble required. YAML allows for comments while this is not supported in JSON. Also JSON is the lingua france for web services. +Compare with a binary format or comma seperated file it is more verbose, but is more self documenting. It is less verbose than [XML](https://en.wikipedia.org/wiki/XML) and like there is a [XML schema definition](https://en.wikipedia.org/wiki/XML_Schema_(W3C)) for validation of XML there is an equivalent for JSON called [JSON schema](https://json-schema.org/). JSON schema is used to describe the shape of a JSON document and make sure root finder consumers know how to provide the input and what to expect from the output. [YAML format](https://yaml.org/) was not chosen, because it is a superset of JSON and JSON is all expressiveness root finder required. YAML allows for comments while this is not supported in JSON. Also JSON is the lingua france for web services. An example of JSON schema: @@ -616,7 +622,7 @@ def calculate(body): return {'root': root} ``` -To run the web service we have to to tell Connexion which specificiation it should expose. +To run the web service we have to to tell Connexion which specification it should expose. ```{.python file=src/py/webservice.py} import connexion From 6ce80647902632a43d96b315e24dbe851386a176 Mon Sep 17 00:00:00 2001 From: Stefan Verhoeven Date: Thu, 12 Mar 2020 13:18:33 +0100 Subject: [PATCH 08/11] Implement suggestions --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 003d1bc..9abacb0 100644 --- a/README.md +++ b/README.md @@ -345,7 +345,7 @@ Run with python src/py/hello.py ``` -The above route will just return the string "Hello World!" in the web browser when visiting [http://localhost:5000/](http://localhost:5000/). It is possible to return a html page aswell, but to make it dynamic it soon becomes a mess of string concatenations. Template languages help to avoid the concatination mess. Flask is configured with the [Jinja2](https://jinja.palletsprojects.com/). A template for the above route could look like: +The above route will just return the string "Hello World!" in the web browser when visiting [http://localhost:5000/](http://localhost:5000/). It is possible to return a html page as well, but to make it dynamic it soon becomes a mess of string concatenations. Template engines help to avoid the concatination mess. Flask is configured with the [Jinja2](https://jinja.palletsprojects.com/) template engine. A template for the above route could look like: ```{.html file=src/py/templates/hello.html} From 28d7bea9eb4ee032859fa9da602e43ad61ef0bbd Mon Sep 17 00:00:00 2001 From: Stefan Verhoeven Date: Thu, 12 Mar 2020 13:21:07 +0100 Subject: [PATCH 09/11] Fix lint warnings --- README.md | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 9abacb0..bca5b5b 100644 --- a/README.md +++ b/README.md @@ -112,6 +112,7 @@ Run with ```{.awk #test-cli} ./newtonraphson.exe ``` + Should output ```shell @@ -198,16 +199,19 @@ int main(int argc, char *argv[]) Where `nlohmann/json.hpp` is a JSON serialization/unserialization C++ header only library to convert a JSON string to and from a data type. This can be compile with + ```{.awk #build-cgi} g++ -Ideps cgi-newtonraphson.cpp -o ./cgi-bin/newtonraphson ``` The CGI script can be tested directly with + ```{.awk #test-cgi} echo '{"guess":-20, "epsilon":0.001}' | ./cgi-bin/newtonraphson ``` It should output + ```{.awk #test-cgi-output} Content-type: application/json @@ -235,7 +239,7 @@ ScriptAlias "/cgi-bin/" "cgi-bin/" Start Apache httpd web server using ```{.shell #run-httpd} -/usr/sbin/apache2 -X -d . -f ./httpd.conf +/usr/sbin/apache2 -X -d . -f ./httpd.conf ``` And in another shell call CGI script using curl @@ -341,6 +345,7 @@ app.run() ``` Run with + ```{.awk #py-hello} python src/py/hello.py ``` @@ -374,11 +379,12 @@ app.run() Where `name` is a variable which gets combined with template to render into a html page. The web application can be started with + ```{.awk #py-hello-templated} python src/py/hello-templated.py ``` -In a web browser you can visit http://localhost:5000/hello/yourname to the web application. +In a web browser you can visit [http://localhost:5000/hello/yourname](http://localhost:5000/hello/yourname) to the web application. Let's make the web application for our Newton raphson algorithm. @@ -444,11 +450,11 @@ And running it with PYTHONPATH=$PWD python src/py/webapp.py ``` -To test we can visit http://localhost:5000 fill the form and press submit to get the result. +To test we can visit [http://localhost:5000](http://localhost:5000) fill the form and press submit to get the result. When performing a long calculation (more than 30 seconds), the end-user requires feedback of the progress. In a normal request/response cycle, feedback is only returned in the response. To give feedback during the calculation, the computation must be offloaded to a task queue. In Python the most used task queue is [celery](http://www.celeryproject.org/). While the calculation is running on some worker it is possible to have a progress page which can check in the queue what the progress is of the calculation. -Celery needs a broker to use a queue and store the results. +Celery needs a broker to use a queue and store the results. Will use [redis](https://redis.io/) in a Docker container as Celery broker. ```{.awk #run-redis} @@ -485,7 +491,7 @@ def calculate(self, epsilon, guess): return {'root': root, 'guess': guess, 'epsilon':epsilon} ``` -Instead of running the calculation when the submit button is pressed. +Instead of running the calculation when the submit button is pressed. We will submit the calculation task to the task queue by using the `.delay()` function. The submission will return a job identifier we can use later to get the status and result of the job. The web browser will redirect to a url with the job identifier in it. @@ -545,7 +551,8 @@ PYTHONPATH=$PWD/../.. celery -A tasks worker ``` To test web service -1. Goto http://localhost:5000, + +1. Goto [http://localhost:5000](http://localhost:5000), 2. Submit form, 3. Refresh result page until progress states are replaced with result. @@ -644,7 +651,7 @@ The web service can be started with PYTHONPATH=$PWD python src/py/webservice.py ``` -We can try out the webservice at http://localhost:8080/ui/, this user interface also shows the curl command which can be used to call the web service. +We can try out the webservice at [http://localhost:8080/ui/](http://localhost:8080/ui/), this user interface also shows the curl command which can be used to call the web service. ## Javascript From 6e19bedc67cc02bf5d06d029ddd7b5fa3c9d738a Mon Sep 17 00:00:00 2001 From: Stefan Verhoeven Date: Thu, 12 Mar 2020 13:33:49 +0100 Subject: [PATCH 10/11] Created long runnning task chapter + talk about Swagger UI --- README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index bca5b5b..76f3684 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,7 @@ - [Python](#python) - [Accessing C++ function from Python](#accessing-c-function-from-python) - [Web application](#web-application) + - [Long running tasks](#long-running-tasks) - [Web service](#web-service) - [Javascript](#javascript) - [Accessing C++ function from Javascript in web browser](#accessing-c-function-from-javascript-in-web-browser) @@ -452,6 +453,8 @@ PYTHONPATH=$PWD python src/py/webapp.py To test we can visit [http://localhost:5000](http://localhost:5000) fill the form and press submit to get the result. +### Long running tasks + When performing a long calculation (more than 30 seconds), the end-user requires feedback of the progress. In a normal request/response cycle, feedback is only returned in the response. To give feedback during the calculation, the computation must be offloaded to a task queue. In Python the most used task queue is [celery](http://www.celeryproject.org/). While the calculation is running on some worker it is possible to have a progress page which can check in the queue what the progress is of the calculation. Celery needs a broker to use a queue and store the results. @@ -567,7 +570,7 @@ The OpenAPI specifiation can either be generated by the web service provider or To make a web service which adheres to the OpenAPI specification contract, it is possible to generate a skeleton using the [generator](https://github.com/OpenAPITools/openapi-generator). Each time the contract changes the generator must be re-run. The generator uses the Python based web framework [Connexion](https://github.com/zalando/connexion). -For the Python based root finding web service, Connexion was used as the web framework as it maps each path+method combination in the contract to a Python function and will handle the validation and serialization. +For the Python based root finding web service, Connexion was used as the web framework as it maps each path+method combination in the contract to a Python function and will handle the validation and serialization. The OpeAPI web service can be tested with [Swagger UI](https://swagger.io/tools/swagger-ui/), the UI allows browsing through the available paths, try them out by constructing a request and shows the curl command which can be used to call the web service. Swagger UI comes bundled with the Connexion framework. The OpenAPI specification for performing root finding would look like @@ -651,7 +654,7 @@ The web service can be started with PYTHONPATH=$PWD python src/py/webservice.py ``` -We can try out the webservice at [http://localhost:8080/ui/](http://localhost:8080/ui/), this user interface also shows the curl command which can be used to call the web service. +We can try out the web service using Swagger UI at [http://localhost:8080/ui/](http://localhost:8080/ui/). ## Javascript From 4919878a6b5c9f95e7e771dba8100d4f63fe49b0 Mon Sep 17 00:00:00 2001 From: Stefan Verhoeven Date: Thu, 12 Mar 2020 13:44:23 +0100 Subject: [PATCH 11/11] Improve explanation of redis --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 76f3684..0ad1753 100644 --- a/README.md +++ b/README.md @@ -457,8 +457,8 @@ To test we can visit [http://localhost:5000](http://localhost:5000) fill the for When performing a long calculation (more than 30 seconds), the end-user requires feedback of the progress. In a normal request/response cycle, feedback is only returned in the response. To give feedback during the calculation, the computation must be offloaded to a task queue. In Python the most used task queue is [celery](http://www.celeryproject.org/). While the calculation is running on some worker it is possible to have a progress page which can check in the queue what the progress is of the calculation. -Celery needs a broker to use a queue and store the results. -Will use [redis](https://redis.io/) in a Docker container as Celery broker. +Celery needs a broker for a queue and result storage. +Will use [redis](https://redis.io/) in a Docker container as Celery broker, because it's simple to setup. Redis can be started with the following command ```{.awk #run-redis} docker run -d -p 6379:6379 redis