diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml new file mode 100644 index 0000000..1efb1a7 --- /dev/null +++ b/.github/workflows/main.yml @@ -0,0 +1,93 @@ +# This is a basic workflow to help you get started with Actions + +name: CI + +# Controls when the action will run. Triggers the workflow on push or pull request +# events but only for the master branch +on: [push, pull_request] + +# A workflow run is made up of one or more jobs that can run sequentially or in parallel +jobs: + cpp: + # The type of runner that the job will run on + runs-on: ubuntu-latest + + # Steps represent a sequence of tasks that will be executed as part of the job + steps: + # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it + - uses: actions/checkout@v2 + + # Should not be needed anymore when https://github.com/NLESC-JCER/cpp2wasm/issues/1 is fixed + - name: Generate source code + uses: docker://nlesc/pandoc-tangle + with: + args: README.md INSTALL.md + + # Should not be needed anymore when https://github.com/entangled/filters/issues/2 is fixed + - name: Correct Makefile + run: perl -pi -e 's/ /\t/' Makefile + + - name: Run C++ examples + run: make test-cli test-cgi + python: + # The type of runner that the job will run on + runs-on: ubuntu-latest + # Redis is needed for Celery + services: + redis: + image: redis + ports: + - 6379:6379 + steps: + # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it + - uses: actions/checkout@v2 + + # Should not be needed anymore when https://github.com/NLESC-JCER/cpp2wasm/issues/1 is fixed + - name: Generate source code + uses: docker://nlesc/pandoc-tangle + with: + args: README.md INSTALL.md + + # Should not be needed anymore when https://github.com/entangled/filters/issues/2 is fixed + - name: Correct Makefile + run: perl -pi -e 's/ /\t/' Makefile + + - uses: actions/setup-python@v1 + with: + python-version: '3.x' # Version range or exact version of a Python version to use, using SemVer's version range syntax + architecture: 'x64' # optional x64 or x86. Defaults to x64 if not specified + + - name: Install Python dependencies + run: make py-deps && pip install httpie + + - name: Run Python example + run: make test-py + + - name: Start web application in background + run: make run-webapp & + + - name: Test web application + run: http --ignore-stdin -f localhost:5001 epsilon=0.001 guess=-20 + + - name: Start web service in background + run: make run-webservice & + + - name: Test web service + run: make test-webservice + + - name: Start Celery web app in background + run: make run-celery-webapp & + + - name: Start Celery worker in background + run: | + cd src/py + PYTHONPATH=$PWD/../.. celery -A tasks worker & + cd ../.. + + - name: Test Celery web app + run: | + http --ignore-stdin -hf localhost:5000 epsilon=0.001 guess=-20 | tee response.txt + # Parse result url from response + RESULT_URL=$(cat response.txt |grep Location |awk '{print $2}') + sleep 2 + http --ignore-stdin $RESULT_URL diff --git a/INSTALL.md b/INSTALL.md index 290dfe9..ec54320 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -1,12 +1,15 @@ # Installation +![CI](https://github.com/NLESC-JCER/cpp2wasm/workflows/CI/badge.svg) + ## Dependencies To run the commands in the README.md the following items are required -1. [Apache httpd server 2.4](http://httpd.apache.org/) -1. Python devel with `sudo apt install python3-dev` +1. [Apache httpd server 2.4](http://httpd.apache.org/) with `sudo apt install -y apache2` +1. Python devel with `sudo apt install -y python3-dev` 1. [Emscriptem](https://emscripten.org/docs/getting_started/downloads.html) +1. [Docker Engine](https://docs.docker.com/install/) ## Generating code from Markdown @@ -30,9 +33,9 @@ docker run --rm -ti --user $(id -u) -v ${PWD}:/data nlesc/pandoc-tangle README.m All the commands in the README.md can be captured in a Makefile like so: ```{.makefile file=Makefile} -.PHONY: clean test entangle deps +.PHONY: clean test entangle py-deps start-redis stop-redis run-webservice run-celery-webapp run-webapp -deps: pip-pybind11 pip-flask pip-celery pip-connexion +py-deps: pip-pybind11 pip-flask pip-celery pip-connexion pip-pybind11: <> @@ -43,8 +46,8 @@ pip-flask: pip-celery: <> -pip-connexion - > +pip-connexion: + <> newtonraphson.exe: cli-newtonraphson.cpp <> @@ -61,11 +64,41 @@ test-cgi: cgi-bin/newtonraphson newtonraphsonpy.*.so: py-newtonraphson.cpp <> -test-py: example.py newtonraphsonpy.*.so - python example.py +test-py: src/py/example.py newtonraphsonpy.*.so + PYTHONPATH=${PWD} python src/py/example.py -test: test-cli test-cgi test-py +test: test-cli test-cgi test-py test-webservice clean: $(RM) newtonraphson.exe newtonraphsonpy.*.so cgi-bin/newtonraphson + +start-redis: + <> + +stop-redis: + <> + +run-webapp: newtonraphsonpy.*.so + <> + +run-webservice: newtonraphsonpy.*.so + <> + +test-webservice: + <> + +# Unable to get worker runnig correctly from Makefile, the newtonraphsonpy.*.so cannot be found +# run-celery-worker: newtonraphsonpy.*.so +# <> + +run-celery-webapp: newtonraphsonpy.*.so + <> +``` + +For example the Python dependencies can be installed with + +```shell +make py-deps ``` + +See [GitHub Actions workflow](.github/workflows/main.yml) for other usages of the Makefile. diff --git a/Makefile b/Makefile index 4e2e7fe..7291eac 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,18 @@ -.PHONY: clean test entangle +.PHONY: clean test entangle py-deps start-redis stop-redis run-webservice run-celery-webapp run-webapp + +py-deps: pip-pybind11 pip-flask pip-celery pip-connexion + +pip-pybind11: + pip install pybind11 + +pip-flask: + pip install flask + +pip-celery: + pip install celery[redis] + +pip-connexion: + pip install connexion[swagger-ui] newtonraphson.exe: cli-newtonraphson.cpp g++ cli-newtonraphson.cpp -o newtonraphson.exe @@ -17,9 +31,31 @@ newtonraphsonpy.*.so: py-newtonraphson.cpp py-newtonraphson.cpp -o newtonraphsonpy`python3-config --extension-suffix` test-py: example.py newtonraphsonpy.*.so - python example.py + PYTHONPATH=${PWD} python src/py/example.py -test: test-cli test-cgi test-py +test: test-cli test-cgi test-py test-webservice clean: - $(RM) newtonraphson.exe newtonraphsonpy.*.so cgi-bin/newtonraphson \ No newline at end of file + $(RM) newtonraphson.exe newtonraphsonpy.*.so cgi-bin/newtonraphson + +start-redis: + docker run --rm -d -p 6379:6379 --name some-redis redis + +stop-redis: + docker stop some-redis + +run-webapp: newtonraphsonpy.*.so + PYTHONPATH=${PWD} python src/py/webapp.py + +run-webservice: newtonraphsonpy.*.so + PYTHONPATH=${PWD} python src/py/webservice.py + +test-webservice: + curl -X POST "http://localhost:8080/api/newtonraphson" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"epsilon\":0.001,\"guess\":-20}" + +# Unable to get worker runnig correctly from Makefile, the newtonraphsonpy.*.so cannot be found +# run-celery-worker: newtonraphsonpy.*.so +# <> + +run-celery-webapp: newtonraphsonpy.*.so + PYTHONPATH=${PWD} python src/py/webapp-celery.py \ No newline at end of file diff --git a/README.md b/README.md index d438313..0125220 100644 --- a/README.md +++ b/README.md @@ -16,6 +16,8 @@ - [Form](#form) - [Visualization](#visualization) +![CI](https://github.com/NLESC-JCER/cpp2wasm/workflows/CI/badge.svg) + Document describing a way that a scientist with a C++ algorithm can make it available as a web application. The [Newton raphson root finding algorithm](https://en.wikipedia.org/wiki/Newton%27s_method) will be the use case. @@ -318,7 +320,7 @@ py-newtonraphson.cpp -o newtonraphsonpy`python3-config --extension-suffix` In Python it can be used: -```{.python file=example.py} +```{.python file=src/py/example.py} from newtonraphsonpy import NewtonRaphson finder = NewtonRaphson(epsilon=0.001) @@ -454,16 +456,16 @@ app = Flask(__name__) <> -app.run() +app.run(port=5001) ``` And running it with -```{.awk #py-webapp} -PYTHONPATH=$PWD python src/py/webapp.py +```{.awk #run-webapp} +PYTHONPATH=${PWD} python src/py/webapp.py ``` -To test we can visit [http://localhost:5000](http://localhost:5000) fill the form and press submit to get the result. +To test we can visit [http://localhost:5001](http://localhost:5001) fill the form and press submit to get the result. ### Long running tasks @@ -472,8 +474,8 @@ When performing a long calculation (more than 30 seconds), the end-user requires Celery needs a broker for a queue and result storage. Will use [redis](https://redis.io/) in a Docker container as Celery broker, because it's simple to setup. Redis can be started with the following command -```{.awk #run-redis} -docker run -d -p 6379:6379 redis +```{.awk #start-redis} +docker run --rm -d -p 6379:6379 --name some-redis redis ``` To use Celery we must install the redis flavoured version with @@ -543,7 +545,7 @@ def result(jobid): Putting it all together -```{.python file=src/py/awebapp.py} +```{.python file=src/py/webapp-celery.py} from flask import Flask, render_template, request, redirect, url_for app = Flask(__name__) @@ -555,18 +557,18 @@ app = Flask(__name__) <> if __name__ == '__main__': - app.run() + app.run(port=5000) ``` Start the web application like before with -```{.awk #py-awebapp} -PYTHONPATH=$PWD python src/py/awebapp.py +```{.awk #run-celery-webapp} +PYTHONPATH=${PWD} python src/py/webapp-celery.py ``` Tasks will be run by the Celery worker. The worker can be started with -```{.awk #py-worker} +```{.awk #run-celery-worker} cd src/py PYTHONPATH=$PWD/../.. celery -A tasks worker ``` @@ -577,6 +579,12 @@ To test web service 2. Submit form, 3. Refresh result page until progress states are replaced with result. +The redis server can be shutdown with + +```{.awk #stop-redis} +docker stop some-redis +``` + ### Web service A web application is meant for consumption by humans and web service is meant for consumption by machines or other programs. @@ -675,10 +683,15 @@ app.run(port=8080) The web service can be started with ```{.awk #run-webservice} -PYTHONPATH=$PWD python src/py/webservice.py +PYTHONPATH=${PWD} python src/py/webservice.py ``` We can try out the web service using the Swagger UI at [http://localhost:8080/ui/](http://localhost:8080/ui/). +Or by running a curl command like + +```{.awk #test-webservice} +curl -X POST "http://localhost:8080/api/newtonraphson" -H "accept: application/json" -H "Content-Type: application/json" -d "{\"epsilon\":0.001,\"guess\":-20}" +``` ## Javascript diff --git a/cgi-bin/.gitignore b/cgi-bin/.gitignore new file mode 100644 index 0000000..e69de29 diff --git a/src/py/templates/.gitignore b/src/py/templates/.gitignore new file mode 100644 index 0000000..e69de29