Merge c8ac7dd into a1c8502

.codacy.yml

@@ -1,4 +1,5 @@
 ---
 exclude_paths:
+  - 'pywps/alembic/versions/**'
   - 'tests/**'
   - 'docs/conf.py'

.travis.yml

@@ -1,10 +1,11 @@
+dist: xenial   # required for Python >= 3.7
 language: python
 
 sudo: required
-dist: trusty
 
 python:
   - "3.6"
+  - "3.7"
 
 git:
   submodules: false
@@ -25,13 +26,18 @@ addons:
 git:
     submodules: false
 
+env:
+  - PYWPS_CFG="$TRAVIS_BUILD_DIR/tests/test-pywps.cfg"
+
 install:
   - pip install pip --upgrade
   - pip install -r requirements.txt
   - pip install -r requirements-gdal.txt
   - pip install -r requirements-extra.txt
   - pip install -r requirements-dev.txt
   - pip install coveralls
+  - pip install -e .
+  - pywps -c $PYWPS_CFG migrate
 
 script:
   - python -m unittest tests

alembic.ini

@@ -0,0 +1,2 @@
+[alembic]
+script_location = pywps:alembic

docs/configuration.rst

@@ -16,6 +16,7 @@ The configuration file has several sections:
 
     * `metadata:main` for the server metadata inputs
     * `server` for server configuration
+    * `jobqueue` for job queue configuration
     * `processing` for processing backend configuration
     * `logging` for logging configuration
     * `grass` for *optional* configuration to support `GRASS GIS
@@ -101,6 +102,12 @@ configuration file <https://docs.pycsw.org/en/latest/configuration.html>`_.
     https://docs.python.org/2/library/codecs.html#standard-encodings).  Default
     value is 'UTF-8'
 
+:processes:
+    optional parameter to configure the processes list using a Python expression.
+    This expression will be loaded using ``importlib``.
+    For example use `myapp.processes.processes` which points to the Python
+    list of your processes in your application: ``processes = [Sleep(), SayHello()]``.
+
 :parallelprocesses:
     maximum number of parallel running processes - set this number carefully.
     The effective number of parallel running processes is limited by the number
@@ -187,6 +194,10 @@ configuration file <https://docs.pycsw.org/en/latest/configuration.html>`_.
     Connection string to database where the login about requests/responses is to be stored. We are using `SQLAlchemy <https://docs.sqlalchemy.org/en/latest/core/engines.html#database-urls>`_
     please use the configuration string. The default is SQLite3 `:memory:` object, however this has `known issues <https://github.com/geopython/pywps/issues?utf8=%E2%9C%93&q=is%3Aissue+async+sqlite>`_ with async processing and should be avoided.
 
+:db_echo:
+    flag to enable database logging.
+
+    Default = `false`.
 
 [grass]
 -------
@@ -195,8 +206,15 @@ configuration file <https://docs.pycsw.org/en/latest/configuration.html>`_.
   directory of the GRASS GIS instalation, refered as `GISBASE
   <https://grass.osgeo.org/grass73/manuals/variables.html>`_
 
+[jobqueue]
+--------
+
+:pause:
+  pausing in seconds between periodical check for new stored requests
 
 [s3]
+----
+
 :bucket:
   Name of the bucket to store files in. e.g. ``my-wps-results``
 
@@ -269,10 +287,12 @@ Sample file
   [grass]
   gisbase=/usr/local/grass-7.3.svn/
 
+  [jobqueue]
+  pause=30
+
   [s3]
   bucket=my-org-wps
   region=us-east-1
   prefix=appname/coolapp/
   public=true
   encrypt=false
-

docs/deployment.rst

@@ -3,22 +3,61 @@
 Deployment to a production server
 =================================
 
+PyWPS consists from two main parts:
+* PyWPS :py:class:`pywps.app.Service` the main process to accept client requests,
+* :py:module:`pywps.queue` responsible for calling the stored requests in the asynchronous mode.
+
+--------------
+Service module
+--------------
+The :py:class:`pywps.app.Service` class is responsible for *synchronous*
+request executions: GetCapabilites, DescribeProcess and Execute in sync. mode,
+in this case, the Service class will
+
+1. Accept request
+2a. In case, the request is to be executed in *synchronous* mode, it will be
+        directly executed - sync requests are immediately executed.
+2b. In case, the request is to be executed as *asynchronous* mode, request
+    will be stored in to the database.
+
+This means: asynchronous requests are not executed by the Service class, they
+will be just stored into database.
+
+------------
+Queue module
+------------
+The :py:module:`pywps.queue` has a `JobQueueService` to be started separately - it will
+start a process, which will periodically check for the database stored
+requests and in case, some process is there, it will removed and executed.
+
+The service does only start as many processes are allowed in the
+`parallelprocesses` process number.
+
+The service does not accept requests from the client, it is just watching the
+database and starting process jobs.
+
+
+------------
+Installation
+------------
 As already described in the :ref:`installation` section, no specific deployment
-procedures are for PyWPS when using flask-based server. But this formula is not 
+procedures are for PyWPS when using flask-based server. But this formula is not
 intended to be used in a production environment. For production, `sudo service apache2 restartApache httpd
 <https://httpd.apache.org/>`_ or `nginx <https://nginx.org/>`_ servers are
-more advised. PyWPS is runs as a `WSGI
+more advised. PyWPS is run as a `WSGI
 <https://wsgi.readthedocs.io/en/latest/>`_ application on those servers. PyWPS
 relies on the `Werkzeug <http://werkzeug.pocoo.org/>`_ library for this purpose.
 
+Then the job queue service has to be started too.
+
 Deploying an individual PyWPS instance
 --------------------------------------
 
-PyWPS should be installed in your computer (as per the :ref:`installation` 
-section). As a following step, you can now create several instances of your WPS 
+PyWPS should be installed in your computer (as per the :ref:`installation`
+section). As a following step, you can now create several instances of your WPS
 server.
 
-It is advisable for each PyWPS instance to have its own directory, where the 
+It is advisable for each PyWPS instance to have its own directory, where the
 WSGI file along with available processes should reside. Therefore create a new
 directory for the PyWPS instance::
 
@@ -29,21 +68,40 @@ directory for the PyWPS instance::
 
 .. note:: In this configuration example it is assumed that there is only one
         instance of PyWPS on the server.
-        
-Each instance is represented by a single `WSGI` script (written in Python), 
+
+Each instance is represented by a single `WSGI` script (written in Python),
 which:
 
     1. Loads the configuration files
     2. Serves processes
     3. Takes care about maximum number of concurrent processes and similar
 
+Starting PyWPS using Werkzeug
+-----------------------------
+
+PyWPS has a command line interface to start a PyWPS instance with Werkzeug for
+development purposes. You need a configuration file ```pywps.cfg`` pointing to
+the available processes::
+
+    [server]
+    processes = myapp.processes.processes
+
+Start the pywps with the following command::
+
+    $ pywps -c pywps.cfg start
+
+You can also start the pywps service and the job queue as seperated processes::
+
+    $ pywps -c pywps.cfg start --no-jobqueue
+    $ pywps -c pywps.cfg jobqueue 
+
 Creating a PyWPS `WSGI` instance
 --------------------------------
 
-An example WSGI script is distributed along with the pywps-flask service, as 
-described in the :ref:`installation` section. The script is actually 
-straightforward - in fact, it's a just wrapper around the PyWPS server with a 
-list of processes and configuration files passed as arguments. Here is an 
+An example WSGI script is distributed along with the pywps-flask service, as
+described in the :ref:`installation` section. The script is actually
+straightforward - in fact, it's a just wrapper around the PyWPS server with a
+list of processes and configuration files passed as arguments. Here is an
 example of a PyWPS WSGI script::
 
     $ $EDITOR /path/to/pywps/pywps.wsgi
@@ -86,9 +144,10 @@ example of a PyWPS WSGI script::
         processes at hand that can be directly included. Also it assumes, that
         the configuration file already exists - which is not the case yet.
 
-        The Configuration is described in next chapter (:ref:`configuration`), 
+        The Configuration is described in next chapter (:ref:`configuration`),
         as well as process creation and deployment (:ref:`process`).
 
+
 Deployment on Apache2 httpd server
 ----------------------------------
 
@@ -118,17 +177,17 @@ You then can edit your site configuration file
         The `outputpath` directory need also be accessible from the URL mentioned in `outputurl` configuration.
 
 And of course restart the server::
-    
+
     $ sudo service apache2 restart
 
 
 Deployment on Nginx-Gunicorn
 ----------------------------
 
-.. note:: We will use Greenunicorn  for pyWPS deployment, since it is a very simple to configurate server. 
+.. note:: We will use Greenunicorn  for pyWPS deployment, since it is a very simple to configurate server.
 
    For difference between WSGI server consult:  `WSGI comparison <https://www.digitalocean.com/community/tutorials/a-comparison-of-web-servers-for-python-based-web-applications>`_.
-   
+
    uWSGU is more popular than gunicorn, best documentation is probably to be found at `Readthedocs <https://uwsgi-docs.readthedocs.io/en/latest/WSGIquickstart.html>`_.
 
 We need nginx and gunicorn server::
@@ -139,31 +198,31 @@ We need nginx and gunicorn server::
 It is assumed that PyWPS  is installed in your system (if not see: ref:`installation`) and we will use pywps-flask as installation example.
 
 First, cloning the pywps-flask example to the root / (you need to be sudoer or root to run the examples)::
-   
+
    $ cd /
    $ git clone https://github.com/geopython/pywps-flask.git
 
-Second, preparing the WSGI script for gunicorn. It is necessary that the 
-WSGI script located in the pywps-flask service is identified as a python module by gunicorn, 
-this is done by creating a link with .py extention to the wsgi file::  
-   
+Second, preparing the WSGI script for gunicorn. It is necessary that the
+WSGI script located in the pywps-flask service is identified as a python module by gunicorn,
+this is done by creating a link with .py extention to the wsgi file::
+
    $ cd /pywps-flask/wsgi
-   $ ln -s ./pywps.wsgi ./pywps_app.py 
-   
+   $ ln -s ./pywps.wsgi ./pywps_app.py
+
 Gunicorn can already be tested by setting python path on the command options::
-   
-   $ gunicorn3 -b 127.0.0.1:8081  --workers $((2*`nproc --all`)) --log-syslog  --pythonpath /pywps-flask wsgi.pywps_app:application   
-  
-The command will start a gunicorn instance on the localhost IP and port 8081, logging to systlog 
-(/var/log/syslog), using pywps process folder /pywps-flask/processes and loading module wsgi.pywps_app and object/function application for WSGI.  
-
-.. note::  Gunicorn uses a prefork model where the master process forks processes (workers) 
-   that willl accept incomming connections. The --workers flag sets the number of processes, 
-   the default values is 1 but the recomended value is 2 or 4 times the number of CPU cores.      
-
-Next step is to configure NGINX,  by pointing to the WSGI server by changing the location paths of the  default  
-site file but editing file /etc/nginx/sites-enabled as follows::: 
-   
+
+   $ gunicorn3 -b 127.0.0.1:8081  --workers $((2*`nproc --all`)) --log-syslog  --pythonpath /pywps-flask wsgi.pywps_app:application
+
+The command will start a gunicorn instance on the localhost IP and port 8081, logging to systlog
+(/var/log/syslog), using pywps process folder /pywps-flask/processes and loading module wsgi.pywps_app and object/function application for WSGI.
+
+.. note::  Gunicorn uses a prefork model where the master process forks processes (workers)
+   that willl accept incomming connections. The --workers flag sets the number of processes,
+   the default values is 1 but the recomended value is 2 or 4 times the number of CPU cores.
+
+Next step is to configure NGINX,  by pointing to the WSGI server by changing the location paths of the  default
+site file but editing file /etc/nginx/sites-enabled as follows:::
+
    server {
         listen 80 default_server;
         listen [::]:80 default_server;
@@ -185,19 +244,19 @@ site file but editing file /etc/nginx/sites-enabled as follows:::
                 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                 proxy_pass http://127.0.0.1:8081;
                 }
-   
+
    }
- 
-It is likely that part of the proxy configuration is already set on the file /etc/nginx/proxy.conf.  
-Of course the necessatyrestart of nginx :: 
-   
+
+It is likely that part of the proxy configuration is already set on the file /etc/nginx/proxy.conf.
+Of course the necessatyrestart of nginx ::
+
    $ service nginx restart
-   
+
 The service will now be available on the IP of the server or localhost ::
-   
+
    http://localhost/wps?request=GetCapabilities&service=wps
- 
-The current gunicorn instance was launched by the user. In a production server it is necessary to set gunicorn as a service  
+
+The current gunicorn instance was launched by the user. In a production server it is necessary to set gunicorn as a service
 
 On ubuntu 16.04  the systemcltd system requires a service file that will start the gunicorn3 service. The service file (/lib/systemd/system/gunicorn.service)
 has to be configure as follows::
@@ -214,24 +273,30 @@ has to be configure as follows::
    ExecStart=/usr/bin/gunicorn3 -b 127.0.0.1:8081   --preload --workers $WORKERS --log-syslog --pythonpath /pywps-flask wsgi.pywps_app:application
    ExecReload=/bin/kill -s HUP $MAINPID
    ExecStop=/bin/kill -s TERM $MAINPID
-   
+
    [Install]
    WantedBy=multi-user.target
 
 And then enable the service and then reload the systemctl daemon::
-   
+
    $ systemctl enable gunicorn3.service
    $ systemctl daemon-reload
    $ systemctl restart gunicorn3.service
 
 And  to check that everything is ok::
-   
+
    $ systemctl status gunicorn3.service
 
 .. note::
-   
+
    Todo NGIX + uWSGI
 
+------------------
+Job queue starting
+------------------
+The job queue has to be started from command line::
+
+        pywps jobqueue --config /path/to/configuration/pywps.cfg
 
 .. _deployment-testing:
 
@@ -258,11 +323,11 @@ The result should be an XML-encoded error message.
         </ows:Exception>
     </ows:ExceptionReport>
 
-The server responded with the :py:class:`pywps.exceptions.MissingParameterValue` 
-exception, telling us that the parameter `service` was not set. This is  
-compliant with the OGC WPS standard, since each request mast have at least the 
-`service` and `request` parameters. We can say for now, that this PyWPS 
-instance is properly deployed on the server, since it returns proper exception 
+The server responded with the :py:class:`pywps.exceptions.MissingParameterValue`
+exception, telling us that the parameter `service` was not set. This is
+compliant with the OGC WPS standard, since each request mast have at least the
+`service` and `request` parameters. We can say for now, that this PyWPS
+instance is properly deployed on the server, since it returns proper exception
 report.
 
 We now have to configure the instance by editing the `pywps.cfg` file and adding