Skip to content
This repository has been archived by the owner on Nov 14, 2023. It is now read-only.

Latest commit

 

History

History
514 lines (337 loc) · 17.3 KB

README.rst

File metadata and controls

514 lines (337 loc) · 17.3 KB
Raw

Copernicus WPS Demo

Travis Build GitHub license

Demo with WPS processes for Copernicus.

Overview

In Copernicus we want to provide processing capabilities for a climate data store. The processing capabilities are exposed using the Web Processing Service standard interface with the PyWPS implementation. This demo contains several example processes implemented with ESMValTool and NCL. The processes are defined with different WPS profiles mainly to show ways how the input data of the tools can be described and restricted.

Installation

The installation is done with Buildout. It is using the Python distribution system Anaconda to maintain software dependencies.

If Anaconda is not available then a minimal Anaconda will be installed during the installation process in your home directory ~/anaconda.

The installation process setups a conda environment named copernicus. All additional packages are going into this conda environment. The location is ~/.conda/envs/copernicus.

Now, check out the code from the GitHub repo and start the installation:

$ git clone https://github.com/cp4cds/copernicus-wps-demo.git
$ cd copernicus-wps-demo
$ make clean install

After successful installation you need to start the services. All installed files (config etc ...) are by default in your home directory ~/birdhouse. Now, start the services:

$ make start  # starts supervisor services
$ make status # shows supervisor status

The deployed WPS service is available on http://localhost:8096/wps?service=WPS&version=1.0.0&request=GetCapabilities.

Check the log files for errors:

$ cd ~/birdhouse
$ tail -f  var/log/pywps/copernicus.log
$ tail -f  var/log/supervisor/copernicus.log

For other install options run make help and read the documention of the Makefile.

Configuration

If you want to run on a different hostname or port then change the default values in custom.cfg:

$ cd wps
$ vim custom.cfg
$ cat custom.cfg
[settings]
hostname = localhost
http-port = 8096
archive-root = /path/to/esgf/cmip5/archive

The path to ESGF archive is configured in custom.cfg with the archive-root option. The configuration file esgf_config.xml for the ESGF coupling module will be generated.

After any change to your custom.cfg you need to run make update (offline mode) or make install again and restart the supervisor service:

$ make install
$ make restart
$ make status

Running WPS service in test environment

For development purposes you can run the WPS service without Nginx and supervisor. Use the following instructions:

# get the source code
$ git clone https://github.com/cp4cds/copernicus-wps-demo.git
$ cd copernicus-wps-demo

# create conda environment
$ conda env create -f environment.yml

# activate conda environment
$ source activate copernicus-wps-demo

# install copernicus code into conda environment
$ python setup.py develop

# start the WPS service
$ copernicus

# open your browser on the default service url
$ firefox http://localhost:5000/wps

# ... and service capabilities url
$ firefox http://localhost:5000/wps?service=WPS&request=GetCapabilities

The copernicus service command-line has more options:

$ copernicus -h

For example you can start the WPS with enabled debug logging mode:

$ copernicus --debug

Or you can overwrite the default PyWPS configuration by providing your own PyWPS configuration file (just modifiy the options you want to change):

# edit your local pywps configuration file
$ cat mydev.cfg
[logging]
level = WARN
file = /tmp/mydev.log

# start the service with this configuration
$ copernicus -c mydev.cfg

WPS Client Examples

The examples for the WPS clients are using a demo WPS serivce on the bovec test machine at DKRZ.

https://bovec.dkrz.de/ows/proxy/copernicus?Service=WPS&Request=GetCapabilities&Version=1.0.0

There is currently not much data attached (especially for oberstation data). Please use the default data selection parameters, otherwise no result might be returned.

CURL with HTTP Get requests

GetCapabilities

Run GetCapabilities request to see which processes are available:

$ curl -s -o caps.xml \
  "https://bovec.dkrz.de/ows/proxy/copernicus?Service=WPS&Request=GetCapabilities&Version=1.0.0"

DescribeProcess

Run DescribeProcess request to see input/output parameters of the mydiag process:

$ curl -s -o describe.xml \
  "https://bovec.dkrz.de/ows/proxy/copernicus?Service=WPS&Request=DescribeProcess&Version=1.0.0&identifier=mydiag"

Execute (sync mode)

Run Exceute in synchronous mode for mydiag with default input parameters:

$ curl -s -o execute.xml \
  "https://bovec.dkrz.de/ows/proxy/copernicus?Service=WPS&Request=Execute&Version=1.0.0&identifier=mydiag&DataInputs=model=MPI-ESM-LR;experiment=historical;ensemble=r1i1p1;start_year=2000;end_year=2001"

Warning

The execution request my have a time-out. Please use the asynchronous mode for real testing.

A status document is returned. Open the URL with the reference to the output plot:

<wps:Output>
  <ows:Identifier>output</ows:Identifier>
  <wps:Reference xlink:href="https://bovec.dkrz.de/download/wpsoutputs/copernicus/75ad4c42-207b-11e8-9a75-109836a7cf3a/ta_u_y2we.pdf" mimeType="application/pdf"/>
</wps:Output>

Execute (async mode)

Run Exceute in asynchronous mode for mydiag:

$ curl -s -o execute.xml \
  "https://bovec.dkrz.de/ows/proxy/copernicus?Service=WPS&Request=Execute&Version=1.0.0&identifier=mydiag&DataInputs=model=MPI-ESM-LR;experiment=historical;ensemble=r1i1p1;start_year=2000;end_year=2001&storeExecuteResponse=true&status=true"

A status document is returned.

<wps:ExecuteResponse
  statusLocation="https://bovec.dkrz.de/download/wpsoutputs/copernicus/621d8f28-207d-11e8-99d0-109836a7cf3a.xml">
  <wps:Process wps:processVersion="2.0.0">
    <ows:Identifier>mydiag</ows:Identifier>
    <ows:Title>Simple plot</ows:Title>
    <ows:Abstract>Generates a plot for temperature using ESMValTool. It is a diagnostic used in the ESMValTool tutoriaal doc/toy-diagnostic-tutorial.pdf. The default run uses the following CMIP5 data: project=CMIP5, experiment=historical, ensemble=r1i1p1, variable=ta, model=MPI-ESM-LR, time_frequency=mon</ows:Abstract>
  </wps:Process>
  <wps:Status creationTime="2018-03-05T14:59:12Z">
    <wps:ProcessAccepted>PyWPS Process mydiag accepted</wps:ProcessAccepted>
  </wps:Status>
</wps:ExecuteResponse>

Check the status document given by the statusLoction URL until the job has finished:

$ curl -s -o status.xml \
  "https://bovec.dkrz.de/download/wpsoutputs/copernicus/621d8f28-207d-11e8-99d0-109836a7cf3a.xml"

The final status document should similar to this one:

<wps:ExecuteResponse
  statusLocation="https://bovec.dkrz.de/download/wpsoutputs/copernicus/621d8f28-207d-11e8-99d0-109836a7cf3a.xml">
<wps:Process wps:processVersion="2.0.0">
 <ows:Identifier>mydiag</ows:Identifier>
</wps:Process>
<wps:Status creationTime="2018-03-05T14:59:27Z">
 <wps:ProcessSucceeded>PyWPS Process Simple plot finished</wps:ProcessSucceeded>
</wps:Status>
<wps:ProcessOutputs>
 <wps:Output>
   <ows:Identifier>output</ows:Identifier>
   <wps:Reference xlink:href="https://bovec.dkrz.de/download/wpsoutputs/copernicus/621d8f28-207d-11e8-99d0-109836a7cf3a/ta_ZqKo5r.pdf" mimeType="application/pdf"/>
 </wps:Output>
</wps:ProcessOutputs>
</wps:ExecuteResponse>

Open the URL pointing to the plot output.

OWSLib Python module

OWSLib is a Python library to interact with OWS/OGC services like WPS, WMS, etc. It is using the Python requests library.

.. todo::
  Add IPython notebook.

http://birdhouse-workshop.readthedocs.io/en/latest/advanced/owslib.html

Birdy Command line client

Birdy is a WPS command line client.

Install birdy (Linux, macOS):

$ conda install -c birdhouse -c conda-forge birdhouse-birdy

Set WPS service:

$ export WPS_SERVICE=https://bovec.dkrz.de/ows/proxy/copernicus # demo service on bovec
# OR
$ export WPS_SERVICE=http://localhost:8096/wps  # your local WPS service

See what processes are available:

$ birdy -h

Run mydiag:

$ birdy mydiag -h
$ birdy mydiag --model MPI-ESM-LR --experiment historical --ensemble r1i1p1 --start_year 2000 --end_year 2001

Check the process status. The processes should finish after 10 seconds with a response simliar to this one:

[ProcessSucceeded 100/100] PyWPS Process Simple plot finished
Output:
namelist=https://bovec.dkrz.de/download/wpsoutputs/copernicus/eceb7ef8-2078-11e8-acba-109836a7cf3a/namelist_sr0bYf.yml (text/plain)
log=https://bovec.dkrz.de/download/wpsoutputs/copernicus/eceb7ef8-2078-11e8-acba-109836a7cf3a/main_log_fXjRR1.txt (text/plain)
output=https://bovec.dkrz.de/download/wpsoutputs/copernicus/eceb7ef8-2078-11e8-acba-109836a7cf3a/ta_XWGYZt.pdf (application/pdf)

Open the ouptut URL in Browser to see the plot.

Phoenix Web Client

You can run the demo processes directly without log-in on Phoenix.

Job status is monitored. When job has finished you can either show the output directly or show the output details: https://bovec.dkrz.de/monitor/details/79eb1c39-e6a3-4944-b90d-00cc71addcaf/outputs

WPS Client Examples with x509 Certificate

A WPS service can be secured with x509 certificates by using the Twitcher OWS security proxy. A WPS Execute request can only be run when the WPS client provides a valid x509 proxy certificate.

In the following examples we will use a Copernicus WPS demo service which is protected by a Twitcher security proxy. It will only accept x509 proxy certificates from ESGF to execute a process. The GetCapabilites and DescribeProcess requests are public.

CURL with HTTP Get requests

The following examples are using curl. You may also like to use the Firefox RestClient plugin.

GetCapabilities

Run GetCapabilities request to see which processes are available:

$ curl -s -o caps.xml \
  "https://bovec.dkrz.de:5000/ows/proxy/copernicus?Service=WPS&Request=GetCapabilities&Version=1.0.0"

DescribeProcess

Run DescribeProcess request to see input/output parameters of the mydiag process:

$ curl -s -o describe.xml \
  "https://bovec.dkrz.de:5000/ows/proxy/copernicus?Service=WPS&Request=DescribeProcess&Version=1.0.0&identifier=mydiag"

Execute (sync mode)

Run Exceute in synchronous mode for mydiag with default input parameters:

$ curl -s -o execute.xml \
  "https://bovec.dkrz.de:5000/ows/proxy/copernicus?Service=WPS&Request=Execute&Version=1.0.0&identifier=mydiag&DataInputs=model=MPI-ESM-LR;experiment=historical;ensemble=r1i1p1;start_year=2000;end_year=2001"

You should get an exception report asking you to provide a x509 certificate:

<ExceptionReport>
  <Exception exceptionCode="NoApplicableCode" locator="AccessForbidden">
    <ExceptionText>A valid X.509 client certificate is needed.</ExceptionText>
  </Exception>
</ExceptionReport>

Get a valid x509 certifcate from ESGF, for example using the esgf-pyclient. See the logon example. Let's say your proxy certificate is in the file cert.pem. Run the curl example above with this certificate:

$ curl -s -o execute.xml --cert cert.pem --key cert.pem \
  "https://bovec.dkrz.de:5000/ows/proxy/copernicus?Service=WPS&Request=Execute&Version=1.0.0&identifier=mydiag&DataInputs=model=MPI-ESM-LR;experiment=historical;ensemble=r1i1p1;start_year=2000;end_year=2001"

If your certificate is valid then your process will be executed (sync mode) and you will get a XML result document providing you with URL references to a generated plot:

<wps:Output>
  <ows:Identifier>output</ows:Identifier>
  <wps:Reference xlink:href="http://bovec.dkrz.de:8000/wpsoutputs/copernicus/3c0eb52e-2608-11e8-b551-dea873cae3fc/ta_P9MhKW.pdf" mimeType="application/pdf"/>
</wps:Output>

Try more examples as shown in the examples above using a x509 certificate.

Using Python requests library

In this example we show how you can use the Python requests library to run WPS requests.

import requests

# GetCapabilites
url = "https://bovec.dkrz.de:5000/ows/proxy/emu?request=GetCapabilities&service=WPS"
requests.get(url, verify=True).text
# DescribeProcess
url = "https://bovec.dkrz.de:5000/ows/proxy/emu?request=DescribeProcess&service=WPS&version=1.0.0&identifier=hello"
requests.get(url, verify=True).text
# Execute with client certifcate cert.pem
url = "https://bovec.dkrz.de:5000/ows/proxy/emu?request=Execute&service=WPS&version=1.0.0&identifier=hello&DataInputs=name=Copernicus"
requests.get(url, cert="cert.pem" verify=True).text

See the requests documentation for details.

Using OWSLib Python library

An example with OWSLib.

Make sure you have the latest version from the conda birdhouse channel:

$ conda install -c birdhouse -c conda-forge owslib

Run the hello process with a client certificate:

from owslib.wps import WebProcessingService
wps = WebProcessingService(url="https://bovec.dkrz.de:5000/ows/proxy/emu",
                           verify=True, cert="cert.pem")

exc = wps.execute(identifier='hello', inputs=[('name', 'tux')])
exc.checkStatus()
exc.getStatus()
exc.isSucceded()
exc.processOutputs[0].data

Using Birdy

An example with Birdy

Install latest birdy (Linux, macOS) from conda birdhouse channel:

$ conda install -c birdhouse -c conda-forge birdhouse-birdy owslib
$ export WPS_SERVICE=https://bovec.dkrz.de:5000/ows/proxy/emu
$ birdy -h
$ birdy hello -h
$ birdy --cert cert.pem hello --name Copernicus

Using Docker

Get docker images using docker-compose:

$ docker-compose pull

Start the demo with docker-compose:

$ docker-compose up -d  # runs with -d in the background
$ docker-compose logs -f  # check the logs if running in background

By default the WPS service should be available on port 8080:

$ firefox "http://localhost:8080/wps?service=wps&request=GetCapabilities"

Alternatively you can change the port by using environment variables, for example:

$ HTTP_PORT=8096 docker-compose up -d # wps service will be available on port 8096

Run docker exec to watch logs:

$ docker ps     # find container name
copernicuswpsdemo_wps_1
$ docker exec copernicuswpsdemo_wps_1 tail -f /opt/birdhouse/var/log/supervisor/copernicus.log
$ docker exec copernicuswpsdemo_wps_1 tail -f /opt/birdhouse/var/log/pywps/copernicus.log

Use docker-compose to stop the containers:

$ docker-compose down

Testdata

For the demo processes you can fetch CMIP5 test-data from the ESGF archive. You need a valid ESGF credentials which you can fetch for example with esgf-pyclient.

For the examples you need CMIP5 data with the following facets:

  • project=CMIP5
  • experiment=historical
  • ensemble=r1i1p1
  • variable=ta, tas, or pr
  • model=MPI-ESM-LR
  • time_frequency=mon

You can use wget to download ESGF NetCDF files (-x option to create directories):

$ wget --certificate cert.pem --private-key cert.pem --ca-certificate cert.pem -N -x -P /path/to/esgf/cmip5/archive