Skip to content
(Google Summer of Code 2018) The project aims primarily is to add plugins in istSOS3 for unit of measure conversion. The user can convert unit in another specified unit using plugins. For Unit of measure conversion in istSOS3 we added postgresql-unit and pint libraries which has a powerful feature of unit conversion along with many unit conversi…
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.cache/v/cache
build/lib/istsos
dist
docker
docs
examples
gsoc
istsos.egg-info
istsos
screenshots
.travis.yml
CONTRIBUTING.md
DEVELOPING.md
LICENSE.md
MANIFEST.in
README.md
config-example.json
config.json
config.pickle
requirements.txt
setup.py

README.md

istSOS3

In this repository you can find the istSOS3 Python3 module. By following the instruction you will be able to run and use the core of istSOS3 on your machine. For full packaged versions please go to (todo: link of the dowload pages).

Quick Installation

To install istSOS3, you ca use pip:

pip3 install istsos

Installation guide for developers

Make a fork of the official repository if you think to contribute to the development, then clone your fork to your workstation.

If you want just to make some tests you can clone directly the official repository.

git clone https://github.com/istSOS/istsos3.git

Install all the dependencies

sudo apt-get install postgresql postgis python3-pip

And some Python3 dependencies:

sudo pip3 install lxml \
    jsonschema \
    python-dateutil \
    tornado \
    psycopg2 \
    aiopg \
    pytest

Preparing the istSOS3 server

Installa and create a PostgreSQL database named istsos3 and execute the sql file (postgresql_9.4_istsos_3.0.0.sql) that you can find in the istsos3/assets folder.

sudo -u postgres createdb -E UTF8 istsos3
sudo -u postgres psql \
  -d istsos3 -a \
  -f istsos3/istsos/assets/postgresql_9.4_istsos_3.0.0.sql

Set your postgres user password

sudo -u postgres psql
alter user postgres password 'postgres';

Ctrl-D to exit from psql console

Create the configuration file adapting the loader attribute.

cd istsos3
cp config-example.json config.json

Edit your config file:

{
    "type": "aiopg",
    "host": "localhost",
    "port": "5432",
    "user": "postgres",
    "password": "postgres",
    "database": "istsos3"
}

There is a basic server implementation using Townado Web

python3 examples/server_tornado.py

Listening at http://localhost:8887/sos

Testing

Lib used: https://docs.pytest.org/en/latest/

[#todo emprove this] For testing in the istsos/test/ folder there same tests that has been implemented.

Testing the SOS requests

in the folder istsos/test/actions/servers/sos_2_0_0/ you can find some examples. To run a single test, execute this from the terminal:

pytest -s istsos/test/actions/servers/sos_2_0_0/test_getCapabilitiesOp.py

or

pytest -s istsos/test/actions/servers/sos_2_0_0/test_describeSensorOp.py

Benchmark

[#todo emprove this] You can also do some basic benchmarking with the files in the examples/speed folder.

Maybe we can take a look at this tools: https://github.com/wg/wrk

python curl.py http://localhost/sos

or directly using curl:

curl -s -w '\ntime_namelookup=%{time_namelookup}\ntime_pretransfer=%{time_pretransfer}\ntime_starttransfer=%{time_starttransfer}\ntime_total=%{time_total}\n\n' -o /dev/null "http://the.request?to=test"

Documentation

In the docs folder there the sphings file that can be used to generate the docs html page.

To build the docs:

    cd docs
    make html

Continuous Integration (CI)

https://about.gitlab.com/features/gitlab-ci-cd/

The tests are executed automatically at each commit on the remote repository. The test are performed inside a docker environment.

Installation

It's possible to execute the test locally, to do that follow the steps below:

Install docker executor

Add the GPG key for the official Docker repository:

    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Add Docker repo to APT sources:

    sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

Install Docker

    sudo apt-get update
    sudo apt-get install docker-ce

Run docker without sudo

    sudo usermod -aG docker ${USER}
    su - ${USER}

Install gitlab-ci runner

Download the correct deb (gitlab-ci-multi-runner_amd64.deb) from the following uri:

    https://gitlab-ci-multi-runner-downloads.s3.amazonaws.com/v9.5.0/deb/gitlab-ci-multi-runner_amd64.deb

Install gitlab-ci-runner:

    sudo dkpg -i gitlab-ci-multi-runner_amd64.deb

Run test locally with gitlab-ci

Locally you can only run one test at time, the test run only on commited changes.

    cd git-root
    gitlab-runner exec docker [test_name]

Add new jobs to gitleb-ci

The .gitlab-ci.yml file defines sets of jobs with constraints of how and when they should be run. The jobs are defined as top-level elements with a name (in our case rest) and always have to contain the script keyword.

Each job run independently from each other. If a test fails the job will be stopped.

Example:

rest:
    stage: test

    script:
        - pytest -s istsos/test/actions/servers/rest/test_uom.py
        - pytest -s istsos/test/actions/servers/rest/test_observedProperties.py
        - pytest -s istsos/test/actions/servers/rest/test_material.py
        - pytest -s istsos/test/actions/servers/rest/test_method.py
        - pytest -s istsos/test/actions/servers/rest/test_offering.py
        - pytest -s istsos/test/actions/servers/rest/test_specimen.py

istSOS3 lib usage

Usage example:

[#todo to be improved]

import asyncio

from istsos.application import Server
from istsos.entity.httpRequest import HttpRequest

@asyncio.coroutine
def execute():
    server = yield from Server.create()
    request = HttpRequest(
        "GET",
        "sos",
        parameters={
            "service": "SOS",
            "version": "2.0.0",
            "request": "GetObservation",
            "offering": "T_LUGANO",
            "temporalFilter": (
                "om:phenomenonTime,"
                "2009-01-01T00:00:00+0100/"
                "2009-01-02T00:00:00+0100"
            )
        }
    )
    response = yield from server.execute_http_request(request, stats=True)
    print("\nLoaded %s observations" % len(response['observations']))

loop = asyncio.get_event_loop()
loop.run_until_complete(
   asyncio.gather(execute())
)
loop.close()

Initialize a new config file

Initializing the istSOS Server with a config dictionary. A new config file will be created in the given path (default to config.pickle).

from istsos.application import State

state = State(
    config={
        "proxy": "http://localhost/istsos3/",
        "cache": False,
        "loader": {
            "type": "aiopg",
            "host": "localhost",
            "port": "5432",
            "user": "postgres",
            "password": "postgres",
            "database": "istsos3"
        }
    }
)
You can’t perform that action at this time.