Skip to content

Commit

Permalink
Release washington (#29)
Browse files Browse the repository at this point in the history 
* added birdy and emu

* added cookiecutter to release

* added malleefowl

* added hummingbird to release notes

* added ansible to release

* updated phoenix, twitcher, fp, finch, kingfisher.

* updated files and folder section in architecture

* include finch& kingfisher in list, change dedication for flyingpigeon

* admin guide ecosystem update

* finch description

* include an user guideline (placeholer)

* fixed links and rough update of admin guide.

* add files and folders
  • Loading branch information
@cehbrecht @nilshempelmann
cehbrecht authored and nilshempelmann committed Dec 9, 2018
1 parent a17df42 commit ba24bc5
Show file tree
Hide file tree
Showing 6 changed files with 177 additions and 89 deletions.
124 changes: 58 additions & 66 deletions docs/source/admin_guide.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Administrator Guidelines
:depth: 2


.. warning:: This section is outdated ...
.. warning:: This section needs is outdated and needs to be rewritten!

.. _birdhouse_ecosystem:

Expand All @@ -18,79 +18,54 @@ Set up a birdhouse ecosystem server
If you are already familiar with installing single standalone WPS (follow the :ref:`installation` guides in the documentations of e.g. emu), then you are ready to set up a birdhouse containing flyingpigeon (providing scientific analyses methods), malleefowl (to search and fetch data) and the pheonix (a graphic interface for a web browser including a WMS).

General Remarks
...............
~~~~~~~~~~~~~~~

| Check the :ref:`requirements` of your system!
| The installation is done as **normal user**, root rights are causing conflicts.

Clone the Repositories from GitHub
..................................
Prepare Installation
~~~~~~~~~~~~~~~~~~~~

It is recommended to collect the repositories in a seperate folder (e.g. birdhouse, but can have a name of your choice)::
It is recommended to collect the repositories in a separate folder (e.g. birdhouse, but can have a name of your choice)::

$ mkdir birdhouse
$ cd birdhouse


* **fetch the source code:**
Get the source code from GitHub
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code-block:: ini
$ git clone https://github.com/bird-house/flyingpigeon.git
$ git clone https://github.com/bird-house/pyramid-phoenix.git
$ git clone https://github.com/bird-house/malleefowl.git
* **phoenix password**

To be able to log into the Phoenix GUI once the services are running, it is necessary to generate a password:
go into the pyramid-phoenix folder and run::

$ make passwd

This will automatically write a password hash into pyramid-phoenix/custom.cfg


* **installation**
Run Installation
~~~~~~~~~~~~~~~~

You can run the installation with default settings.
It will create an anaconda environment into your HOME direcory and deploy all required software dependecies there.
*read the ''changing the default configuration' first if you would like to change the defaults.*
It will create a conda environment and deploy all required software dependencies there.

.. note:: Read the *changing the default configuration* if you want to customize the configuration.

In **all** of the tree folders (malleefowl, flyingpigeon and pyramid-phoenix) run::

$ make install

This installation will take some minutes to fetch all dependencies and install them into seperate conda environments.
With the default settings, the installation creates the following folders::

$ ls ~/anaconda/

contains general software required by anaconda::

$ ls ~/.conda/envs/

contains the seperate environments of the birds for their specific software dependencies::
This installation will take some minutes to fetch all dependencies and install them into separate conda environments.

$ ls ~/birdhouse/var/
Start the Services
~~~~~~~~~~~~~~~~~~

the local cache for fetched input files, output files and logs. This folder is growing (while fetching files and storing job outputs) under productive usage of birdhouse.

* **start the services**

in **one** of the birds run::
in **all** of the birds run::

$ make start

or::

$ make restart

and to check if the services are running, run::

$ make status

* **launching the Phoenix GUI**
Launching the Phoenix Web App
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If the services are running, you can launch the GUI in a common web browser. By default, phoenix is set to port 8081::

Expand All @@ -103,42 +78,51 @@ or::
Now you can log in (upper right corner) with your Phoenix password created previously.
Phoenix is just a graphical interface with no more function than looking nice ;-).

* **register a service in the GUI**
Register a service in Phoenix Web App
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. note:: Please read the `Phoenix documentation <https://pyramid-phoenix.readthedocs.io/en/latest/user_guide.html#>`_

Your first administrator step is to register flyingpigeon as a service. For that, log in with your phoenix password.
In the upper right corner is a tool symbol to open the 'settings'. Click on 'Services' and the 'Register a Service'.
Your first administration step is to register *flyingpigeon* as a service.
For that, log in with your phoenix password.
In the upper right corner is a tool symbol to open the `settings`.
Click on `Services` and the `Register a Service`.

flyingpigeon is per default at port 8093.
Flyingpigeon is per default on port 8093.

the appropriate url is::
The appropriate url is::

http://localhost:8093/wps

Provide service title and name as you like:
Service Title: Flyingpigeon
Service Name: flyingpigeon
* Service Title: Flyingpigeon
* Service Name: flyingpigeon

check 'Service Type' : 'Web Processing Service' (default) and register.
check `Service Type`: `Web Processing Service` (default) and register.

Optionally, you can check 'Public access?', to allow unregistered users to launch jobs. (**NOT recommended**)
Optionally, you can check `Public access?`, to allow unregistered users to launch jobs. (**NOT recommended**)

Launching a Job
~~~~~~~~~~~~~~~

* **launching a job**

Now your birdhouse ecosysem is set up. The also installed malleefowl is already running in the background and will do a lot of work silently. Ther is **no need to register malleefowl** manually!
Now your birdhouse ecosysem is set up.
The also installed malleefowl is already running in the background and will do a lot of work silently.
There is **no need to register malleefowl** manually!

Launching a job can be performed as a process (Process menu) or with the wizard. To get familliar with the processes provided by each of the birds, read the approriate documentation for each of the services listed in the `overview: <http://birdhouse.readthedocs.io/en/latest/index.html>`_

* **changing the default configuration:**

The default configuration can be changed by creating a Makefile.config file. There is an example provided to be used::
Changing the default configuration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. _note: files and folder section (architecture chapter)

$ cp Makefile.config.example Makefile.config
You can customize the configuration of the service. Please read the documentation, for example:

and set the appropriate path. You have to **do this in all** bird repositories.
* `Phoenix documentation <https://pyramid-phoenix.readthedocs.io/en/latest/configuration.html>`_
* `Flyingpigeon documentation <https://flyingpigeon.readthedocs.io/en/latest/configuration.html>`_

Furthermore, you might change the hostname (to provide your service to the outside), ESGF-node connection, the port or the log-level for more/less information in the administrator logfiles.
Here is an example pyramid-phoenix/custom.cfg:
Furthermore, you might change the hostname (to make your service accessible from outside), ESGF-node connection,
the port or the log-level for more/less information in the administrator logfiles.
Here is an example `pyramid-phoenix/custom.cfg`:

.. code-block:: ini
Expand All @@ -156,12 +140,15 @@ Here is an example pyramid-phoenix/custom.cfg:
wps-url = http://localhost:8091/wps
* **Administration HELP:**
Update Phoenix Password
~~~~~~~~~~~~~~~~~~~~~~~

In case of questions or trouble shooting, feel welcome to join the birdhouse chat and get into contact with the developers directly:
To be able to log into the Phoenix GUI once the services are running, it is necessary to generate a password:
go into the pyramid-phoenix folder and run::

`Birdhouse-Chatroom <https://gitter.im/bird-house/birdhouse>`_
$ make passwd

This will automatically write a password hash into pyramid-phoenix/custom.cfg

.. _backups:

Expand All @@ -173,4 +160,9 @@ With the following command you can make a dump of the ``users`` collection of th

$ mongodump --port 27027 --db phoenix_db --collection users

-->
Asking for Support
------------------

In case of questions or trouble shooting, feel welcome to join
the `birdhouse chat <https://gitter.im/bird-house/birdhouse>`_
and get into contact with the developers directly.
71 changes: 50 additions & 21 deletions docs/source/architecture.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,50 +54,79 @@ Server Side Components

WPS services for climate data analysis:

* `Flyingpigeon`_: services for the climate impact community
* Emu_: some example WPS processes for demo
* Flyingpigeon_: Testbed for new process development
* `Black Swan`_: services for the extreme weather event assessments
* `Hummingbird`_: provides cdo and compliance-checker as a service
* `Emu`_: some example WPS processes for demo
* Hummingbird_: provides cdo and compliance-checker as a service
* Finch_: services for climate indices calculation
* Kingfisher_: Services for Earth-Observation data analysis

Many climate analysis operations are implemented using OpenClimateGIS_
including the `python package icclim <http://icclim.readthedocs.io/en/latest/>`_.

Supporting Services and libraries:

* `Twitcher`_: an OWS Security Proxy
* `Malleefowl`_: access to climate data (ESGF, ...) as a service
* `Eggshell`_: provides common functionallity for Birdhouse WPS services
* Twitcher_: an OWS Security Proxy
* Malleefowl_: access to climate data (ESGF, ...) as a service
* Eggshell_: provides common functionallity for Birdhouse WPS services

You can find the source code of all birdhouse components on GitHub_.
Docker images with birdhouse components are on `Docker Hub`_

.. _filesandfolder:

Files and Folders
-----------------
.................

.. _note: See also administrator guidelines

This is an overview of folder structure and important files for administration of a server-side birdhouse ecosystem.

It is recommended to clone the separated WPS services (birds) into one top level folder like:

.. code-block:: sh
$ ~/birdhouse/emu
$ ~/birdhouse/pyramid-pheonix
$ ~/birdhouse/finch
$ ~/birdhouse/malleefowl
...
The dependencies of each bird is deployed as `conda environment` and per default located at:

.. code-block:: sh
$ ~/.conda/envs/
The environment of a bird is defined in `./{birdname}/environment.yml`.

.. warning:: This section is outdated. We are moving to a new deployment without Buildout.
Process descriptions are placed in `./{birdname}/{birdname}/processes/` while modules designed and used for the service
are situated in `./{birdname}/{birdname}/`. Here are also static data like shapefiles, templates or additional data used by the processes.

The birds have a similar folder structure. While library dependencies are stored within the conda deployment
.. code-block:: sh
$ ./{birdname}/{birdname}/data/shapefiles
$ ./{birdname}/{birdname}/templates
Three folder locations have to be pointed out:
Each birdhouse compartment has a documentation build with `Sphinx` and the corresponding files are situated in

* **repository clones:** The fetched code by ``git clone``. It is recommended to store the repositories in ``~/birdhouse``
* **anaconda**: By default, the installation process creates a folder ``~/anaconda`` for general anaconda-specific software.
* **conda environments:** All birds (repositories) are built with their own environment to avoid missmatch of dependencies.
By default, the conda environments are in ``~/.conda/envs/``.
.. code-block:: sh
To change the default settings, create a ``Makefile.config`` with::
$ ./{birdname}/docs
$ cp Makefile.config.example Makefile.config
When running a service, files and folders for input data, result storage, file cache of simply logfiles
are defined in the `./{birdname}/.config.cfg`. Default configuration is defined in `./{birdname}/{birdname}/default.cfg`
as well as an example can be found in `~./{birdname}/etc`.
For more options of configuration see the `pywps configuration instructions <https://pywps.readthedocs.io/en/master/configuration.html>`_

and change the paths accordingly to your needs.
For development and deployment testing the installations be checked running tests (`make test`). Test descriptions testdata
are situated in:

Furthermore, in ``environment.yml``, the conda packages can be defined. It is recommended to pin the version. The bird-specific packages are defined here, while in ``requirements/conda_pinned``, general versions are set.
.. code-block:: sh
There are **log files** situated at:: ``~/birdhouse/var/log/pywps/``
$ ./{birdname}/tests
$ ./{birdname}/tests/testdata
.. _note:: See also the administration guideline to set up a birdhouse ecosystem.
.. _todo:: add locations of mongodb celery etc...

.. _GitHub: https://github.com/bird-house
.. _`Docker Hub`: https://hub.docker.com/r/birdhouse
2 changes: 2 additions & 0 deletions docs/source/conf.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -339,6 +339,8 @@
.. _Hummingbird: http://birdhouse-hummingbird.readthedocs.io/en/latest/
.. _Emu: http://emu.readthedocs.io/en/latest/
.. _Black Swan: https://github.com/bird-house/blackswan
.. _Finch: https://github.com/bird-house/finch
.. _Kingfisher: https://kingfisher.readthedocs.io/en/latest/
.. _Birdy: http://birdy.readthedocs.io/en/latest/
.. _Cookiecutter: http://cookiecutter-birdhouse.readthedocs.io/en/latest/
.. _Eggshell: https://eggshell.readthedocs.io/en/latest/
Expand Down
1 change: 1 addition & 0 deletions docs/source/guidelines.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,3 +12,4 @@ To guide you through the learning curve of installation modules of birdhouse and
installation
admin_guide
dev_guide
user_guide
36 changes: 34 additions & 2 deletions docs/source/release_notes.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,41 @@ Release Notes
Washington (December 2018, v0.6.1)
==================================

Birdhouse will be at the `AGU 2018 in Washington D.C. <https://fallmeeting.agu.org/2018/>`_.
Birdhouse was present at the
`AGU 2018 <https://fallmeeting.agu.org/2018/>`_
and
`ESGF Face to Face 2018 <https://esgf.llnl.gov/2018-F2F.html>`_
both in Washington D.C.

.. warning:: In the making ...
Highlighted Changes:

* Improved *Birdy* `WPSClient` as a pythonic library for WPS client with support for Jupyter Notebooks.
* Converted *Malleefowl* and *FlyingPigeon* to new deployment layout without buildout.
* New birds: *Finch* WPS for Climate Indicators and *Kingfisher* for Earth Observation Data Analysis.
* *FlyingPigeon* has been reborn as the *Curious Climate Explorer*. Most of its original functionallity
has moved to other birds: *BlackSwan*, *Kingfisher* and *Finch*.

Released Birds:

* Ansible Playbook for PyWPS `0.2.0 <https://github.com/bird-house/ansible-wps-playbook/releases/tag/v0.2.0>`_
* Cookiecutter Template for PyWPS `0.3.1 <https://github.com/bird-house/cookiecutter-birdhouse/releases/tag/v0.3.1>`_
* Birdy WPS Client: `0.5.0 <https://github.com/bird-house/birdy/releases/tag/v0.5.0>`_
* Emu WPS: `0.9.1 <https://github.com/bird-house/emu/releases/tag/v0.9.1>`_
* Hummingbird WPS: `0.6.1 <https://github.com/bird-house/hummingbird/releases/tag/v0.6.1>`_
* Malleefowl WPS: `0.7.0 <https://github.com/bird-house/malleefowl/releases/tag/v0.7.0>`_

Maintained Birds with Buildout:

* Phoenix Web App: `0.8.3 <https://github.com/bird-house/pyramid-phoenix/releases/tag/v0.8.3>`_
* Twitcher WPS Proxy: `0.3.8 <https://github.com/bird-house/twitcher/releases/tag/v0.3.8>`_

New Birds in the making:

* FlyingPigeon (reborn): https://github.com/bird-house/flyingpigeon
* Kingfisher: https://github.com/bird-house/kingfisher
* Finch: https://github.com/bird-house/finch
* Black Swan: https://github.com/bird-house/blackswan
* Eggshell: https://github.com/bird-house/eggshell

Dar es Salaam (September 2018, v0.6.0)
======================================
Expand Down
32 changes: 32 additions & 0 deletions docs/source/user_guide.rst
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
.. _user_guide:

User Guidelines
===============

.. contents::
:local:
:depth: 2

.. warning::
Work in progress. Examples will come soon.

You can connect to a WPS service in the following ways:

* using a command-line tool in your terminal.
* using a web based application from your browser.
* using a Python library from a Jupyter notebook or your Python scripts.

Command-line
------------

.. todo:: birdy example

Phoenix Web App
---------------

.. todo:: Screen-shot of Phoenix

Python Library
--------------

.. todo:: Example of a python call in a Notebook

0 comments on commit ba24bc5

Please sign in to comment.