aggregation of documentation (#44)

* architectue to framework
* test with fetch extern files
* include pelican
* include aggregation extention
* git doc fetch directive partially working.
* added toctree directive. renamed include directive.
* remove link to pavics backend
* doc aggregation developer guide
* git directives doc
* rewording
* rename title architecture to project

.gitignore

@@ -47,6 +47,7 @@ docs/make.bat
 docs/doctrees/
 docs/html/
 docs/build/
+docs/source/_gitext/
 
 # External Sources
 #src/external

docs/source/communication.rst

@@ -32,15 +32,11 @@ Blog-post
 
 In the blog_ you can find interesting articles and information related to birdhouse in general.
 We also inform regularly abut the main steps forward in the software development that you can keep track on whats going on in the birdhouse.
-If you want to receive a notification of new articles follow birdhouse news on our blog_.
-
-Special issues:
-...............
+If you want to receive a notification of new articles follow birdhouse news on our blog_:
 
 * `The IT Landscape for Climate Services  <https://medium.com/birdhouse-newsletter/the-it-landscape-for-climate-services-4e21c32c4ffb>`_
 * `Cyberinfrastructures for Sustainable Development <https://medium.com/birdhouse-newsletter/cyber-structures-for-sustainable-development-74b3e4deeff1>`_
 
-.. todo:: insert follow image
 
 Newsletter
 ----------

docs/source/conf.py

@@ -19,6 +19,7 @@
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
 # sys.path.insert(0, os.path.abspath('.'))
+sys.path.append(os.path.abspath("./ext"))
 
 # -- General configuration ------------------------------------------------
 
@@ -34,6 +35,7 @@
     'sphinx.ext.viewcode',
     'sphinxcontrib.bibtex',
     'sphinx.ext.graphviz',
+    'docaggregation'
 ]
 
 # autoapi_type = 'python'
@@ -44,6 +46,8 @@
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
+intersphinx_mapping = {'pavics': ('https://raw.githubusercontent.com/Ouranosinc/pavics-sdi/master/docs/source/', None)}
+
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 # source_suffix = ['.rst', '.md']
@@ -316,6 +320,7 @@
                        'flyingpigeon': ('https://flyingpigeon.readthedocs.io/en/latest/', None),
                        'hummingbird': ('https://birdhouse-hummingbird.readthedocs.io/en/latest/', None),
                        'emu': ('https://emu.readthedocs.io/en/latest/', None),
+                       'pelican': ('https://github.com/bird-house/pelican', None),
                        'birdy': ('https://birdy.readthedocs.io/en/latest/', None),
                        'bootstrap': ('https://birdhousebuilderbootstrap.readthedocs.io/en/latest/', None),
                         }
@@ -338,6 +343,7 @@
 .. _Twitcher: http://twitcher.readthedocs.io/en/latest/
 .. _Flyingpigeon: http://flyingpigeon.readthedocs.io/en/latest/
 .. _Hummingbird: http://birdhouse-hummingbird.readthedocs.io/en/latest/
+.. _Pelican: https://github.com/bird-house/pelican
 .. _Emu: http://emu.readthedocs.io/en/latest/
 .. _Black Swan: https://github.com/bird-house/blackswan
 .. _Finch: https://github.com/bird-house/finch

docs/source/dev_guide.rst

@@ -129,7 +129,25 @@ It is important to keep the :ref:`codestyle` and write explanations to your func
 
 The main `documentation`_ (which you are reading now) is the starting point to
 get an overview of birdhouse. Each birdhouse component comes with
-its own Sphinx documentation and is referenced by the main birdhouse document.
+its own Sphinx documentation and is referenced by the main birdhouse document. Projects using birdhouse components like PAVICS_ or `COPERNICUS Data Store`_ generally have their own documentation as well. To include documentation from external repository here, two custom made sphinx directives can be used. The `gittoctree` directive behaves like a normal table of content directive (`toctree`), but takes as an argument the URL to the git repo and refers to files inside this directory through their full path. The `gitinclude` directive acts like an normal `include` directive, but takes as a first argument the URL to the git repo this file belongs to. For example:
+
+.. code-block:: sphinx
+   :linenos:
+
+   Here is the text of the birdhouse main documentation. At the place where you want to integrate
+   a part of a remote sphinx documentation stored in a `git` repository you can fetch the docs
+   parts and integrated it with a table of content referring to external files:
+
+   .. gittoctree:: https://github.com/Ouranosinc/pavics-sdi.git
+
+      docs/source/arch/backend.rst
+
+   or include an individual file: 
+
+   .. gitinclude:: https://github.com/Ouranosinc/pavics-sdi.git docs/source/arch/backend.rst
+
+   The directive will clone and checkout the repository, then include these external files as if they were part of the native documentation. 
+
 
 .. _codestyle:
 
@@ -228,7 +246,7 @@ out of the release cycles bug fix patches can be released every time ( communica
 
 * patch v1.1.1_patch1 bugfix
 
-
+.. _`COPERNICUS Data Store`: https://cds.climate.copernicus.eu/#!/home
 .. _`OSGeo Code of Conduct`: http://www.osgeo.org/code_of_conduct
 .. _`documentation`: https://github.com/bird-house/birdhouse-docs
 .. _`GitHub`: https://github.com/bird-house

docs/source/ext/docaggregation.py

@@ -0,0 +1,78 @@
+import os, re, subprocess
+from docutils.parsers.rst.directives.misc import Include as BaseInclude
+from sphinx.directives import TocTree
+from sphinx.util.docutils import SphinxDirective
+
+
+def checkout(env, repo, branch='master'):
+    """Checkout git repository and return the relative path to the source directory."""
+
+    cache = os.path.join(env.srcdir, '_gitext')
+    root = os.path.join(cache, re.sub('[\W\-]+', '_', repo))
+
+    # Clone the repo
+    if not os.path.exists(root):
+        os.makedirs(root)
+        subprocess.run(['git', 'clone', '--depth=1', repo, root])
+
+    # Fetch latest version and checkout branch
+    subprocess.run(['git', 'fetch', repo], cwd=root)
+    subprocess.call(['git', 'co', branch], cwd=root)
+
+    return os.path.relpath(root, start=env.srcdir)
+
+
+class GitTocTree(TocTree):
+    """Table of content directive drawing from a git repository.
+
+    Example
+    -------
+    .. gittoctree:: <https://url_to_git_repo>
+
+       <relative_path_1>
+       <relative_path_2>
+
+    """
+    required_arguments = 1
+
+    def run(self):
+        env = self.state.document.settings.env
+        root = checkout(env, self.arguments[0])
+
+        # Link the content to the cloned repository
+        self.content = [os.path.join(root, content) for content in self.content]
+
+        return super().run()
+
+
+class GitInclude(BaseInclude, SphinxDirective):
+    """Include directive for file fetched from a git repository.
+
+    Arguments
+    ---------
+    repo : str
+      HTTP url to the git repository.
+    path : str
+      Relative path to the file inside the repository.
+
+    Example
+    -------
+
+    .. gitinclude:: <https://url_to_git_repo> <relative path to file>
+    """
+    required_arguments = 2
+
+    def run(self):
+        env = self.state.document.settings.env
+        root = checkout(env, self.arguments[0])
+
+        # Point path to the file in the cloned repo
+        rel_filename, filename = self.env.relfn2path(os.path.join(root, self.arguments[1]))
+        self.arguments = [filename]
+        self.env.note_included(filename)
+        return super().run()
+
+
+def setup(app):
+    app.add_directive('gitinclude', GitInclude)
+    app.add_directive('gittoctree', GitTocTree)

docs/source/architecture.rst → docs/source/framework.rst

@@ -1,42 +1,39 @@
-.. _birdhouse_overview:
+.. _framework:
 
-
-Architecture
-============
+*******************
+birdhouse framework
+*******************
 
 .. contents::
     :local:
     :depth: 2
 
 
-Framework Overview
-------------------
+Birdhouse is organized in separate stand-alone software components. Most components are named after birds, which gives the  project its name birdhouse. The components can be categorized into :ref:`client_components`, i.e. tools for end-users, and :ref:`server_components`, i.e. back-end elements of the architecture.
 
-.. image:: _images/framework.png
 
-ESGF_ is currently the main climate data resource (but more resources are possible).
-ESGF Solr-index is used to find ESGF data.
-The ESGF identity provider with OpenIDs and X509 certificate is used for authentication.
+.. _framework_structure:
+
+Framework structure
+-------------------
 
 There are several WPS services. Malleefowl_ is the main one for the Phoenix_ client.
 Malleefowl is used to search, download (with caching) ESGF data and to retrieve certificates.
 Malleefowl has also a workflow engine (dispel4py_) to chain WPS processes.
 
 The results of the WPS processes are stored on the file system and are accessible via URL (with a token id).
-
 Results can be shown on a Map using a Web Mapping Service (ncWMS, adagucserver).
-
 The PyCSW Catalog Service is used to register WPS services and also to publish WPS outputs.
 Published results in the PyCSW can also used as input source for processes again.
 
-WPS serivces can be accessed through web-applications like Phoenix or from scripts.
+ESGF_ is currently the main climate data resource (but more resources are possible).
+ESGF Solr-index is used to find ESGF data. The ESGF identity provider with OpenIDs and X509 certificate is used for authentication.
 
-.. note:: See also the `Birdhouse Presentation`_.
+WPS serivces can be accessed through web-applications like Phoenix or from scripts.
 
-.. _Birdhouse Presentation: https://github.com/bird-house/birdhouse-presentation
+.. image:: _images/framework.png
 
-Birdhouse is the home of Web Processing Services used in climate science and
-components to support them (the birds):
+.. note:: See also the :ref:`Publications and Presentations <publications>` for more information and details.
 
 
 .. _client_components:
@@ -59,6 +56,7 @@ WPS services for climate data analysis:
 * `Black Swan`_: services for the extreme weather event assessments
 * Hummingbird_: provides cdo and compliance-checker as a service
 * Finch_: services for climate indices calculation
+* Pelican_: Supporting ESGF compute API
 * Kingfisher_: Services for Earth-Observation data analysis
 
 Many climate analysis operations are implemented using OpenClimateGIS_
@@ -74,11 +72,9 @@ You can find the source code of all birdhouse components on GitHub_.
 Docker images with birdhouse components are on `Docker Hub`_
 
 Files and Folders
-.................
-
-.. _note: See also administrator guidelines
+-----------------
 
-This is an overview of folder structure and important files for administration of a server-side birdhouse ecosystem.
+This is an overview of folder structure and important files for :ref:`administration of a server-side <admin_guide>` birdhouse ecosystem.
 
 It is recommended to clone the separated WPS services (birds) into one top level folder like:
 
@@ -125,8 +121,5 @@ are situated in:
   $ ./{birdname}/tests
   $ ./{birdname}/tests/testdata
 
-.. _note:: See also the administration guideline to set up a birdhouse ecosystem.
-.. _todo:: add locations of mongodb celery etc...
-
 .. _GitHub: https://github.com/bird-house
 .. _`Docker Hub`: https://hub.docker.com/r/birdhouse

docs/source/guidelines.rst

@@ -3,6 +3,9 @@
 **********
 Guidelines
 **********
+.. contents::
+    :local:
+    :depth: 3
 
 To guide you through the learning curve of installation modules of birdhouse and set up an running birdhouse ecosystem, administer the server-side birdhouse components or even improve and develop your own specific functions, here are some general guidelines:
 

docs/source/index.rst

@@ -4,11 +4,11 @@
    :maxdepth: 2
 
    overview
-   architecture
+   framework
+   projects
    guidelines
    ideas/index
    communication
-   projects
    publications
    release_notes
    license

docs/source/overview.rst

@@ -11,15 +11,15 @@ Overview
 
 .. _introduction:
 
+Birdhouse is a collaborative project open for the community to participate. It is a software framework containing a collection of `Web Processing Service`_ (WPS). The deployed algorithms are focusing on Earth Systems and environmental data processing with the philosophy of streamlining the software development and deployment. By supporting climate, earth observation and biodiversity data and processes, Birdhouse can be used in a wide array of Earth sciences projects and workflows. The core benefit of this project is to allow the seamless use of climate services developed by a diverse network of national meteorological offices, regional climate service providers, academics, not-for-profit research centers and private industry. As governments move toward open-data policies, there will be a need for analytical services that extract value out of the deluge of information. Using an interoperable software architecture, institutions can provide both data and services allowing users to process the data remotely from a laptop, instead of having to acquire and maintain large storage infrastructures.
 
-Introduction
-------------
 
-Birdhouse is a collaborative project open for the community to participate. It is a software framework containing a collection of `Web Processing Service`_ (WPS). The deployed algorithms are focusing on Earth Systems and environmental data processing with the
-philosophy of streamlining the software development and deployment. By supporting climate, earth observation and biodiversity data and processes, Birdhouse can be used in a wide array of Earth sciences projects and workflows. The core benefit of this project is to
-allow the seamless use of climate services developed by a diverse network of national meteorological offices, regional climate service providers, academics, not-for-profit research centers and private industry. As governments move toward open-data policies, there will be a need for analytical services that extract value out of the deluge of information. Using an interoperable software architecture, institutions can provide both data and services allowing users to process the data remotely from a laptop, instead of having to acquire and maintain large storage infrastructures.
+.. _documentation_structure
+
+Documentation structure
+-----------------------
 
-The documentation of birdhouse is representing its nature of separate software components and therefor organized in separate documentations. Birdhouse is being used by international working groups pitching up projects with individual focus on the projects objectives. The following graph shows an overview of the documentations organization and how they are aggregated to each other for a better understanding of where to find which information:
+The birdhouse documentation reflects the fact that it is an assemblage of independent software components. It's therefore organized according to the :ref:`birdhouse framework structure <framework_structure>`. Birdhouse is being used by international working groups who deploy subsets of components tailored to their user base. The following graph shows an overview of the documentation's organization:
 
 
 .. image:: _images/Documentation_aggrgation.png

docs/source/projects.rst

@@ -1,49 +1,53 @@
 
 .. _projects:
 
-Projects using birdhouse
-========================
-
+Project examples
+================
 
 .. contents::
     :local:
     :depth: 3
 
-Birdhouse is used in the following projects:
+The birdhouse :ref:`framework is modular organized <framework_structure>` to enable a flexible architecture design depending on the projects needs. Due to the OCG Standard, software components non-birdhouse components can be combined for interoperability. Here are some examples of real projects to show the flexibility and potential of the birdhouse framework.
+
+PAVICS
+......
+
+* PAVICS_: Platform for climate analysis and visualization by Ouranos_ and CRIM_, Canada.
+* PAVICS-Hydro_ : Additional services for PAVICS_ allowing users to perform hydrological modeling and analysis.
+
+
+.. gittoctree:: https://github.com/Ouranosinc/pavics-sdi.git
+
+    docs/source/arch/backend.rst
+
+
+.. gitinclude:: https://github.com/Ouranosinc/pavics-sdi.git docs/source/arch/backend.rst
 
-.. todo:: add some introduction sentences for each project
 
 COPERNICUS
-----------
+..........
 
 * CP4CDS: Climate Projects for the `Climate Data Store`_ (part of the European Union's `Copernicus Climate Change Service`_).
 
 OGC-Testbeds
-------------
+............
 
 .. todo:: Add references to OGC testbed.
 
 * OGC Testbed 13: Enhancement of scheduling services
 * OGC Testbed 14: Enhancement of security
 
-PAVICS
-------
-
-* PAVICS_: Platform for climate analysis and visualization by Ouranos_ and CRIM_, Canada.
-* PAVICS-Hydro_ : Additional services for PAVICS_ allowing users to perform hydrological modeling and analysis.
-
-.. literalinclude :: https://raw.githubusercontent.com/Ouranosinc/pavics-sdi/master/docs/source/index.rst
-   :language: rst
-
 
 A2C2
-----
+....
 
 * A2C2_: Atmospheric flow Analogues for Climate Change
 
 .. _A2C2: https://a2c2.lsce.ipsl.fr/
 .. _PAVICS: https://ouranosinc.github.io/pavics-sdi/
 .. _PAVICS-Hydro: https://medium.com/birdhouse-newsletter/web-processing-services-for-hydrological-modeling-7b5eb5c426ed
+.. _PAVICS_architecture: https://ouranosinc.github.io/pavics-sdi/_sources/arch/backend.rst.txt
 .. _Ouranos: https://www.ouranos.ca/
 .. _CRIM: https://www.crim.ca/en
 .. _Climate Data Store: https://cds.climate.copernicus.eu/