On the documentation

EGA-archive · Oct 29, 2018 · 5ed1ce2 · 5ed1ce2
1 parent a06d3e2
commit 5ed1ce2
Show file tree

Hide file tree

Showing 7 changed files with 102 additions and 114 deletions.
diff --git a/deploy/test/outgest.py b/deploy/test/outgest.py
@@ -2,7 +2,7 @@
 import sys
 
 import requests
-import json
+#import json
 
 TOKEN_URL = "https://egatest.crg.eu/idp/token"
 

diff --git a/docs/bootstrap.rst b/docs/bootstrap.rst
diff --git a/docs/extra.md b/docs/extra.md
@@ -0,0 +1,53 @@
+Configuration
+-------------
+
+A few files are required in order to connect the different components.
+
+The main configurations are set by default, and it is possible to
+overwrite any of them. All Python components can indeed be started
+using the ``--conf <file>`` switch to specify the configuration file.
+
+The settings are loaded, in order:
+
+* from environment variables (where the naming convention is uppercase ``section_option`` (as in ``default.ini``), e.g. ``VAULT_DRIVER`` or ``POSTGRES_DB``,
+* from the package's ``defaults.ini``,
+* from the file ``/etc/ega/conf.ini`` (if it exists),
+* and finally from the file specified as the ``--conf`` argument.
+
+Therefore, there is no need to update the ``defaults.ini``. Instead,
+reset/update any key/value pairs by creating a custom configuration file and pass it
+to ``--conf`` as a command-line argument.
+
+See a `full description of the environment variable settings
+<https://github.com/NBISweden/LocalEGA/wiki/Configuration-Settings-%7C-Environment-Variables>`_.
+
+
+Logging
+-------
+
+A similar mechanism is used to overwrite the default logging settings.
+
+The ``--log <file>`` argument is used to configuration where the logs
+go.  Without it, we look at the ``DEFAULT/log_conf`` key/value pair
+from the loaded configuration.  If the latter doesn't exist, there is
+no logging capabilities.
+
+The ``<file>`` argument can either be a file path in ``INI`` or
+``YAML`` format, or a *keyword*. In the latter case, the logging
+mechanism will search for a log file, using that keyword, in the
+`default loggers
+<https://github.com/EGA-archive/LocalEGA/tree/master/lega/conf/loggers>`_. Currently,
+``default``, ``debug``, ``console``, ``logstash`` and
+``logstash-debug`` are available.
+
+Using the `logstash logger
+<https://github.com/EGA-archive/LocalEGA/blob/master/lega/conf/loggers/logstash-debug.yaml>`_,
+we leverage the famous *ELK* stack, which stands for **E**\
+lasticsearch, **L**\ ogstash and **K**\ ibana. Logstash receives the
+logs. Elasticsearch stores them and make them searchable. Kibana
+contacts the Elasticsearch service to display the logs in a web
+interface.
+
+.. image:: /static/Kibana.png
+   :target: _static/Kibana.png
+   :alt: Kibana
diff --git a/docs/index.rst b/docs/index.rst
@@ -47,26 +47,15 @@ whether the ingestion was successful, or whether there was an error.
 
 ----
 
-Getting started
-===============
-
-.. toctree::
-   :maxdepth: 2
-   :name: setup
-
-   Getting started      <setup>
-   Bootstrap & Deploy   <bootstrap>
-
-Information about the Architecture
-==================================
-
 .. toctree::
    :maxdepth: 2
    :name: architecture
 
+   Installation         <setup>
    Inbox                <inbox>
    Ingestion            <ingestion>
    CEGA from/to LEGA    <connection>
+   More on...           <extra>
 
 ..
    Miscellaneous

diff --git a/docs/setup.rst b/docs/setup.rst
@@ -11,61 +11,59 @@ The sources for LocalEGA can be downloaded and installed from the `EGA-archive G
 
 Local EGA uses a microservice architecture and we use `Docker`_ to deploy it.
 
+In order to simplify the setup of LocalEGA's components, we have
+developed a bootstrap script, taylored for a local `Docker`_
+deployment. Prior to running the bootstrap script, a settings file
+must be created, defining the following variables: (place them in a
+newly created file ``deploy/bootstrap/settings.rc``).
+
+.. csv-table::
+   :header: "Key", "Value", "Example","Description"
+   :widths: 2, 2, 1, 2
+
+   "``DOCKER_PORT_inbox``", "integer", "2222", "Port mapping to access the container from the host or external network"
+   "``DOCKER_PORT_outgest``", "integer", "10443", "Port mapping to access the container from the host or external network"
+   "``CEGA_REST_PASSWORD``", "string", "", "Password to connect to the Central EGA Users ReST endpoint"
+   "``CEGA_CONNECTION``", "string", "amqps://<user>:<password>@hellgate.crg.eu:5271/<vhost>", "CentralEGA [RabbitMQ URI](https://www.rabbitmq.com/uri-spec.html)"
+   "``LEGA_MQ_PASSWORD``", "string", "``$(generate_password 16)``", "Password for the Local MQ broker admin user"
+   "``SSL_SUBJ``", "string", "``/C=ES/ST=Spain/L=Barcelona/O=CRG/OU=SysDevs/CN=LocalEGA/emailAddress=all.ega@crg.eu``", "Used to create the self-signed certificates"
+   "``EC_KEY_COMMENT``", "string", "LocalEGA@CRG", "For the elliptic key, used by Crypt4GH"
+   "``EC_KEY_PASSPHRASE``", "string", "``$(generate_password 16)``", ""
+   "``EC_KEY_COMMENT``", "string", "LocalEGA-signing@CRG", ""
+   "``EC_KEY_PASSPHRASE``", "string", "``$(generate_password 16)``", ""
+   "``DB_LEGA_IN_PASSWORD``", "string", "``$(generate_password 16)``", "Password for the ``lega_in`` database user"
+   "``DB_LEGA_OUT_PASSWORD``", "string", "``$(generate_password 16)``", "Password for the ``lega_out`` database user"
+   "``S3_ACCESS_KEY``", "string", "``$(generate_password 16)``", "Access key for the S3 storage"
+   "``S3_SECRET_KEY``", "string", "``$(generate_password 32)``", "Secret key for the S3 storage"
+
+You are now ready to run the bootstrap script:
 
-Configuration
-=============
-
-A few files are required in order to connect the different components.
-
-The main configurations are set by default, and it is possible to
-overwrite any of them. All Python components can indeed be started
-using the ``--conf <file>`` switch to specify the configuration file.
-
-The settings are loaded, in order:
-
-* from environment variables (where the naming convention is uppercase ``section_option`` (as in ``default.ini``), e.g. ``VAULT_DRIVER`` or ``POSTGRES_DB``,
-* from the package's ``defaults.ini``,
-* from the file ``/etc/ega/conf.ini`` (if it exists),
-* and finally from the file specified as the ``--conf`` argument.
+.. code-block:: console
 
-Therefore, there is no need to update the ``defaults.ini``. Instead,
-reset/update any key/value pairs by creating a custom configuration file and pass it
-to ``--conf`` as a command-line argument.
+    $ make -C deploy bootstrap
 
-See a `full description of the environment variable settings
-<https://github.com/NBISweden/LocalEGA/wiki/Configuration-Settings-%7C-Environment-Variables>`_.
+This script creates random passwords, configuration files, necessary
+public/secret keys and connect the different components together. All
+interesting settings are found in the ``private`` directory. Look
+especially at its ``.trace`` file.
 
+Once the bootstrap files are generated, you can spin up the LocalEGA components, using:
 
-Logging
-=======
+.. code-block:: console
 
-A similar mechanism is used to overwrite the default logging settings.
+    $ make up
 
-The ``--log <file>`` argument is used to configuration where the logs
-go.  Without it, we look at the ``DEFAULT/log_conf`` key/value pair
-from the loaded configuration.  If the latter doesn't exist, there is
-no logging capabilities.
+Use ``make ps`` to see its status.
 
-The ``<file>`` argument can either be a file path in ``INI`` or
-``YAML`` format, or a *keyword*. In the latter case, the logging
-mechanism will search for a log file, using that keyword, in the
-`default loggers
-<https://github.com/EGA-archive/LocalEGA/tree/master/lega/conf/loggers>`_. Currently,
-``default``, ``debug``, ``console``, ``logstash`` and
-``logstash-debug`` are available.
+.. note::
+   You can find different deployment strategies, covered by our
+   partners, for environments like `Docker Swarm`_, `Kubernetes`_, `Openstack`_.
 
-Using the `logstash logger
-<https://github.com/EGA-archive/LocalEGA/blob/master/lega/conf/loggers/logstash-debug.yaml>`_,
-we leverage the famous *ELK* stack, which stands for **E**\
-lasticsearch, **L**\ ogstash and **K**\ ibana. Logstash receives the
-logs. Elasticsearch stores them and make them searchable. Kibana
-contacts the Elasticsearch service to display the logs in a web
-interface.
+.. _EGA-archive Github repo: https://github.com/EGA-archive/LocalEGA
+.. _Docker: https://github.com/EGA-archive/LocalEGA/tree/master/deploy
+.. _Docker Swarm: https://github.com/NBISweden/LocalEGA-deploy-swarm
+.. _Kubernetes: https://github.com/NBISweden/LocalEGA-deploy-k8s
+.. _Openstack: https://github.com/NBISweden/LocalEGA-deploy-terraform
 
-.. image:: /static/Kibana.png
-   :target: _static/Kibana.png
-   :alt: Kibana
 
 
-.. _EGA-archive Github repo: https://github.com/EGA-archive/LocalEGA
-.. _Docker: https://github.com/EGA-archive/LocalEGA/tree/master/deploy
diff --git a/docs/tests.rst b/docs/tests.rst
diff --git a/setup.py b/setup.py
@@ -17,7 +17,7 @@
 ''',
       packages=['lega', 'lega/utils', 'lega/conf'],
       include_package_data=False,
-      package_data={ 'lega': ['conf/loggers/*.yaml', 'conf/defaults.ini'] },
+      package_data={'lega': ['conf/loggers/*.yaml', 'conf/defaults.ini']},
       zip_safe=False,
       entry_points={
           'console_scripts': [
@@ -30,5 +30,5 @@
               'ega-cleanup   = lega.cleanup:main',
           ]
       },
-      platforms = 'any',
+      platforms='any',
 )