Merge pull request #1 from neicnordic/docs/changes

docs updates

README.md

@@ -1,26 +1,39 @@
 # NeIC Sensitive Data Archive
 
-The [code](lega) is written in Python (3.6+).
-
 Recommended provisioning methods provided for production are:
 
 * on a [Kubernetes cluster](https://github.com/neicnordic/sda-helm/), using `kubernetes` and `helm` charts;
 * on a [Docker Swarm cluster](https://github.com/neicnordic/LocalEGA-deploy-swarm), using `gradle` and `docker swarm`.
 
-# Architecture
+## Architecture
+
+SDA is divided into several components, which can be deployed either for Federated EGA or as an stand-alone SDA.
+
+### Core Components
 
-SDA is divided into several components, as docker containers.
+Source code for core components (unless specified otherwise) is available at: https://github.com/neicnordic/sda-pipeline 
 
 | Component     | Role |
 |---------------|------|
-| db            | A Postgres database with appropriate schemas and isolations |
-| mq            | A (local) RabbitMQ message broker with appropriate accounts, exchanges, queues and bindings, connected to the CentralEGA counter-part. |
-| inbox         | SFTP, S3 or HTTPS server, acting as a dropbox, where user credentials are fetched from CentralEGA or via ELIXIR AAI. |
+| inbox         | SFTP, S3 or HTTPS server, acting as a dropbox, where user credentials are fetched from CentralEGA or via ELIXIR AAI. https://github.com/NBISweden/sda-s3proxy/ or https://github.com/neicnordic/sda-inbox-sftp |
+| intercept     | The intercept service relays message between the queue provided from the federated service and local queues. **(Required for Federated EGA use case)** |
 | ingest        | Split the Crypt4GH header and move the remainder to the storage backend. No cryptographic task, nor access to the decryption keys. |
 | verify        | Decrypt the stored files and checksum them against their embedded checksum. |
 | archive       | Storage backend: as a regular file system or as a S3 object store. |
 | finalize      | Handle the so-called _Accession ID_ to filename mappings from CentralEGA. |
-| data out API  | Provides a download/data access API for streaming archived data either in encrypted or decrypted format - source at: https://github.com/neicnordic/LocalEGA-DOA |
-| metadata      | Component used in standalone version of SDA. Provides an interface and backend to submit Metadata and associated with a file in the Archive. _source not part of this repo_ |
+| data out API  | Provides a download/data access API for streaming archived data either in encrypted or decrypted format - source at: https://github.com/neicnordic/sda-doa |
 
-Find the [NeIC SDA documentation](https://neic-sda.readthedocs.io) hosted on [ReadTheDocs.org](https://readthedocs.org/).
+### Associated components
+
+| Component     | Role |
+|---------------|------|
+| db            | A Postgres database with appropriate schemas and isolations https://github.com/neicnordic/sda-db/ |
+| mq            | A (local) RabbitMQ message broker with appropriate accounts, exchanges, queues and bindings, connected to the CentralEGA counter-part. https://github.com/neicnordic/sda-mq/ |
+
+
+### Stand-alone components
+
+| Component     | Role |
+|---------------|------|
+| metadata      | Component used in standalone version of SDA. Provides an interface and backend to submit Metadata and associated with a file in the Archive. https://github.com/neicnordic/sda-metadata-mirror/ with UI https://github.com/neicnordic/FormSubmission_UI |
+| orchestrate   | Component that automates ingestion in stand-alone deployments of SDA Pipeline https://github.com/neicnordic/sda-orchestration |

docs/.gitignore

@@ -4,5 +4,3 @@
 _build/
 static/*.key
 
-# These should be generated every time
-lega.utils/

docs/connection.rst

@@ -19,7 +19,7 @@ and ``SDA``/``LocalEGA``.
 Local Message Broker
 --------------------
 
-.. note:: Source code repository for MQ component is available at: https://github.com/neicnordic/LocalEGA-mq
+.. note:: Source code repository for MQ component is available at: https://github.com/neicnordic/sda-mq
 
 
 Configuration

docs/dataout.rst

@@ -3,7 +3,7 @@
 Data Retrieval API
 ==================
 
-.. note:: Source code repository for Data Out API is available at: https://github.com/neicnordic/LocalEGA-DOA
+.. note:: Source code repository for Data Out API is available at: https://github.com/neicnordic/sda-doa
 
 Configuration
 -------------

docs/db.rst

@@ -7,15 +7,15 @@ We use a Postgres database (version 11.6+ ) to store intermediate data,
 in order to track progress in file ingestion. The ``lega`` database
 schema is documented below.
 
-.. note:: Source code repository for DB component is available at: https://github.com/neicnordic/LocalEGA-db
+.. note:: Source code repository for DB component is available at: https://github.com/neicnordic/sda-db
 
 The database container will initialize and create the necessary
 database structure and functions if started with an empty area.
 Procedures for *backing up the database* are important but considered
 out of scope for the secure data archive project.
 
 Look at `the SQL definitions
-<https://github.com/neicnordic/LocalEGA-db/tree/master/initdb.d>`_ if
+<https://github.com/neicnordic/sda-db/tree/master/initdb.d>`_ if
 you are also interested in the database triggers.
 
 Configuration
@@ -92,7 +92,7 @@ bootstrapped schema version) as well as creating the corresponding
 migration script to perform the changes on a database in use.
 
 Migration scripts should be placed in `/migratedb.d/` in the
-LocalEGA-db repo (https://github.com/neicnordic/LocalEGA-db). We
+sda-db repo (https://github.com/neicnordic/sda-db). We
 recommend naming them corresponding to the schema version they provide
 migration to. There is an "eqmpty" migration script (`01.sql`) that can
 be used as a template.

docs/index.rst

@@ -95,6 +95,7 @@ Miscellaneous
    :maxdepth: 1
    :name: extra
 
+   SDA-Pipeline              <https://neicnordic.github.io/sda-pipeline/pkg/sda-pipeline/>
    Tests                     <tests>
    Contributing              <https://github.com/neicnordic/neic-sda/blob/master/CONTRIBUTING.md>
 

docs/submission.rst

@@ -150,7 +150,7 @@ Environment variables used:
 
 As mentioned above, the implementation is based on Java library Apache Mina SSHD.
 
-.. note:: Sources are located at the separate repository: https://github.com/neicnordic/LocalEGA-inbox
+.. note:: Sources are located at the separate repository: https://github.com/neicnordic/sda-inbox-sftp
           Essentially, it's a Spring-based Maven project, integrated with the :ref:`mq`.
 
 
@@ -159,7 +159,7 @@ As mentioned above, the implementation is based on Java library Apache Mina SSHD
 S3 Proxy Inbox
 ^^^^^^^^^^^^^^
 
-.. note:: Sources are located at the separate repository: https://github.com/neicnordic/S3-Upload-Proxy
+.. note:: Sources are located at the separate repository: https://github.com/neicnordic/sda-s3proxy
 
 The S3 Proxy uses access tokens as the main authentication mechanism.
 

docs/table.html

@@ -31,6 +31,13 @@
       <td>Submission</td>
       <td><i class="fa fa-battery-full ega-stable" title="Stable"></i></td>
     </tr>
+    <tr>
+      <td>Intercept</td>
+        <td>relays message between the queue provided from the federated service and local queues.
+        </td>
+        <td>Submission</td>
+        <td><i class="fa fa-battery-full ega-stable" title="Stable"></i></td>
+      </tr>
     <tr>
       <td>Ingest</td>
       <td>Splits the Crypt4GH header and moves it to database. The remainder of the file
@@ -73,5 +80,13 @@
       <td>Submission Data Retrieval</td>
       <td><i class="fa fa-battery-half ega-dev" title="Work in progress"></i></td>
     </tr>
+    <tr>
+      <td>Orchestrator</td>
+        <td>Component used in standalone version of SDA. Provides an automated ingestion and dataset ID and 
+          file ID mapping.
+        </td>
+        <td>Submission Data Retrieval</td>
+        <td><i class="fa fa-battery-half ega-dev" title="Work in progress"></i></td>
+      </tr>
   </tbody>
 </table>