Docs/more adjustments and guides (#92)

Docs/more adjustments and guides

aggregate-mappings.json

@@ -7,7 +7,8 @@
         "sda/cmd/verify/verify.md": "docs/services/verify.md",
         "sda/cmd/s3inbox/s3inbox.md": "docs/services/s3inbox.md",
         "sda/cmd/syncapi/syncapi.md": "docs/services/syncapi.md",
-        "sda/sda.md": "docs/services/sda.md",
-        "GETTINGSTARTED.md": "docs/guides/sda-dev-test-doc.md"
+        "sda/cmd/sync/sync.md": "docs/services/sync.md",
+        "GETTINGSTARTED.md": "docs/guides/sda-dev-test-doc.md",
+        "sda/sda.md": "docs/services/sda.md"
     }
 }

aggregate-repositories.sh

@@ -31,7 +31,7 @@ do
 
     # add special use case for sda.md links
 
-    sed -i -E 's#cmd\/(.+)\/#''#g' docs/services/sda.md
+    sed -i -E 's#cmd\/([a-z0-9\-]+)\/#''#g' docs/services/sda.md
     git add docs/services/sda.md
 
     # update wordlist

docs/deploy.md

@@ -2,16 +2,14 @@ Deployments and Local Bootstrap
 ===============================
 
 We use different deployment strategies for environments like Docker
-Swarm, Kubernetes or a local-machine. The local machine environment is
-recommended for development and testing, while
-[Kubernetes](https://kubernetes.io/) and [Docker
-Swarm](https://docs.docker.com/engine/swarm/) for production.
+Swarm, Kubernetes or a local-machine. The [local development and testing](guides/local-dev-and-testing.md) guide is
+recommended for local-machine, while
+[Kubernetes](https://kubernetes.io/) and [Docker Swarm](https://docs.docker.com/engine/swarm/) are recommended for production.
 
 The production deployment repositories are:
 
 -   [Kubernetes Helm charts](https://github.com/neicnordic/sensitive-data-archive/tree/main/charts);
--   [Docker Swarm
-    deployment](https://github.com/neicnordic/LocalEGA-deploy-swarm/).
+-   [Docker Swarm deployment](https://github.com/neicnordic/LocalEGA-deploy-swarm/).
 
 `neicnordic/sensitive-data-archive`, provides the SDA services as well as PostgreSQL and RabbitMQ configuration. The following container image is used in the deployments where the tag separates between services:
 
@@ -20,4 +18,13 @@ The production deployment repositories are:
 - `ghcr.io/neicnordic/sensitive-data-archive:<version>-sftp-inbox` - sftp inbox
 - `ghcr.io/neicnordic/sensitive-data-archive:<version>-auth` - authentication service
 - `ghcr.io/neicnordic/sensitive-data-archive:<version>-download` - download service
-- `ghcr.io/neicnordic/sensitive-data-archive:<version>` - all other services such as: `finalize`, `ingest`, `intercept`, `verify`, `mapper` and `s3inbox`
+- `ghcr.io/neicnordic/sensitive-data-archive:<version>` - all other services such as: `finalize`, `ingest`, `intercept`, `verify`, `mapper`, `sync`, `syncapi` and `s3inbox`
+
+Guides
+------
+
+Different nodes of the `FederatedEGA` network, and projects using the stand-alone SDA have made different decisions in how to deploy the system.
+Adaptations needs to be made depending on the system to deploy on, as well as the requirements of your deployment.
+
+- [Deploying with Docker Swarm](guides/deploy-swarm.md)
+- [Deploying with Kubernetes](guides/deploy-k8s.md)

docs/dictionary/wordlist.txt

@@ -291,3 +291,9 @@ MAPPINGROUTING
 bigpicture
 stableIDs
 syncapi
+CENTERPREFIX
+HOSTKEY
+PEMKEYPASS
+PEMKEYPATH
+SYNCPUBKEYPATH
+rabbitmqctl

docs/guides/local-dev-and-testing.md

@@ -1,6 +1,5 @@
 # Local development and testing guide
 
-
 ## Guide summary
 
 This guide provides a brief introduction on how to locally install and run the Sensitive Data Archive (SDA) components on your development system, and run all the tests locally. 
@@ -29,7 +28,7 @@ The SDA code itself contains some very useful development helpers, that let's yo
  - ...and some more useful actions
 
 Once you have cloned the [neicnordic/sensitive-data-archive](https://github.com/neicnordic/sensitive-data-archive/) to your development system, you can start to
- explore these tools as explained in [this page](/guides/sda-dev-test-doc/).
+ explore these tools as explained in [this page](sda-dev-test-doc.md).
 
 ## What's next, once everything is up and running?
 

docs/guides/troubleshooting.md

@@ -1,31 +1,34 @@
 # Troubleshooting
 
-TODO:
-This guide is a stub and has yet to be finished.
-If you have feedback to give on the content you would like to see, please contact us on
-[github](https://github.com/neicnordic/neic-sda)!
-
 In this guide we aim to give some general tips on how to troubleshoot and restore services to working order.
 
 ## After deployment checklist
 
-After having deployed the SDA services in a Federated setup, the following steps can be followed to ensure that everything is up and running correctly.
+After having deployed the SDA services in a `FederatedEGA` setup, the following steps can be followed to ensure that everything is up and running correctly.
 
 ### Services running
 
-The first step is to verify that the services are up and running and the credentials are valid. Make sure that,
+The first step is to verify that the services are up and running and the credentials are valid. Make sure that:
 
 - credentials for access to RabbitMQ and Postgres are securely injected to the respective services in the form of secrets
-- all the pods/containers are in `Ready`/`Up` status.
+- all the pods/containers are in `Ready`/`Up` status and and no restarts among the pods/containers.
+    - for `FederatedEGA` setup the following pods are required: `intercept`, `ingest`, `verify`, `finalize`, `mapper` and a [Data Retrieval API](/docs/dataout.md)
+    - check the pods/container logs contain as the last message (after they have started): `time="2023-12-12T19:25:02Z" level=info msg="Starting <service-name> service"` for `intercept`, `ingest`, `verify`, `finalize`, `mapper`
+    - check the pods/container logs contain as the last message (after they have started) `time="2023-12-12T19:28:16Z" level=info msg="(5/5) Starting web server"` for `download` Data Retrieval service
 
-Next step is to make sure that the remote connections (CEGA RabbitMQ) are working. Login to the RabbitMQ admin page and check that,
+Next step is to make sure that the remote connections (`CentralEGA` RabbitMQ) are working. Login to the RabbitMQ admin page and check that:
 
-- the Federation status of the Admin tab is in state `running`
-- the Shovel status of the Admin tab is in state `running` for all 5 shovels.
+- the [Federation](https://www.rabbitmq.com/federation.html) status of the Admin tab is in state `running`
+  or using `rabbitmqctl federation_status` from the command line of a RabbitMQ pod/container.
+- the [Shovel](https://www.rabbitmq.com/shovel.html) status of the Admin tab is in state `running` for all shovels 
+  or using `rabbitmqctl shovel_status` from the command line of a RabbitMQ pod/container.
 
 ## End-to-end testing
 
-NOTE: This guide assumes that there exists a test instance account with `CentralEGA`. Make sure that the account is approved and added to the submitters group.
+> NOTE: 
+> This guide assumes that there exists a test instance account with `CentralEGA`. Make sure that the account is approved and added to the submitters group.
+> The [local development and testing](guides/local-dev-and-testing.md) guide provides the scripts for testing different parts of the setup, that can be used
+> as a reference.
 
 ### Upload file(s)
 
@@ -34,16 +37,16 @@ Upload one or a number of files of different sizes and check that,
 - the file(s) exists in the configured `inbox` of the storage backend (e.g. S3 bucket or POSIX path)
 - the file(s) entry exists in the database in the `sda.files` and `sda.file_event_log` tables
 - If the `s3inbox` is used, there should be an `uploaded` event for each specific file in the `sda.file_event_log`
-- the file(s) exists in the CEGA Submission portal (here for the test instance) Files listing, which can be accessed after pressing the three lines menu button.
+- the file(s) exists in the `CentralEGA` [Submission portal](https://ega-archive.org/submission/metadata/submission/sequencing-phenotype/submitter-portal/) (the submission portal URL address is specific for each `FederatedEGA` node). `Files` listing, which can be accessed after pressing the three lines menu button.
 
 ### Make a test submission
 
-Make a submission with the portal and select the file(s) that were uploaded in the previous step. Once the analysis or runs (one of the two is required) step is finished, the messages for the ingestion of the files should appear in the logs of the `ingest` service. Make sure that,
+Make a submission with the portal and select the file(s) that were uploaded in the previous step. Once the analysis or runs (one of the two is required) step is finished, the messages for the ingestion of the files should appear in the logs of the `ingest` service. Make sure that:
 
 - the messages are arriving for the file(s) included in the submission
-- the `ingestion`, `verify` and `finalise` processes are started and send a message when finished
-- the data in `sda.files` are correct
-- the files are logged in the `sda.file_event_log` for each of the services and files
+- the `ingestion`, `verify` and `finalize` processes are started and send a message when finished
+- the data in `sda.files` table are correct
+- the files are logged in the `sda.file_event_log` table for each of the services and files
 - the file(s) exists in the configured `archive` storage backend, see the `archive_file_path` in the `sda.files` table for the name of the archived file(s)
 - the archived file(s) exists in the configured `backup` storage backend
 - delete one run in the submitter portal, then and add it back again to make sure the cancel message is working as intended.

docs/services/sda.md

@@ -19,5 +19,5 @@ There are also additional support services:
 
 1. [Intercept](intercept.md) relays messages from `CentralEGA` to the system.
 2. [s3inbox](s3inbox.md) proxies uploads to the an S3 compatible storage backend.
-3. [sync]() project.
-4. [syncapi]() project for mirroring data between two installations of SDA.
+3. [sync](sync.md) The sync service mirrors ingested data between sites in the [Bigpicture](https://bigpicture.eu/) project.
+4. [syncapi](syncapi.md) The sync-api service is used in the [Bigpicture](https://bigpicture.eu/) project for mirroring data between two installations of SDA.

docs/structure.md

@@ -77,14 +77,14 @@ Deployment related choices
 
 ### Federated vs stand-alone
 
-In a Federated setup, the `FederatedEGA` archive node setup locally need to exchange status updates with the `CentralEGA` in a synchronized manner to basically orchestrate two parallel processes
+In a Federated setup, the `FederatedEGA` archive node setup locally need to exchange status updates with the `CentralEGA` in a synchronized manner to basically orchestrate two parallel processes:
 
 1. The multi-step process of uploading and safely archiving encrypted files holding both sensitive phenome and genome data.
 
 2. The process of the Submitter annotating the archived data in an online portal at `CentralEGA`, resulting in assigned accession numbers for items such as DataSet, Study, Files etc.
 
 
-In a stand-alone setup, the deployed service has less remote synchronisation to worry about, but on the other hand more components might be required (e.g ([orchestrator](services/orchestrator))) to also handle annotations/meta-data locally, as well as to deal with identifiers etc.
+In a stand-alone setup, the deployed service has less remote synchronisation to worry about (there is no dependency on `CentralEGA` provided services), on the other hand more components might be required (e.g (orchestrator)) to also handle annotations/meta-data locally, as well as to deal with identifiers and send all the relevant messages to the appropriate services etc.
 
 The NeIC SDA is targeting both types of setup but also to allow for the possibility to re-use components in more use cases than initially envisioned.
 
@@ -97,11 +97,11 @@ The components of SDA are all container based using Docker standards for buildin
 2. Docker Swarm
 3. PodMan
 
-For testing on local developer PC's etc, Docker compose is catered for in several of the test codes available in the SDA software repositories.
+For testing on local developer PC's etc, Docker compose is part of the [local development and testing](guides/local-dev-and-testing.md) guide.
 
 
 ### Choice of storage back-end
 
 To support different needs of different deployment locations, SDA is heavily configurable in several aspects. For the main archive storage, SDA supports both S3 storage and POSIX filesystem storage options, utilizing the exact same microservices with different parameters.
 
-For other storage dependent functionality, such as upload areas (aka inbox) and download areas (aka outbox), there are different choices of microservices (using different storage technology and transfer protocols) that can be orchestrated together with the main SDA microservices to meet local needs and requirements. 
+For other storage dependent functionality, such as upload areas i.e. `inbox` and data retrieval API, there are different choices of microservices (using different storage technology and transfer protocols) that can be orchestrated together with the main SDA microservices to meet local needs and requirements. 

docs/submission.md

@@ -101,7 +101,8 @@ Structure of the message and its contents are described in
 > services/actuators match those used for the events initiated by the
 > respective services, except for the interactions in case of errors,
 > which are highlighted with red. The optional fragments are only executed
-> if errors occur during ingestion, verify or finalize. **Note that the time axis in this diagram is all about the sequence of events not duration.**
+> if errors occur in `ingest`, `verify` or `finalize` services. 
+> **Note that the time axis in this diagram is all about the sequence of events not duration.**
 
 ### Ingestion Steps
 

mkdocs.yml

@@ -20,13 +20,17 @@ nav:
       - Interfacing with CEGA: "connection.md"
   - "Services":
       - SDA: "services/sda.md"
+      - "services/intercept.md"
       - "services/ingest.md"
       - "services/verify.md"
       - "services/finalize.md"
       - "services/mapper.md"
       - "services/syncapi.md"
+      - "services/sync.md"
       - Data Retrieval API: "dataout.md"
   - "Guides":
       - Contributing: "CONTRIBUTING.md"
       - Local dev and testing: "guides/local-dev-and-testing.md"
+      - Deploying with Kubernetes: "guides/deploy-k8s.md"
+      - Troubleshooting: "guides/troubleshooting.md"