Skip to content

Commit

Permalink
Changes by create-pull-request action (#91)
Browse files Browse the repository at this point in the history
  • Loading branch information
pontus committed Dec 8, 2023
2 parents 0f38dce + 4aea765 commit 4fb41f6
Show file tree
Hide file tree
Showing 3 changed files with 80 additions and 2 deletions.
7 changes: 7 additions & 0 deletions docs/dictionary/wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -284,3 +284,10 @@ bigint
dbschema
erDiagram
jsonb
ACCESSIONROUTING
Bigpicture
INGESTROUTING
MAPPINGROUTING
bigpicture
stableIDs
syncapi
6 changes: 4 additions & 2 deletions docs/services/sda.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ Repository:
[neicnordic/sensitive-data-archive](https://github.com/neicnordic/sensitive-data-archive)

`sda` repository consists of a suite of services which are part of [NeIC Sensitive Data Archive](https://neic-sda.readthedocs.io/en/latest/) and implements the components required for data submission.
It can be used as part of a [Federated EGA](https://ega-archive.org/federated) or as an isolated Sensitive Data Archive.
It can be used as part of a [Federated EGA](https://ega-archive.org/federated) or as a stand-alone (isolated) Sensitive Data Archive.
`sda` was built with support for both S3 and POSIX storage.

The SDA submission pipeline has four main steps:
Expand All @@ -15,7 +15,9 @@ The SDA submission pipeline has four main steps:
3. [Finalize](finalize.md) associates a stable accessionID with each archive file and backups the file.
4. [Mapper](mapper.md) maps file accessionIDs to a datasetID.

There are also three additional support services:
There are also additional support services:

1. [Intercept](intercept.md) relays messages from `CentralEGA` to the system.
2. [s3inbox](s3inbox.md) proxies uploads to the an S3 compatible storage backend.
3. [sync]() project.
4. [syncapi]() project for mirroring data between two installations of SDA.
69 changes: 69 additions & 0 deletions docs/services/syncapi.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# sync-api

The sync-api service is used in the [Bigpicture](https://bigpicture.eu/) project.

## Service Description

The sync service facilitates replication of data and metadata between the nodes in the consortium.

When enabled the service will perform the following tasks:

1. Upon receiving a POST request with JSON data to the `/dataset` route.
1. Parse the JSON blob and validate it against the `file-sync` schema.
2. Build and send messages to start ingestion of files.
3. Build and send messages to assign stableIDs to files.
4. Build and send messages to map files to a dataset.

## Configuration

There are a number of options that can be set for the sync service.
These settings can be set by mounting a yaml-file at `/config.yaml` with settings.

ex.

```yaml
log:
level: "debug"
format: "json"
```

They may also be set using environment variables like:

```bash
export LOG_LEVEL="debug"
export LOG_FORMAT="json"
```

### Service settings

- `SYNC_API_PASSWORD`: password for the API user
- `SYNC_API_USER`: User that will be allowed to send POST requests to the API

### RabbitMQ broker settings

These settings control how sync connects to the RabbitMQ message broker.

- `BROKER_HOST`: hostname of the rabbitmq server
- `BROKER_PORT`: rabbitmq broker port (commonly `5671` with TLS and `5672` without)
- `BROKER_EXCHANGE`: exchange to send messages to
- `BROKER_USER`: username to connect to rabbitmq
- `BROKER_PASSWORD`: password to connect to rabbitmq
- `BROKER_PREFETCHCOUNT`: Number of messages to pull from the message server at the time (default to 2)

The default routing keys for sending ingestion, accession and mapping messages can be overridden by setting the following values:

- `SYNC_API_ACCESSIONROUTING`
- `SYNC_API_INGESTROUTING`
- `SYNC_API_MAPPINGROUTING`

### Logging settings

- `LOG_FORMAT` can be set to “json” to get logs in json format. All other values result in text logging
- `LOG_LEVEL` can be set to one of the following, in increasing order of severity:
- `trace`
- `debug`
- `info`
- `warn` (or `warning`)
- `error`
- `fatal`
- `panic`

0 comments on commit 4fb41f6

Please sign in to comment.