sdx-seft-publisher-service

Microservice for publishing SEFT files from internal to RAS

Purpose of this service

This service takes files produced by internal systems and publishes them out to the ONS RAS platform running in the cloud. The legacy internal systems do not currently have the ability to publish directly so this service provides a service layer shim to enable the files to be moved as required.

Once the internal systems are fully redeveloped this service will be either retired or retooled and integrated into the new platform.

Installation

This application presently installs required packages from requirements files:

• requirements.txt: packages for the application, with hashes for all packages: see https://pypi.org/project/hashin/
• test-requirements.txt: packages for testing and linting

It's also best to use pyenv and pyenv-virtualenv, to build in a virtual environment with the currently recommended version of Python. To install these, see:

• https://github.com/pyenv/pyenv
• https://github.com/pyenv/pyenv-virtualenv
• (Note that the homebrew version of pyenv is easiest to install, but can lag behind the latest release of Python.)

Getting started

Once your virtual environment is set, install the requirements:

$ make build

To test, first run make build as above, then run:

$ make test

It's also possible to install within a container using docker. From the sdx-seft-publisher directory:

$ docker build -t sdx-seft-publisher .

Usage

Using Docker compose

You can run this service using the docker compose file in sdc-ci-upload-compose and running docker compose up -d. This will run all the necessary services that it needs to communicate with. All you will need to do then is put a .xlxs file inside the Documents/ftp directory and file should be put on the FTP.

Running Standalone

You can run this service on its own but to do so, a FTP server and a Rabbit queue need to be created to communicate with it. If you want to run it on its own, your best option is to go to sdc-ci-upload-compose and run docker-compose up -d sdx-seft-publisher-service. This will build all the services that sdx-seft-publisher needs to run, pull a file from an FTP and put it on a Rabbit queue.

Alternatively you can create the FTP server and the Rabbit queue with the following commands:

docker run -d -p 5672:5672 rabbitmq:3-management:latest
docker run -d -p 2021:2021 onsdigital/pure-ftpd:latest

To start the service, use the command:

$ make start

Configuration

Environment variable	Default	Description
SEFT_RABBITMQ_HOST	localhost	RabbitMQ host
SEFT_RABBITMQ_PORT	5672	RabbitMQ port
SEFT_RABBITMQ_DEFAULT_PASS	rabbit	RabbitMQ password
SEFT_RABBITMQ_DEFAULT_USER	rabbit	RabbitMQ user
SEFT_FTP_HOST	127.0.0.1	FTP host
SEFT_FTP_PORT	2121	FTP port
SEFT_FTP_USER	user	FTP user
SEFT_FTP_PASS	password	FTP password
SDX_KEYS_FILE	./jwt-test-keys/keys.yml	Location of the keys file that contains encryption and signing keys
SEFT_FTP_INTERVAL_MS	1800000	Source polling interval (milliseconds)

Test

To run the tests locally:

make test

To run the tests in a Cloudfoundry environment:

cf push seft-publisher-unittest
cf logs seft-publisher-unittest --recent

Signing/Encryption Keys

The payload that is put onto the rabbit queue is signed and encypted. In order to do that, sets of keys are needed to be generated. If a new set of test keys need to be created then follow the steps below:

cd jwt-test-keys
# Create the signing keys
sudo openssl genrsa -out sdc-sdx-outbound-signing-private-v1.pem 4096
sudo openssl rsa -pubout -in sdc-sdx-outbound-signing-private-v1.pem -out sdc-sdx-outbound-signing-public-v1.pem

# Create the encryption keys
sudo openssl genrsa -out sdc-ras-outbound-encryption-private-v1.pem 4096
sudo openssl rsa -pubout -in sdc-ras-outbound-encryption-private-v1.pem -out sdc-ras-outbound-encryption-public-v1.pem

Once this is done, you'll have a set of keys, but you'll need a keys.yml file to be created so that it's useable by sdc-cryptography. To do this, do the following after cloning the sdc-cryptography repo to your machine:

cd <location of sdc-cryptography>/sdc/crypto/scripts
./generate_keys.py <location of sdx-seft-publisher-service>/jwt-test-keys
cp keys.yml <location of sdx-seft-publisher-service>/jwt-test-keys

The keys.yml file should contain 2 entries:

• A public encryption key
• A private signing key

In a non-test setting, we would generate the signing keys (giving the public signing key to the consumer) and we would receive a public encryption key (with the consumer keeping the private encryption to themselves)

License

Copyright ©‎ 2016, Office for National Statistics (https://www.ons.gov.uk)

Released under MIT license, see LICENSE for details.