Data Platform

The Data Platform will be a centralised, in-house platform to provide hosting and tools for data:

• storage
• discovery
• analysis
• dissemination
• governance

About this repository

This is the Ministry of Justice repository for the core infrastructure of the Data Platform.

Please read the contributing guide before sending pull requests, or creating issues.

Data Platform repositories

We have a series of repositories for our work. We have adopted the naming convention of naming each repository starting with data-platform-*.

We also name things so that users can understand what a repository does through its name.

Name	Description
Data Platform (this repository)	Core infrastructure, public facing documentation, feature work, enhancements, and issues
Data Platform products	User-created services that are hosted on the Data Platform
Data Platform support	This repository is used for support and provides templated forms for our users.
Modernisation Platform environments repository	Hosting environment for the data platform

Testing locally

Unit tests are implemented per container using pytest. We use moto to mock out AWS services.

For example, to test daap-get-glue-metadata, run:

cd containers/daap-get-glue-metadata
pip install -r tests/requirements.txt
pytest

To test the full docker image without deploying it to lambda, follow the instructions in Testing AWS base images.

OpenSSF Best Practices

These are the requirements currently met by this repository in line with the FLOSS Best Practices Criteria (Passing Badge)

Basic project site content:

• The project site describes what the software does
• Provides information on how to interact with the project
• explains contribution process

FLOSS license:

• Uses MIT licence

Documentation:

• The project provides basic documentation

Public version-controlled source repository

• GIT version control

Bug-reporting process:

• The project provides a process for users to submit bug reports
• The project uses GitHub issues/ zen hub to track issues

Other security issues:

• secured private credentials via Kuberentes secrets and GitHub secrets

Secured delivery against man-in-the-middle (MITM) attacks:

• Delivered via terraform secured by HTPPS and spcific IAM roles useable only by the GitHub action
• No Cryptographic hases transported over HTTP

Static code analysis:

• Use of CodeQL

Other:

• The project site supports HTTPS using TLS

• The project is maintained

  For more information please see Criteria

Useful links

It may be also useful to look at:

• Technical documentation
• Architecture Decision Records (ADRs)

Getting in touch

We are currently in the research and design phase, and are not yet accepting requests to host new data products.

In the meantime please get in touch via our #data-platform Slack channel with any questions.

Incident contact details

Slack: #data-platform Email: data-platform@digital.justice.gov.uk

Service team contact

As above - preferably our Slack channel: #data-platform (or email data-platform@digital.justice.gov.uk)

Hosting environment

Modernisation Platform

Editing and publishing the Technical Documentation

The published documentation is created by editing *.html.md.erb files, found in the docs source folder.

The syntax is Markdown, more details can be found here.

For guidance see the Tech Docs Template Write your content.

While editing the files locally, you can start a Docker container that will use Middleman to act as a server hosting the web pages. See preview docs.

Every change should be reviewed in a pull request, no matter how minor. PR request reviewer/s should be enabled within the main branch protection settings.

Merging the changes to the main branch automatically publishes the changes via GH Action. See publishing.

Preview docs

You can preview how your changes will look, if you've cloned this repository to your local machine, run this command:

bash scripts/docs/docker.sh preview

This will run a preview web server on http://localhost:4567 which you can open in your browser.

Use bash scripts/docs/docker.sh check to compile the site to HTML and check the URLs are valid.

This is only accessible on your computer, and won't be accessible to anyone else.

For more details see the tech-docs-github-pages-publisher repository.

Publishing

Any changes you push/merge into the main branch should be published to GitHub Pages site automatically.

Template configuration

The web page layout is configured using the config/tech-docs.yml file.

The template can be configured in config/tech-docs.yml

Further configuration options are described on the Tech Docs Template website: Global Configuration.

Last review date

10th August 2023