Skip to content
The repository that contains the server code for the RATOM project
Python Jupyter Notebook Makefile Dockerfile JavaScript Shell
Branch: develop
Clone or download

Latest commit

JeremyGibson Merge pull request #78 from StateArchivesOfNorthCarolina/ratom-207-bu…
…g-fix-modify-change-form-undeleteable

[RATOM-207] Fixes unregistered bug
Latest commit a215561 Apr 3, 2020

Files

Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
api [RATOM-241] Adds tests and extra guard in the task Mar 31, 2020
core Merge pull request #78 from StateArchivesOfNorthCarolina/ratom-207-bu… Apr 3, 2020
deployment [RATOM-227] set memory request to 5GiB on prod Mar 27, 2020
docs turn on flake8 Jan 27, 2020
documentation [RATOM-207] Adds admin documentation removes old graphql Mar 27, 2020
etl [RATOM-235] Provides a stream reader to a stream writer Apr 2, 2020
locale initial project skeleton via django-project-template Nov 6, 2019
notebooks [RATOM-195] Adds a new restricted label type. Mar 6, 2020
ratom [RATOM-222] adds 50 message limit Mar 27, 2020
requirements Bump ansible from 2.9.0 to 2.9.2 in /requirements Feb 26, 2020
.coveragerc [RATOM-14] Remove some things until needed from coverage Feb 7, 2020
.flake8 Merge pull request #68 from StateArchivesOfNorthCarolina/ratom-193-ke… Mar 23, 2020
.gitignore [RATOM-195] Adds a new restricted label type. Mar 6, 2020
.pre-commit-config.yaml split up ansible vars into separate files Feb 4, 2020
.travis.yml remove pre-commit from ci for now Feb 26, 2020
Dockerfile add test-base docker build stage Jan 26, 2020
LICENSE MIT license Mar 5, 2020
Makefile add fail-under 70% on CI Jan 27, 2020
RATOM_V1.postman_collection.json add more pre-commit-hooks Jan 27, 2020
README.rst add logo, license, and dev team to readme Mar 5, 2020
conftest.py [RATOM-161] Fixes and adds test for multiple files. Mar 26, 2020
docker-compose.ci.yml change to string Jan 27, 2020
docker-compose.override.yml-example move kibana to optional overrride file Jan 26, 2020
docker-compose.yml Merge branch 'RATOM-17/ci' into RATOM-56/celery Jan 29, 2020
docker-entrypoint.sh [RATOM-228] adds version and using role migration Mar 26, 2020
manage.py [RATOM-13] complete rework of structure Jan 20, 2020
mypy.ini disable return any warning; not sure how to to use it yet Nov 17, 2019
pytest.ini add test settings Feb 28, 2020
setup.cfg OBLITERATES refs to js... Nov 7, 2019

README.rst

https://github.com/libratom/ratom-logos/raw/master/basic_variations/RATOM_Vector_Logo_v1_300px.png

RATOM API

https://travis-ci.org/StateArchivesOfNorthCarolina/ratom-server.svg?branch=develop

Below you will find basic setup and deployment instructions for the ratom_api project. To begin you should have the following applications installed on your local development system:

Django version

The Django version configured in this template is conservative. If you want to use a newer version, edit requirements/base.txt.

Getting Started

First clone the repository from Github and switch to the new directory:

$ git clone git@github.com:StateArchivesOfNorthCarolina/ratom-server.git
$ cd ratom_api

To setup your local environment you can use the quickstart make target setup, which will install Python dependencies (via pip) into a virtualenv named "ratom_api", configure a local django settings file, and create a database via Postgres named "ratom_api" with all migrations run:

$ make setup
$ workon ratom_api

If you require a non-standard setup, you can walk through the manual setup steps below making adjustments as necessary to your needs.

To setup your local environment you should create a virtualenv and install the necessary requirements:

# Check that you have python3.7 installed
$ which python3.7
$ mkvirtualenv ratom_api -p `which python3.7`
(ratom_api)$ pip install -r requirements/dev.txt

Next, we'll set up our local environment variables. We use django-dotenv to help with this. It reads environment variables located in a file name .env in the top level directory of the project. The only variable we need to start is DJANGO_SETTINGS_MODULE:

(ratom_api)$ cp ratom/settings/local.example.py ratom/settings/local.py
(ratom_api)$ echo "DJANGO_SETTINGS_MODULE=ratom.settings.local" > .env

Create the Postgres database and run the initial migrate:

(ratom_api)$ createdb -E UTF-8 ratom_api
(ratom_api)$ python manage.py migrate

If you want to use Travis to test your project, rename project.travis.yml to .travis.yml, overwriting the .travis.yml that currently exists. (That one is for testing the template itself.):

(ratom_api)$ mv project.travis.yml .travis.yml

Black

Run:

$ pre-commit install

You can also use black to format on save. For example, configuration for VS Code:

{
    "settings": {
        "python.formatting.provider": "black",
        "editor.formatOnSave": true,
        "python.linting.flake8Enabled": true,
        "python.linting.mypyEnabled": true
    }
}

Import

A management command, import_psts, can be used to .pst files into RATOM.

For example:

python manage.py import_psts /Volumes/Seagate/RATOM/RevisedEDRMv1_Complete/kate_symes/* --clean

The search index can be rebuilt with:

python manage.py search_index -f --rebuild --parallel

Development

🤯

Tests

To run the projet unit tests, use:

make test

A full-stack test with a real Enron .pst file is skipped by default. To enable it, run:

TEST_ENRON_DATA_SET=true make test

HTML-based coverage reports are generated into htmlcov/.

Sample Data

An API endpoint, at /api/v1/reset-sample-data/, will reset sample datasets with an authenicated POST request. This URL is disabled unless the RATOM_SAMPLE_DATA_ENABLED environment variable is set to true. This is enabled by default on staging and in development.

To reset the project sample data using a management command, run:

python manage.py reset_sample_data

To create a new sample dataset from an existing account in your database, run:

python manage.py sample_data --account="albert_meyers" --total=10 > ./api/sample_data/albert_meyers.json

Then populate additional entries in api.sample_data.data.SAMPLE_DATA_SETS.

Deployment

Deployment for this project is done by TravisCI on each merge to develop. You can inspect the .travis.yml file to see how it's done, or to update the process. It relies on the caktus.django-k8s Ansible role.

The frontend is deployed to a separate pod via its own repo, using the same process.

You can also test or update the deployment locally in the deployment/ directory:

pip install -r requirements/dev.txt
cd deployment/
ansible-galaxy install -r requirements.yaml
ansible-playbook deploy.yaml

Note: This will deploy the image with the :latest tag. Normally, CI/CD will deploy a tag with a commit sha to ensure the that the Kubernetes Deployment updates the underlying pods. You can override the k8s_container_image_tag on the command line, if needed, to deploy a different image:

ansible-playbook deploy.yaml -l caktus-ratom -e k8s_container_image_tag=my-docker-tag

You can see the available images in DockerHub

License(s)

Logos, documentation, and other non-software products of the RATOM team are distributed under the terms of Creative Commons 4.0 Attribution. Software developed for the RATOM project is distributed under the terms of the MIT License. See the LICENSE file for additional details.

Copyright 2020, The University of North Carolina at Chapel Hill.

Development Team

Developed by Caktus Group for the Review, Appraisal, and Triage of Mail (RATOM) project.

See https://ratom.web.unc.edu/ for RATOM project details, staff bios, and news.

You can’t perform that action at this time.