VOYAGES API

The following notes provide an overview of how to install and run the SV API project, which is a restructuring of SlaveVoyages.org to bring it closer to a true microservices model.

For notes on how to use the API, see the Project Structure readme file

For a Swagger UI presentation of the API documentation's generic public endpoints, go to the root url of the endpoint:

Running locally: http://127.0.0.1:8000/
Current public location (subject to change): https://voyages-api-staging.crc.rice.edu/

System Requirements

For reference, this document was written while testing on a 2022 MacBook Pro running MacOS Ventura and Docker Desktop 4.21.1.

Install the macOS Xcode Command Line Tools.

local:~$ sudo xcodebuild -license
local:~$ xcode-select install

Install Homebrew.

local:~$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Install and configure the GitHub CLI.

local:~$ brew install gh
local:~$ gh auth login

Install Docker Desktop.

Optionally, download and install manually instead of using Homebrew.

local:~$ brew install --cask docker

Getting Started

Change to your local project directory (in this case, ~/Projects).

local:~$ cd Projects

Fork the rice-crc/voyages-api repository and clone to your local environment.

local:~/Projects$ gh repo fork rice-crc/voyages-api --remote --default-branch-only --clone

Local App Deployment

Change to the cloned repository directory.

local:~/Projects$ cd voyages-api

Copy the default config files for each app component.

local:~/Projects/voyages-api$ cp api/voyages3/localsettings.py{-default,}
local:~/Projects/voyages-api$ cp geo-networks/localsettings.py{-default,}
local:~/Projects/voyages-api$ cp people-networks/localsettings.py{-default,}
local:~/Projects/voyages-api$ cp stats/localsettings.py{-default,}

Download the latest database dump from the Google Drive project share and expand into the data/ directory. Rename the expanded file to data/voyages_prod.sql.

Build the API containers. The component containers must be built separately.

local:~/Projects/voyages-api$ docker compose up --build -d voyages-mysql voyages-api voyages-adminer voyages-solr

Note: you can remove the -d option to run the process in the foreground. JCM always does this to watch the logs.

Allow a short bit of time for the mysql container to initialize. Then inject the sql dump.

local:~/Projects/voyages-api$ docker exec -i voyages-mysql mysql -uroot -pvoyages voyages_api <  data/voyages_prod.sql

Verify the data import.

local:~/Projects/voyages-api$ docker exec -i voyages-mysql mysql -uvoyages -pvoyages -e "show databases"
local:~/Projects/voyages-api$ docker exec -i voyages-mysql mysql -uvoyages -pvoyages -e "show tables from voyages_api"
local:~/Projects/voyages-api$ docker exec -i voyages-mysql mysql -uvoyages -pvoyages -e "select * from voyages_api.voyage_voyage limit 1"

Run the app setup and configuration tasks.

local:~/Projects/voyages-api$ docker exec -i voyages-api bash -c 'python3 manage.py collectstatic --noinput'
local:~/Projects/voyages-api$ docker exec -i voyages-api bash -c 'python3 manage.py migrate'
local:~/Projects/voyages-api$ docker exec -i voyages-solr solr create_core -c voyages -d /srv/voyages/solr
local:~/Projects/voyages-api$ docker exec -i voyages-solr solr create_core -c enslavers -d /srv/voyages/solr
local:~/Projects/voyages-api$ docker exec -i voyages-solr solr create_core -c enslaved -d /srv/voyages/solr
local:~/Projects/voyages-api$ docker exec -i voyages-solr solr create_core -c blog -d /srv/voyages/solr
local:~/Projects/voyages-api$ docker exec -i voyages-solr solr create_core -c autocomplete_source_titles -d /srv/voyages/solr
local:~/Projects/voyages-api$ docker exec -i voyages-solr solr create_core -c autocomplete_ship_names -d /srv/voyages/solr
local:~/Projects/voyages-api$ docker exec -i voyages-solr solr create_core -c autocomplete_enslaved_names -d /srv/voyages/solr
local:~/Projects/voyages-api$ docker exec -i voyages-solr solr create_core -c autocomplete_enslaver_aliases -d /srv/voyages/solr
local:~/Projects/voyages-api$ docker exec -i voyages-api bash -c 'python3 manage.py rebuild_indices'
local:~/Projects/voyages-api$ docker exec -i voyages-api bash -c 'python3 manage.py rebuild_autocomplete_indices'

Build the API component containers.

local:~/Projects/voyages-api$ docker compose up --build -d voyages-geo-networks voyages-people-networks voyages-stats

Generating an API Key for the Flask Components

The Flask components of the app require an API key.

Create a new Django superuser account through the CLI.

local:~/Projects/voyages-api$ docker exec -it voyages-api bash -c 'python3 manage.py createsuperuser'

Use those credentials to log in to the Django admin interface at http://127.0.0.1:8000/admin/ and create an API token for the account.

Update the stats/localsettings.py and networks/localsettings.py files with the new token.

Restart the Flask component containers.

local:~/Projects/voyages-api$ docker restart voyages-geo-networks voyages-people-networks voyages-stats

Cleanup

If you want to tear it down:

local:~/Projects/voyagesapi$ docker compose down
local:~/Projects/voyagesapi$ docker container prune -f
local:~/Projects/voyagesapi$ docker image prune -f
local:~/Projects/voyagesapi$ docker volume prune -f
local:~/Projects/voyagesapi$ docker network prune -f

Resources

Note the following project resources:

Voyages API: http://127.0.0.1:8000/
API Stats Component: http://127.0.0.1:5000
API Geo Networks Component: http://127.0.0.1:5005
API People Networks Component: http://127.0.0.1:5006
Solr: http://127.0.0.1:8983
Adminer: http://127.0.0.1:8080
Redis: http://127.0.0.1:6379

Geo Networks

This container runs NetworkX and Pandas in order to build splined geographic sankey maps that are aggregated in a weighted manner on shared routes.

When initialized for the first time, for voyages and people, at region and place levels (A total of 4 runs):

Creates a basic geographic network in NetworkX
Asks the voyages-api component for relevant geo data to flesh the map out
Then pulls the full itinerary for
- the class in question (voyages or people)
- at the resolution in question (regions or places)
For each of those entities
- It draws the full, splined path
- Stores the touched nodes and edges in a dataframe
At the end of each run, it
- Loads the full dataframe into memory
- And dumps the results to a pickle file under /tmp

This can take 15 minutes. When initialized subsequently, it loads the pickles into memory in about 2 seconds.

Adminer

The Adminer container is provided as an optional way of working with the database.

Visit http://127.0.0.1:8080 and log in with the following values.

Server: voyages-mysql
User: voyages
Password: voyages
Database: voyages-api

Development Workflow

This project follows a fork-and-pull workflow.

The rice-crc:voyages-api repository is referred to as upstream and your fork as origin
The upstream/develop branch serves as the default change integration point between developers
A developer makes Pull Requests from their origin/working-branch to upstream/develop

Keep the following in mind when contributing code.

Keep your fork up-to-date with the upstream repository
Always start new work with a new working branch
Periodically fetch and rebase the latest upstream/develop changes onto your local working-branch
Do not make Pull Requests containing untested or unfinished code unless intended for temporary review and discussion
Clean up the work in your local environment once a Pull Request has been accepted and merged

Use the following git process to contribute work.

git fetch upstream            # Pull the latest changes from upstream/develop
git checkout -b <short-desc>  # and create a new working branch

git fetch upstream            # Do work; before any commits, pull the latest
git rebase upstream/develop   # changes from upstream/develop and rebase onto
                              # your working branch

git add . && git commit       # Commit your changes to the working branch
                              # Repeat pulling changes and adding commits until
                              # your work is done

git push origin HEAD          # Push the working branch to your fork and make
gh pr create --fill           # a Pull Request to upstream/develop

git checkout develop          # Once the PR is accepted and merged, delete
git branch -D <short-desc>    # your working branch

git pull                      # Pull the latest changes from upstream/develop
git push                      # and update your fork by pushing to origin/develop

Name		Name	Last commit message	Last commit date
Latest commit History 852 Commits
api		api
data		data
docker		docker
geo-networks		geo-networks
people-networks		people-networks
solr		solr
stats		stats
.dockerignore		.dockerignore
.gitignore		.gitignore
PROJECT_STRUCTURE.md		PROJECT_STRUCTURE.md
README.md		README.md
dev_notes.md		dev_notes.md
docker-compose-content.yml		docker-compose-content.yml
docker-compose-migrate.yml		docker-compose-migrate.yml
docker-compose-staging.yml		docker-compose-staging.yml
docker-compose.yml		docker-compose.yml

rice-crc/voyages-api

Folders and files

Latest commit

History

Repository files navigation

VOYAGES API

System Requirements

Getting Started

Local App Deployment

Generating an API Key for the Flask Components

Cleanup

Resources

Geo Networks

Adminer

Development Workflow

About

Resources

Stars

Watchers

Forks

Languages