Core components of AidData's Geo Framework running on William and Mary's SciClone High Permance Computing cluster.
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
assets
extract-scripts
ingest
mongodb
sciclone
scr
tasks
tests
tools
utils
.gitignore
.travis.yml-notupdated
LICENSE
README.md
SETUP.sh
config.json
repo_list.txt
requirements.txt
requirements_dev.txt

README.md

AidData's Geo Framework

Core components of AidData's Geo Framework running on William and Mary's SciClone High Permance Computing cluster.

http://geo.aiddata.wm.edu

Setup

core hpc config

  • make sure HPC account being used is set as priority user on for vortex-alpha nodes (HPC staff can do this)
  • make sure HPC servers have necessary ports open for mongodb, gmail

prepare database server

  • have IT open mongodb ports for geo.aiddata.wm.edu and all HPC servers (prod and dev servers, where applicable)
  • update mongod.conf
  • copy db_backup_script.sh and add cron (see comments in script for details)

prepare sciclone environment

  • install home directory environment scripts (copy from sciclone in this repo to your sciclone account home directory, or extract from the home_backups dir in /sciclone/aiddata10/geo)
  • load necessary modules by logging out and back in to sciclone or using source ~/.cshrc and source ~/.cshrc.rhel6-opteron
  • install python packages: pip install --user -r pip_list.txt
  • add ssh key from your sciclone account to aiddatageo github (approve rsa key first time manually)

initialize framework

  • run bash setup.sh <branch>

Ingest Datasets

  • utilizes data and ingest files from asdf-datasets repo
  • see ingest dir for specifics on ingesting datasets (related resources in asdf-datasets repo)

Components

Tasks

todo

Tools

todo

Utils

todo

Assets

static files/resources used by other scripts (e.g., images, text templates, pdfs)

Scr

Scratch - misc scripts and tools not critical to running geo framework

Tests

py-test based tests for testing geo utiltiies and components

extract-scripts

Tools for manually running batch extract jobs

Related Repos / Resources

Wiki - Additional details about Geo Framework and Sciclone:

Dataset preparation and ingest:

Website source and related config files (Private repos):

Related:

Previous repos for components of the Geo Framework which were combined in this repo: