Skip to content

Core components of AidData's GeoQuery backend, running on William & Mary's SciClone High Permance Computing cluster.

License

Notifications You must be signed in to change notification settings

aiddata/geo-hpc

Repository files navigation

AidData's Geo Framework

Core components of AidData's Geo Framework running on William and Mary's SciClone High Permance Computing cluster.

http://geo.aiddata.wm.edu

Setup

core hpc config

  • make sure HPC account being used is set as priority user on for vortex-alpha nodes (HPC staff can do this)
  • make sure HPC servers have necessary ports open for mongodb, gmail

prepare database server

  • have IT open mongodb ports for geo.aiddata.wm.edu and all HPC servers (prod and dev servers, where applicable)
  • update mongod.conf
  • copy db_backup_script.sh and add cron (see comments in script for details)

prepare sciclone environment

  • install home directory environment scripts (copy from sciclone in this repo to your sciclone account home directory, or extract from the home_backups dir in /sciclone/aiddata10/geo)
  • load necessary modules by logging out and back in to sciclone or using source ~/.cshrc and source ~/.cshrc.rhel6-opteron
  • install python packages: pip install --user -r pip_list.txt
  • add ssh key from your sciclone account to aiddatageo github (approve rsa key first time manually)

initialize framework

  • run bash setup.sh <branch>

Ingest Datasets

  • utilizes data and ingest files from asdf-datasets repo
  • see ingest dir for specifics on ingesting datasets (related resources in asdf-datasets repo)

Components

Tasks

todo

Tools

todo

Utils

todo

Assets

static files/resources used by other scripts (e.g., images, text templates, pdfs)

Scr

Scratch - misc scripts and tools not critical to running geo framework

Tests

py-test based tests for testing geo utiltiies and components

extract-scripts

Tools for manually running batch extract jobs

Related Repos / Resources

Wiki - Additional details about Geo Framework and Sciclone:

Dataset preparation and ingest:

Website source and related config files (Private repos):

Related:

Previous repos for components of the Geo Framework which were combined in this repo:

About

Core components of AidData's GeoQuery backend, running on William & Mary's SciClone High Permance Computing cluster.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Contributors 4

  •  
  •  
  •  
  •  

Languages