A Tool to help import the content of the Learning Registry into a data store of your choice
Python JavaScript Shell
Switch branches/tags
Nothing to show
Clone or download
#2 Compare This branch is 123 commits ahead, 2 commits behind adlnet:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
cc
conf
couchdb
dev
src
.gitignore
Readme.md
install.sh
license.txt
requirements.txt
sitemap.json
tfidf.py

Readme.md

LR-Data

This is a small utility to help pull the data from the Learning Registry into a datastore of your choice.

Dependencies

LR-Data requires:

Platform-Specific Requirements:

On OS X, you will also need libevent, which can be installed with homebrew: brew install libevent

Install dependencies - Ubuntu shown as example

sudo apt-get update
sudo apt-get install -y virtualenv python-dev libevent-dev libxml2-dev libxmlsec1-dev

Setup

# Create virtual environment
virtualenv env
. env/bin/activate
# Install python library requirements
pip install -U -r requirements.txt

Configuration

All configuration is done in the src/celeryconfig.py file. For information of configuring Celery please see their document. For lr-data configuration modify

config = {

	"lrUrl": "http://lrdev02.learningregistry.org/harvest/listrecords",

	"couchdb":{

		"dbUrl":"http://localhost:5984/lr-data"

	},
	"tasks": {
		"insert": "tasks.save.insertDocumentMongo",

		"validate": "tasks.validate.emptyValidate",
	}

	"redis":{

		"host":"localhost",

		"port":6379,

		"db":0
	}

	"mongodb":{

		"database":"lr",

		"collection":"envelope",

		"host": "localhost",

		"port": 27017,

	},
}

Customizable tasks are defined in the tasks hash. validate is the task name for validating incoming docs. insert is the task you wish to use to save the data

Startup

There are scripts inside of src to get you started.

  • start_celery_workers.sh - will start your default worker threads responsible for harvesting, validating, and saving LR data
  • stop_celery_workers.sh - stops workers
  • start_harvesting.py - sends harvest request into queue for workers to start working

To start harvesting, activate your virtualenv, start the celery workers, then start harvesting request

cd src
. ../env/bin/activate
./start_celery_workers.sh
./start_harvesting.py

Should you want to stop (or pause) the processing you can stop the celery workers:

./stop_celery_workers.sh

When you want to resume processing, you need only start the celery workers again:

./start_celery_workers.sh