Skip to content
This repository has been archived by the owner. It is now read-only.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
irodsqueue
README.rst
setup.py

README.rst

irodsqueue

Instructions below are for testing the development code. Subject to change.

Requirements

  • *nix
  • Python 3.4+
  • Redis 4.0+. To download and install Redis see here
  • virtualenv

Installation

It is recommended to install this package in a virtual environment, e.g:

Make a virtual environment:

virtualenv -p python3 testenv3

Activate the virtual environment:

source testenv3/bin/activate

Install the latest PRC code from github:

pip install git+https://github.com/irods/python-irodsclient

Install this repository's current development branch:

pip install git+https://github.com/irods-contrib/irods_tools_ingest@dev

Usage

Start redis:

cd REDIS_DIR/src
./redis-server

From our virtual environment call the main app, e.g:

irodsqueue --help
irodsqueue ingest --help

Enqueue ingest jobs, for example to ingest a directory into iRODS:

irodsqueue ingest -f --timer /PATH/TO/LOCAL/DIR

Once we have jobs in the queue we can launch workers to process them. Each worker is its own process. We use a custom worker class that opens and maintains iRODS sessions. Open a separate terminal and activate our same virtual environment as above.

Test your iRODS connection:

ils

Launch 16 worker processes (e.g. in bash):

for i in {1..16}; do sleep .1; rq worker -v --burst -w irodsqueue.irodsworker.IrodsWorker & done

Options

You can specify a metadata extraction function to invoke as part of the ingest process, by using the --metadata option when enqueuing jobs. The value should be the path to a python file containing a function named extract_metadata. See examples

Enqueue jobs with metadata extraction:

irodsqueue ingest -f --extract-metadata ~/tests/metadata/test.py --timer /PATH/TO/LOCAL/DIR

Enqueue jobs with file checksums:

irodsqueue ingest -Kf --timer /PATH/TO/LOCAL/DIR
You can’t perform that action at this time.