- Local mongodb install
- Run bundle install to make sure all gems are installed
- Start mongodb server
- Start solr server with
- Start a queue of workers (see resque doc) :
QUEUE=* rake resque:work
- Start a local server : rails server
This is still very early stage.
Loox is a rails app embedding various background processing code, workers and parsers.
Asynchronous work is handled by a mongoid version of resque, which queue items to be processed.
Workers are the basic object called to work on queues and are stored in
/lib/worker/. There are two main types :
AnalyzePathis the main worker, working on the
basequeue. It gets a
parent_id(id of the parent directory) and full path.
It creates/loads the element if it exists and call any
BaseParseror subclass on it. This worker is the initial/main crawler for the collection.
Parsers should be quick for this worker, to avoid blocking the crawl of the collection (basically filesystem parsers)
ParseQueueand subclasses are used to simply parse a specific queue. They will watch for the queue and run specific classes of parsers on it.
- The main example of a
ParseQueueworker is the
FileChangerworker, which is used by default to parse (using any subclass of
files that have been marked as changed. These include slower parsers
Parsers are stored in both
/app/models/parser (for basic parsers, like filesystem stuff) and
/lib/parser/ for more advanced parsers.
BaseParserare too be loaded in the
basequeue. These are file crawling parsers.
FileChangeParseris defined in
/app/modelsbut is just there to be subclassed by any slower parser that wants to be called everytime a file seems to have changed.
Parsers will only be called if they exist as documents in the mongoid database.
Adding a parser is very simple : in rails console :
@ MyParserType.create(:mime_types => [“/audio/”], :priority => 0)@