Loox is a local content indexing and search server.
Ruby JavaScript Shell
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
.yardoc
app
config
db
doc
lib
public
script
test
vendor/plugins
.gitignore
Gemfile
Gemfile.lock
README.textile
Rakefile
config.ru
fields.readme
launchers.readme

README.textile

Needed :

  • Local mongodb install
  • Run bundle install to make sure all gems are installed

How to run

  • Start mongodb server
  • Start solr server with sunspot-solr start
  • Start a queue of workers (see resque doc) : QUEUE=* rake resque:work
  • Start a local server : rails server

This is still very early stage.

How does it work ?

Loox is a rails app embedding various background processing code, workers and parsers.
Asynchronous work is handled by a mongoid version of resque, which queue items to be processed.

Workers

Workers are the basic object called to work on queues and are stored in /lib/worker/. There are two main types :

  • AnalyzePath is the main worker, working on the base queue. It gets a collection_id, parent_id (id of the parent directory) and full path.
    It creates/loads the element if it exists and call any BaseParser or subclass on it. This worker is the initial/main crawler for the collection.
    Parsers should be quick for this worker, to avoid blocking the crawl of the collection (basically filesystem parsers)
  • ParseQueue and subclasses are used to simply parse a specific queue. They will watch for the queue and run specific classes of parsers on it.
  • The main example of a ParseQueue worker is the FileChanger worker, which is used by default to parse (using any subclass of Parser::FileChangeParser)
    files that have been marked as changed. These include slower parsers

Parsers

Parsers are stored in both /app/models/parser (for basic parsers, like filesystem stuff) and /lib/parser/ for more advanced parsers.

  • BaseParser are too be loaded in the base queue. These are file crawling parsers.
  • FileChangeParser is defined in /app/models but is just there to be subclassed by any slower parser that wants to be called everytime a file seems to have changed.

Parsers will only be called if they exist as documents in the mongoid database.
Adding a parser is very simple : in rails console :
@ MyParserType.create(:mime_types => [“/audio/”], :priority => 0)@