Skip to content
This repository

Loox is a local content indexing and search server.

branch: master

Fetching latest commit…

Octocat-spinner-32-eaf2f5

Cannot retrieve the latest commit at this time

Octocat-spinner-32 .yardoc
Octocat-spinner-32 app
Octocat-spinner-32 config
Octocat-spinner-32 db
Octocat-spinner-32 doc
Octocat-spinner-32 lib
Octocat-spinner-32 public
Octocat-spinner-32 script
Octocat-spinner-32 test
Octocat-spinner-32 vendor
Octocat-spinner-32 .gitignore
Octocat-spinner-32 Gemfile
Octocat-spinner-32 Gemfile.lock
Octocat-spinner-32 README.textile
Octocat-spinner-32 Rakefile
Octocat-spinner-32 config.ru
Octocat-spinner-32 fields.readme
Octocat-spinner-32 launchers.readme
README.textile

Needed :

  • Local mongodb install
  • Run bundle install to make sure all gems are installed

How to run

  • Start mongodb server
  • Start solr server with sunspot-solr start
  • Start a queue of workers (see resque doc) : QUEUE=* rake resque:work
  • Start a local server : rails server

This is still very early stage.

How does it work ?

Loox is a rails app embedding various background processing code, workers and parsers.
Asynchronous work is handled by a mongoid version of resque, which queue items to be processed.

Workers

Workers are the basic object called to work on queues and are stored in /lib/worker/. There are two main types :

  • AnalyzePath is the main worker, working on the base queue. It gets a collection_id, parent_id (id of the parent directory) and full path.
    It creates/loads the element if it exists and call any BaseParser or subclass on it. This worker is the initial/main crawler for the collection.
    Parsers should be quick for this worker, to avoid blocking the crawl of the collection (basically filesystem parsers)
  • ParseQueue and subclasses are used to simply parse a specific queue. They will watch for the queue and run specific classes of parsers on it.
  • The main example of a ParseQueue worker is the FileChanger worker, which is used by default to parse (using any subclass of Parser::FileChangeParser)
    files that have been marked as changed. These include slower parsers

Parsers

Parsers are stored in both /app/models/parser (for basic parsers, like filesystem stuff) and /lib/parser/ for more advanced parsers.

  • BaseParser are too be loaded in the base queue. These are file crawling parsers.
  • FileChangeParser is defined in /app/models but is just there to be subclassed by any slower parser that wants to be called everytime a file seems to have changed.

Parsers will only be called if they exist as documents in the mongoid database.
Adding a parser is very simple : in rails console :
@ MyParserType.create(:mime_types => [“/audio/”], :priority => 0)@

Something went wrong with that request. Please try again.