Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Loox is a local content indexing and search server.
Ruby JavaScript Shell
Branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
.yardoc
app
config
db
doc
lib
public
script
test
vendor/plugins
.gitignore
Gemfile
Gemfile.lock
README.textile
Rakefile
config.ru
fields.readme
launchers.readme

README.textile

Needed :

  • Local mongodb install
  • Run bundle install to make sure all gems are installed

How to run

  • Start mongodb server
  • Start solr server with sunspot-solr start
  • Start a queue of workers (see resque doc) : QUEUE=* rake resque:work
  • Start a local server : rails server

This is still very early stage.

How does it work ?

Loox is a rails app embedding various background processing code, workers and parsers.
Asynchronous work is handled by a mongoid version of resque, which queue items to be processed.

Workers

Workers are the basic object called to work on queues and are stored in /lib/worker/. There are two main types :

  • AnalyzePath is the main worker, working on the base queue. It gets a collection_id, parent_id (id of the parent directory) and full path.
    It creates/loads the element if it exists and call any BaseParser or subclass on it. This worker is the initial/main crawler for the collection.
    Parsers should be quick for this worker, to avoid blocking the crawl of the collection (basically filesystem parsers)
  • ParseQueue and subclasses are used to simply parse a specific queue. They will watch for the queue and run specific classes of parsers on it.
  • The main example of a ParseQueue worker is the FileChanger worker, which is used by default to parse (using any subclass of Parser::FileChangeParser)
    files that have been marked as changed. These include slower parsers

Parsers

Parsers are stored in both /app/models/parser (for basic parsers, like filesystem stuff) and /lib/parser/ for more advanced parsers.

  • BaseParser are too be loaded in the base queue. These are file crawling parsers.
  • FileChangeParser is defined in /app/models but is just there to be subclassed by any slower parser that wants to be called everytime a file seems to have changed.

Parsers will only be called if they exist as documents in the mongoid database.
Adding a parser is very simple : in rails console :
@ MyParserType.create(:mime_types => [“/audio/”], :priority => 0)@

Something went wrong with that request. Please try again.