Skip to content

Commit

Permalink
Spaces cleanup
Browse files Browse the repository at this point in the history
  • Loading branch information
kovyrin committed Mar 15, 2010
1 parent faaadbf commit 9b3779b
Showing 1 changed file with 53 additions and 52 deletions.
105 changes: 53 additions & 52 deletions README.rdoc
@@ -1,35 +1,35 @@
= Simple background loops framework

<tt>loops</tt> is a small and lightweight framework for Ruby on Rails, Merb and other ruby
frameworks created to support simple background loops in your application which are usually
used to do some background data processing on your servers (queue workers, batch tasks
<tt>loops</tt> is a small and lightweight framework for Ruby on Rails, Merb and other ruby
frameworks created to support simple background loops in your application which are usually
used to do some background data processing on your servers (queue workers, batch tasks
processors, etc).

_Warning_: If you use some pre-2.0 version of this plugin, read a dedicated paragraph below.


== What would you use it for?

Originally loops plugin was created to make our own loops code a bit more organized. We used
to have dozens of different modules with methods that were called with script/runner and then
used with nohup and other painful backgrounding techniques. When you have such a number of
loops/workers to run in background it becomes a nightmare to manage them on a regular basis
Originally loops plugin was created to make our own loops code a bit more organized. We used
to have dozens of different modules with methods that were called with script/runner and then
used with nohup and other painful backgrounding techniques. When you have such a number of
loops/workers to run in background it becomes a nightmare to manage them on a regular basis
(restarts, code upgrades, status/health checking, etc).

After a few takes on writing our scripts in a more organized way we were able to generalize most
of the code so now our loops started looking like a classes with a single mandatory public method
called *run*. Everything else (spawning many workers, managing them, logging, backgrounding,
After a few takes on writing our scripts in a more organized way we were able to generalize most
of the code so now our loops started looking like a classes with a single mandatory public method
called *run*. Everything else (spawning many workers, managing them, logging, backgrounding,
pid-files management, etc) is handled by the plugin itself.


== But there are dozens of libraries like this! Why do we need yet another one?

The major idea behind this small project was to create a deadly simple and yet robust framework to
be able to run some tasks in background and do not think about spawning many workers, restarting
them when they die, etc. So, if you need to be able to run either one or many copies of your worker
and you do not want to think about re-spawning your scripts when they die and you do not want to
spend megabytes of RAM on separate copies of Ruby interpreter (when you run each copy of your
loop as a separate process controlled by monit/god/etc), then you should try this framework --
The major idea behind this small project was to create a deadly simple and yet robust framework to
be able to run some tasks in background and do not think about spawning many workers, restarting
them when they die, etc. So, if you need to be able to run either one or many copies of your worker
and you do not want to think about re-spawning your scripts when they die and you do not want to
spend megabytes of RAM on separate copies of Ruby interpreter (when you run each copy of your
loop as a separate process controlled by monit/god/etc), then you should try this framework --
you're going to like it.


Expand All @@ -40,7 +40,7 @@ There are two options when approaching db-charmer installation:
* using the gem (recommended)
* install as a Rails plugin

To install as a gem, add this to your environment.rb:
To install as a gem, add this to your environment.rb:

config.gem 'loops'

Expand All @@ -56,10 +56,10 @@ This will install the whole package in your vendor/plugins directory.
For merb applications, just check out the code and place it to the vendor/plugins directory.


After you are done with the installation, you need to generate binary and configuration
After you are done with the installation, you need to generate binary and configuration
files by running:

./script/generate loops
./script/generate loops

This will create the following list of files:
* <tt>./script/loops</tt> - binary file that will be used to manage your loops
Expand All @@ -70,7 +70,7 @@ This will create the following list of files:

== How to use?

Here is a simple loop scaffold for you to start from (put this file to
Here is a simple loop scaffold for you to start from (put this file to
<tt>app/loops/hello_world_loop.rb</tt>):

class HelloWorldLoop < Loops::Base
Expand All @@ -83,7 +83,7 @@ Here is a simple loop scaffold for you to start from (put this file to
end
end

When you have your loop ready to use, add the following lines to your (maybe empty yet)
When you have your loop ready to use, add the following lines to your (maybe empty yet)
<tt>config/loops.yml</tt> file:

hello_world:
Expand Down Expand Up @@ -113,26 +113,26 @@ This is it! To manage your loop, just run one of the following commands:
$ ./script/loops help


_Notice:_ If you use loops as a gem, you will get a system binary called <tt>loops</tt> which you
_Notice:_ If you use loops as a gem, you will get a system binary called <tt>loops</tt> which you
could use instead of your <tt>./script/loops<tt> binary:

$ loops help


== How to run more than one worker?

If you want to have more than one copy of your worker running, that is as simple as adding one
If you want to have more than one copy of your worker running, that is as simple as adding one
option to your loop configuration:

hello_world:
type: simple
sleep_period: 10
workers_number: 1
workers_number: 1

This <tt>workers_number</tt> option would tell loops manager to spawn more than one copy of
your loop and run them in parallel. The only thing you'd need to do is to remember about
concurrent work of your loops. For example, if you have some kind of database table with elements
you need to process, you can create a simple database-based locks system or use any
This <tt>workers_number</tt> option would tell loops manager to spawn more than one copy of
your loop and run them in parallel. The only thing you'd need to do is to remember about
concurrent work of your loops. For example, if you have some kind of database table with elements
you need to process, you can create a simple database-based locks system or use any
memcache-based locks.


Expand All @@ -151,7 +151,7 @@ the loop class to execute:
loop_name: some_module/my_worker
language: French

Now two independent sets of loops are using the same class <tt>SomeModule::MyWorkerLoop</tt>
Now two independent sets of loops are using the same class <tt>SomeModule::MyWorkerLoop</tt>
customized by the language parameter.


Expand All @@ -167,9 +167,9 @@ We use monit to keep loop monitors runnings. You could use something like this i

== ActiveMQ-based workers? What's that?

In some of our worker loops we poll ActiveMQ queue and process its items to perform some
asynchronous operations. So, to make it simpler for us to create such a workers, we've created
really simple loops class extension that wraps your code with basic queue polling/acknowledging
In some of our worker loops we poll ActiveMQ queue and process its items to perform some
asynchronous operations. So, to make it simpler for us to create such a workers, we've created
really simple loops class extension that wraps your code with basic queue polling/acknowledging
code and as the result, you can create a loops like this:

class MyQueueLoop < Loops::Queue
Expand All @@ -190,48 +190,49 @@ With configs like this:
port: 61613
queue_name: blah

Of course, this solution scales perfectly and to make your queue processing faster you just
Of course, this solution scales perfectly and to make your queue processing faster you just
need to add more workers (by adding <tt>workers_number: N</tt> option).

_Warning_: This type of loops requires you to have the <tt>stomp</tt> gem installed in your
_Warning_: This type of loops requires you to have the <tt>stomp</tt> gem installed in your
system.


== There is this <tt>workers_engine</tt> option in the config file. What do you use it for?

There are two so called "workers engines" in loops: <tt>fork</tt> and <tt>thread</tt>.
They're used to control the way process manager would spawn new loops workers:
with the <tt>fork</tt> engine we'll load all the loops classes and then fork ruby interpreter
as many times as many workers we need. With the <tt>thread</tt> engine we'd do Thread.new
instead of forking. Thread engine could be useful if you are sure your loop won't lock
ruby interpreter (it does not do native calls, etc) or if you use some interpreter that does
There are two so called "workers engines" in loops: <tt>fork</tt> and <tt>thread</tt>.
They're used to control the way process manager would spawn new loops workers:
with the <tt>fork</tt> engine we'll load all the loops classes and then fork ruby interpreter
as many times as many workers we need. With the <tt>thread</tt> engine we'd do Thread.new
instead of forking. Thread engine could be useful if you are sure your loop won't lock
ruby interpreter (it does not do native calls, etc) or if you use some interpreter that does
not support forks (like jruby).

The default engine is <tt>fork</tt>.


== What Ruby implementations does it work for?

We've tested and used the plugin on MRI 1.8.6/1.8.7 and on JRuby 1.4.0. At this point we do
not support demonization in JRuby and never tested the code on Ruby 1.9. Obviously because
of JVM limitations you won't be able to use <tt>fork</tt> workers engine in JRuby, but
We've tested and used the plugin on MRI 1.8.6/1.8.7 and on JRuby 1.4.0. At this point we do
not support demonization in JRuby and never tested the code on Ruby 1.9. Obviously because
of JVM limitations you won't be able to use <tt>fork</tt> workers engine in JRuby, but
threaded workers do pretty well.


== Migrating from pre-2.0 releases

Before version 2.0 has been released, this code was developed as a Rails plugin only and
did not have any versions numbering system in place. So we call all those old versions
a pre-2.0 releases. If you use one of those relases (if your loops plugin does not have
the <tt>VERSION.yml</tt> file in the root directory), be careful when upgrading because
there are a few incompatible changes we have made in the loops command: <tt>-h</tt>,
<tt>-a</tt>, <tt>-s</tt>, <tt>-L</tt> and <tt>-l</tt> options were deprecated
and replaced with a friendlier word commands. Use <tt>loops help</tt> to get help.
Before version 2.0 has been released, this code was developed as a Rails plugin only and
did not have any versions numbering system in place. So we call all those old versions
a pre-2.0 releases. If you use one of those relases (if your loops plugin does not have
the <tt>VERSION.yml</tt> file in the root directory), be careful when upgrading because
there are a few incompatible changes we have made in the loops command: <tt>-h</tt>,
<tt>-a</tt>, <tt>-s</tt>, <tt>-L</tt> and <tt>-l</tt> options were deprecated
and replaced with a friendlier word commands.
Use <tt>loops help</tt> to get help.


== Who are the authors?

This plugin has been created in Scribd.com for our internal use and then the sources were opened
This plugin has been created in Scribd.com for our internal use and then the sources were opened
for other people to use. All the code in this package has been developed by Alexey Kovyrin,
Dmytro Shteflyuk and Alexey Verkhovsky for Scribd.com and is released under the MIT license.
For more details, see LICENSE file.

0 comments on commit 9b3779b

Please sign in to comment.