Interface to pragmatically setup a queue job for Q4M!
So… What's Q4M
Q4M is a open-sourced MySQL Engine. that attempts to take away some logic from your application by encapsulating it inside of mysql.
The documentation is kinda bad, but the magic that it does in the background is sweeeeeet!
You basically store all of your background jobs inside a MySQL table. that table's engine should be a queue (q4m) engine!
drop table if exists shell_jobs, ruby_jobs; create table shell_jobs (command text) engine=queue; create table ruby_jobs (command text) engine=queue;
That engine will work under some kind of transaction. When you connect to the database the engine will give you a list of all the pending jobs. When you start a transaction you'll receive ONE job to work with. In the mean time the whole table gets locked.. And if you do a 'select *' all results will return emtpy!
This is great, you can never access two jobs at the same time. if another worker/machine connects they'll get a list of jobs except the one is wrapped in the transaction!
So that means that the locking only happens on the connection that is perfoming the transaction but any other workers will get the full list of jobs except the ones that are being processed by other workers.
When the first transaction gets terminated the engine will automatically unlock the table and delete the record.
This is great, because you can have any number of workers working concurrently without making a mess!
Meaning that a job can never be called in twice by anyone, and that you can grow horizontally, the limit is the sky!
The job is done here!
Go do something more interesting like clustering! ;)
Seek & destroy, Seek & destroy!
So what exactly does this gem do?
Gives you an interface to quickly setup how specificly a job is supposed to run!
Basically, you define a class create an method called execute, which receives a job argument. truly a MySQL record in the form of an array, what you do from there is up to you!
if something goes wrong just call the queue_abort method and everything gets rolled back!
kazu@utopia:~$ gem install q4m
require 'rubygems' require 'q4m' # Specify how shell jobs will run class ShellJob < ::Q4m::Queue::Job def execute job queue_abort unless system(job.first) end end host = 'localhost' user = 'root' pass = 'secret' database = 'my_db' # Instantiate and run a job! mysql = Mysql.new host, user, pass, database job = ShellJob.new mysql, Logger.new('q4m.log') job.run
Note that: The ShellJob class will try to connect to the 'shell_jobs' q4m table by default, if you want to use a different table override the table_name method and specify one of your liking!
class ShellJob < ::Q4m::Queue::Job def execute job queue_abort unless system(job.first) end def table_name 'my_cool_table' end end
This is great!
Basically any machine running rails with this code, can pull jobs and do some hard work for you! even if they are not part of your cluster!
The only requirement is that they can access the mysql-database.
and no, you don't need to install that engine locally!
Go farther just create a new job class and run whatever you like!
This is how I installed it.
wget http://q4m.kazuhooku.com/dist/old/mysql-5.1.41-linux-x86_64-glibc23-with-fast-mutexes-q4m-0.8.9.tar.gz unzip *.gz tar -xvf mysql-5.1.41-linux-x86_64-glibc23-with-fast-mutexes-q4m-0.8.9.tar cd q4m-0.8.9-linux-x86_64 sudo cp libqueue_engine.so /usr/lib/mysql/plugin sudo chmod 777 /usr/lib/mysql/plugin/libqueue_engine.so mysql -u root -p -f mysql < support-files/install.sql
You might wanna go to:
And see tha latest on how to get it installed.
just thought of posting it, so you'll get an idea.
Yeah, but what's your distro?
it's right here!
deploy@staging> cat /etc/*-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=10.04 DISTRIB_CODENAME=lucid DISTRIB_DESCRIPTION="Ubuntu 10.04 LTS"