Skip to content
Distributed task execution system for complex workflows.
Ruby CSS HTML Other
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin
examples
lib
spec
.gitignore
.rspec
.travis.yml
CHANGELOG.md
CODE_OF_CONDUCT.md
Gemfile
LICENSE
README.md
Rakefile
qyu.gemspec

README.md

Qyu 九

Gem Version Build Status Maintainability codecov Press

Requirements:

  • Ruby 2.4.0 or newer

Installation

Add this line to your application's Gemfile:

gem 'qyu'

And then execute:

$ bundle

Or install it yourself as:

$ gem install qyu

Configuration

To start using Qyu; you need a queue configuration and a state store configuration. Here's an example:

Qyu.configure(
  queue: {
    type: :memory
  },
  store: {
    type: :memory,
    lease_period: 60
  },
  # optional Defaults to STDOUT
  logger: Logger.new(STDOUT)
)

Usage

[1] Connect all instances to the same state store and message queue
[2] Create a workflow
[3] Initialize a worker as follows

w = Qyu::Worker.new do
  # callbacks
  callback :execute, :before do
    Qyu.logger.info 'Waiting for task..'
  end

  callback :execute, :after do
    Qyu.logger.info 'Done'
  end

  # payload validation
  validates :times, presence: true, type: :integer

  # failure queue
  failure_queue false

  # timeout in seconds for each task processed by the worker
  timeout 120
end

w.work('queue-name') do |task|
  # to get the payload passed to the task
  task.payload
  # = { 'param_1' => true, 'param_2': [10, 11, 12], 'param_3' => 2 }

  # get the job
  task.job

  # to manually start a task
  task.job.create_task(task, 'next-task-name', payload)
rescue StandardError => ex
  # If you rescue the error for debugging or reporting purposes, you have to raise it at the end
  #
  # do something
  #
  raise ex
end

[4] Start worker
[5] Start creating Jobs using the previously created workflow

Plugins

The memory queue and store is just for testing purposes. For production; use one of the following:

Stores

ActiveRecord: https://github.com/QyuTeam/qyu-store-activerecord
Redis: https://github.com/QyuTeam/qyu-store-redis

Queues

Amazon SQS: https://github.com/QyuTeam/qyu-queue-sqs
Redis: https://github.com/QyuTeam/qyu-queue-redis

Glossary

Workflow

The workflow specifies the entry points (starts), the tasks, their order, eventual dependencies between them, and synchronisation conditions.

Job

A job is essentially a collection of tasks and an initial JSON payload.

Task

A task is one unit of work. It is an instance of an entry from a workflow. You can think of it as the workflow's entries define the classes, while a task is a materialised instance of it, saved in the state store and enqueued on the message queue.

In the state store a task has:

  • id
  • name - it appears as the key in the workflow's tasks
  • queue_name - the queue where the task was enqueued on creation
  • payload - the entry/input parameters for the particular task
  • parent_task_id - the ID of the task which created/enqueued the current task

When a task is created (saved & enqueued) then its id is put in a JSON message { task_id: task.id} and enqueued on the specified task's message queue. When a worker picks up the message from the queue, decodes the task id, and retrieves it from the state store.

Worker

A worker is sitting on a queue, waiting for something.

Sync Worker

A worker waiting for other workers to finish

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/QyuTeam/qyu.

You can’t perform that action at this time.