Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate from Sinatra to a Rack application #3075

Merged
merged 24 commits into from Aug 24, 2016
Merged
Changes from 22 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
9ea167d
Migrate Sidekiq::Web to a pure Rack application
badosu Jul 26, 2016
821abb2
Merge branch 'remove-sinatra' of https://github.com/badosu/sidekiq in…
mperham Jul 26, 2016
20e1035
Implement several extension points
mperham Jul 27, 2016
91fd77d
Minor cleanups and improve extension compatibility
badosu Jul 27, 2016
d05066d
Implement compatibility interface with sinatra
badosu Jul 29, 2016
e8cdbd8
Add necessary web infrastructure to support Pro/Ent
mperham Jul 29, 2016
9ae228c
Send Content-Length header
badosu Jul 29, 2016
cb59b5b
Cache ERB templates
badosu Jul 29, 2016
64fa207
Remove unnecessary constraints attribute
badosu Jul 29, 2016
44614b5
Implement additional methods and optimize router
badosu Jul 29, 2016
a9f0cca
Change template rendering engine
badosu Jul 29, 2016
921031b
Change template rendering engine
badosu Jul 29, 2016
38ea3b5
Merge branch 'badosu-remove-sinatra' of github.com:mperham/sidekiq in…
mperham Jul 29, 2016
34664e9
Add simple Sidekiq::Web example
badosu Jul 30, 2016
288db95
Add Sidekiq::Web.use
badosu Jul 30, 2016
290175e
Test authorization and allow instance middlewares
badosu Jul 30, 2016
d2c1f49
web compat with Pro/Enterprise
mperham Aug 1, 2016
5cff7e7
Fix referencing issue
badosu Aug 1, 2016
8d8fc59
Allow for customizing session and protection
badosu Aug 1, 2016
29b3577
Make session and protection the first middlewares
badosu Aug 2, 2016
a742e96
Capture only param in route :param.extension
badosu Aug 2, 2016
a781492
Unescape PATH_INFO
badosu Aug 2, 2016
6caed02
notes, bump
mperham Aug 20, 2016
2c31448
merge master
mperham Aug 20, 2016
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -0,0 +1,5 @@
source 'https://rubygems.org'

gem 'sidekiq', path: '../../'
gem 'thin'
gem 'pry'
@@ -0,0 +1,14 @@
require 'sidekiq/web'
require 'redis'

$redis = Redis.new

class SinatraWorker
include Sidekiq::Worker

def perform(msg="lulz you forgot a msg!")
$redis.lpush("sinkiq-example-messages", msg)
end
end

run Sidekiq::Web
@@ -1,26 +1,28 @@
# frozen_string_literal: true
require 'erb'
require 'yaml'
require 'sinatra/base'

require 'sidekiq'
require 'sidekiq/api'
require 'sidekiq/paginator'
require 'sidekiq/web_helpers'
require 'sidekiq/web/helpers'

module Sidekiq
class Web < Sinatra::Base
include Sidekiq::Paginator
require 'sidekiq/web/router'
require 'sidekiq/web/action'
require 'sidekiq/web/application'

enable :sessions
use ::Rack::Protection, :use => :authenticity_token unless ENV['RACK_ENV'] == 'test'
require 'rack/protection'

set :root, File.expand_path(File.dirname(__FILE__) + "/../../web")
set :public_folder, proc { "#{root}/assets" }
set :views, proc { "#{root}/views" }
set :locales, ["#{root}/locales"]
require 'rack/builder'
require 'rack/file'
require 'rack/session/cookie'

helpers WebHelpers
module Sidekiq
class Web
ROOT = File.expand_path("#{File.dirname(__FILE__)}/../../web")
VIEWS = "#{ROOT}/views".freeze
LOCALES = ["#{ROOT}/locales".freeze]
LAYOUT = "#{VIEWS}/layout.erb".freeze
ASSETS = "#{ROOT}/assets".freeze

DEFAULT_TABS = {
"Dashboard" => '',
@@ -32,6 +34,18 @@ class Web < Sinatra::Base
}

class << self
def settings
self
end

def middlewares
@middlewares ||= []
end

def use(*middleware_args, &block)
middlewares << [middleware_args, block]
end

def default_tabs
DEFAULT_TABS
end
@@ -41,227 +55,94 @@ def custom_tabs
end
alias_method :tabs, :custom_tabs

attr_accessor :app_url
end

get "/busy" do
erb :busy
end

post "/busy" do
if params['identity']
p = Sidekiq::Process.new('identity' => params['identity'])
p.quiet! if params[:quiet]
p.stop! if params[:stop]
else
processes.each do |pro|
pro.quiet! if params[:quiet]
pro.stop! if params[:stop]
end
def locales
@locales ||= LOCALES
end
redirect "#{root_path}busy"
end

get "/queues" do
@queues = Sidekiq::Queue.all
erb :queues
end

get "/queues/:name" do
halt 404 unless params[:name]
@count = (params[:count] || 25).to_i
@name = params[:name]
@queue = Sidekiq::Queue.new(@name)
(@current_page, @total_size, @messages) = page("queue:#{@name}", params[:page], @count)
@messages = @messages.map { |msg| Sidekiq::Job.new(msg, @name) }
erb :queue
end

post "/queues/:name" do
Sidekiq::Queue.new(params[:name]).clear
redirect "#{root_path}queues"
end

post "/queues/:name/delete" do
Sidekiq::Job.new(params[:key_val], params[:name]).delete
redirect_with_query("#{root_path}queues/#{params[:name]}")
end

get '/morgue' do
@count = (params[:count] || 25).to_i
(@current_page, @total_size, @dead) = page("dead", params[:page], @count, reverse: true)
@dead = @dead.map { |msg, score| Sidekiq::SortedEntry.new(nil, score, msg) }
erb :morgue
end

get "/morgue/:key" do
halt 404 unless params['key']
@dead = Sidekiq::DeadSet.new.fetch(*parse_params(params['key'])).first
redirect "#{root_path}morgue" if @dead.nil?
erb :dead
end

post '/morgue' do
redirect request.path unless params['key']

params['key'].each do |key|
job = Sidekiq::DeadSet.new.fetch(*parse_params(key)).first
retry_or_delete_or_kill job, params if job
def views
@views ||= VIEWS
end
redirect_with_query("#{root_path}morgue")
end

post "/morgue/all/delete" do
Sidekiq::DeadSet.new.clear
redirect "#{root_path}morgue"
end

post "/morgue/all/retry" do
Sidekiq::DeadSet.new.retry_all
redirect "#{root_path}morgue"
end

post "/morgue/:key" do
halt 404 unless params['key']
job = Sidekiq::DeadSet.new.fetch(*parse_params(params['key'])).first
retry_or_delete_or_kill job, params if job
redirect_with_query("#{root_path}morgue")
end

def session_secret=(secret)
@session_secret = secret
end

get '/retries' do
@count = (params[:count] || 25).to_i
(@current_page, @total_size, @retries) = page("retry", params[:page], @count)
@retries = @retries.map { |msg, score| Sidekiq::SortedEntry.new(nil, score, msg) }
erb :retries
attr_accessor :app_url, :session_secret, :redis_pool
attr_writer :locales, :views
end

get "/retries/:key" do
@retry = Sidekiq::RetrySet.new.fetch(*parse_params(params['key'])).first
redirect "#{root_path}retries" if @retry.nil?
erb :retry
def settings
self.class.settings
end

post '/retries' do
redirect request.path unless params['key']

params['key'].each do |key|
job = Sidekiq::RetrySet.new.fetch(*parse_params(key)).first
retry_or_delete_or_kill job, params if job
end
redirect_with_query("#{root_path}retries")
def use(*middleware_args, &block)
middlewares << [middleware_args, block]
end

post "/retries/all/delete" do
Sidekiq::RetrySet.new.clear
redirect "#{root_path}retries"
def middlewares
@middlewares ||= Web.middlewares.dup
end

post "/retries/all/retry" do
Sidekiq::RetrySet.new.retry_all
redirect "#{root_path}retries"
def call(env)
app.call(env)
end

post "/retries/:key" do
job = Sidekiq::RetrySet.new.fetch(*parse_params(params['key'])).first
retry_or_delete_or_kill job, params if job
redirect_with_query("#{root_path}retries")
def self.call(env)
@app ||= new
@app.call(env)
end

get '/scheduled' do
@count = (params[:count] || 25).to_i
(@current_page, @total_size, @scheduled) = page("schedule", params[:page], @count)
@scheduled = @scheduled.map { |msg, score| Sidekiq::SortedEntry.new(nil, score, msg) }
erb :scheduled
def app
@app ||= build
end

get "/scheduled/:key" do
@job = Sidekiq::ScheduledSet.new.fetch(*parse_params(params['key'])).first
redirect "#{root_path}scheduled" if @job.nil?
erb :scheduled_job_info
def self.register(extension)
extension.registered(WebApplication)
end

post '/scheduled' do
redirect request.path unless params['key']
private

params['key'].each do |key|
job = Sidekiq::ScheduledSet.new.fetch(*parse_params(key)).first
delete_or_add_queue job, params if job
def using?(middleware)
middlewares.any? do |(m,_)|
m.kind_of?(Array) && (m[0] == middleware || m[0].kind_of?(middleware))
end
redirect_with_query("#{root_path}scheduled")
end

post "/scheduled/:key" do
halt 404 unless params['key']
job = Sidekiq::ScheduledSet.new.fetch(*parse_params(params['key'])).first
delete_or_add_queue job, params if job
redirect_with_query("#{root_path}scheduled")
end

get '/' do
@redis_info = redis_info.select{ |k, v| REDIS_KEYS.include? k }
stats_history = Sidekiq::Stats::History.new((params[:days] || 30).to_i)
@processed_history = stats_history.processed
@failed_history = stats_history.failed
erb :dashboard
end

REDIS_KEYS = %w(redis_version uptime_in_days connected_clients used_memory_human used_memory_peak_human)
def build
middlewares = self.middlewares
klass = self.class

get '/dashboard/stats' do
redirect "#{root_path}stats"
end

get '/stats' do
sidekiq_stats = Sidekiq::Stats.new
redis_stats = redis_info.select { |k, v| REDIS_KEYS.include? k }

content_type :json
Sidekiq.dump_json(
sidekiq: {
processed: sidekiq_stats.processed,
failed: sidekiq_stats.failed,
busy: sidekiq_stats.workers_size,
processes: sidekiq_stats.processes_size,
enqueued: sidekiq_stats.enqueued,
scheduled: sidekiq_stats.scheduled_size,
retries: sidekiq_stats.retry_size,
dead: sidekiq_stats.dead_size,
default_latency: sidekiq_stats.default_queue_latency
},
redis: redis_stats
)
end
unless using?(::Rack::Protection) || ENV['RACK_ENV'] == 'test'
middlewares.unshift [[::Rack::Protection, { use: :authenticity_token }], nil]
end

get '/stats/queues' do
queue_stats = Sidekiq::Stats::Queues.new
unless using? ::Rack::Session::Cookie
unless secret = Web.session_secret

This comment has been minimized.

Copy link
@badosu

badosu Aug 18, 2016

Author Contributor

@mperham I guess you wanted to comment here?

I am not sure what is exactly the use case, but if sharing the session is required I think this will not work at all, because since Rails 4 it uses it's own middleware.

Using Sidekiq::Web::use users can follow the approach used in these tutorials: https://robots.thoughtbot.com/how-to-share-a-session-between-sinatra-and-rails http://stderr.timfischbach.de/2013/09/14/rails-4-and-sinatra-session-sharing.html

However, they should have a way to prevent usage of the ::Rack::Session::Cookie middleware, that is not implemented today.

If just reusing the same session secret is the use case, then I guess this would be a more straightforward way to configure it, as it enables setting more options directly:

### condiftion if Rails
Sidekiq::Web.use Rack::Session::Cookie, secret: 'v3rys3cr31', host: 'nicehost.org'
###

We might use this other approach as well: http://ap4y.github.io/2013/11/03/integrating-multiple-ruby-web-applications.html

Let me know what are the use cases for this that I'll take a look at what we can do and how to best integrate with Rails.

This comment has been minimized.

Copy link
@mperham

mperham Aug 18, 2016

Owner

Thanks for those links, that's really helpful. mounting in config/routes.rb is the main use case I wanted to handle; sounds like this is automatically handled by Rails.

This comment has been minimized.

Copy link
@mperham

mperham Aug 18, 2016

Owner

Oh, maybe we shouldn't set up any session middleware at all by default. Rails is the 90% use case and should provide one to us. We can document that config.ru users need to set up a session handler and we could provide a simple method to do it for them, e.g. Sidekiq::Web.default_session_handler!(secret).

require 'securerandom'
secret = SecureRandom.hex(64)
end

content_type :json
Sidekiq.dump_json(
queue_stats.lengths
)
end
middlewares.unshift [[::Rack::Session::Cookie, { secret: secret }], nil]
end

private
::Rack::Builder.new do
%w(stylesheets javascripts images).each do |asset_dir|
map "/#{asset_dir}" do
run ::Rack::File.new("#{ASSETS}/#{asset_dir}")
end
end

def retry_or_delete_or_kill job, params
if params['retry']
job.retry
elsif params['delete']
job.delete
elsif params['kill']
job.kill
end
end
middlewares.each {|middleware, block| use *middleware, &block }

def delete_or_add_queue job, params
if params['delete']
job.delete
elsif params['add_to_queue']
job.add_to_queue
run WebApplication.new(klass)
end
end
end

Sidekiq::WebApplication.helpers WebHelpers
Sidekiq::WebApplication.helpers Sidekiq::Paginator

Sidekiq::WebAction.class_eval "def _render\n#{ERB.new(File.read(Web::LAYOUT)).src}\nend"
end

if defined?(::ActionDispatch::Request::Session) &&
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.