Migrate from Sinatra to a Rack application #3075

badosu · 2016-07-27T19:39:31Z

Continuing discussion from #3069

About extensions:

It is not needed anymore to include explicitly home_path on redirect calls, as seen here
I had to reinclude code to set local variables inside partials, however It is not compatible with ruby 2.0.0 and I can't see a way to do it without using eval.
params includes only the querystring parameters accessible via String keys instead of symbols
route_params includes those that come from the route, accessible via symbols
The halt call should be replaced to next, e.g. halt 404 -> next 404

We will have to decide what to deprecate and what not, fortunately it looks pretty straightforward to upgrade old extensions to the new model.

Migrate Sidekiq::Web a pure Rack application to avoid sinatra as dependency. rack-protection is still needed. The application is mounted on top of Rack::Builder, mantaining all of the previous http interface. Rack apps being used: - Rack::File to serve assets - Rack::Session::Cookie, the secret can be configured via Sidekiq::Web.session_secret - Rack::Protection, same as before when using sinatra - Sidekiq::WebApplication, described below. Sidekiq::WebApplication is a very simple rack application composed of a Sidekiq::WebRouter and a Sidekiq::WebAction dispatcher. This terminology was adopted to be able to mantain Sidekiq::Web as a Rack app. The Router is heavily inspired on Rack::Router[0] (and in many parts identical), however not being retrocompatible. The Action is a wrapper to provide convenience, DRY code and maintain the old interface. I tried to mantain most of the old application structures so that customizations and monkey-patches are easily adjustable or even further work be done to enforce retrocompatibility. Testing welcome! 0: https://github.com/pjb3/rack-router

…to badosu-remove-sinatra

mperham · 2016-07-27T19:48:43Z

If you can't do it with Ruby 2.0, how does Sinatra do it? Does it use eval?
Ok about halt. You should add a halt method that raises an informative error.

mperham · 2016-07-27T19:55:43Z

I believe Sinatra uses throw/catch to implement halt. Would it be possible to keep the same pattern?

badosu · 2016-07-27T23:46:19Z

Do you have a list of known Sidekiq extensions so that I can take a look at what I need to do to support them all?

mperham · 2016-07-27T23:58:28Z

You could look thru the Related Projects wiki page.

On Jul 27, 2016, at 16:46, Amadeus Folego notifications@github.com wrote:

Do you have a list of known Sidekiq extensions so that I can take a look at what I need to do to support them all?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

badosu · 2016-07-28T18:07:45Z

@mperham After some time thinking about it, I`ve decided to make this change totally compatible with the previous sinatra implementation, these are the major points I've identified so far:

The redirect call should not include the custom base path
The render method should accept the same arguments as the previous impl.
Implement halt
params should be an indifferent access hash and also include the params from the route

I'll take a look at this on this weekend. Feel free to push forward if you want as well.

mperham · 2016-07-28T22:24:04Z

That's great, do what you feel is right. A change this big deserves a major version bump, since it'll for sure break someone, but do what you can to minimize the upgrade pain.

badosu · 2016-07-29T06:47:35Z

Okay, I am almost finished now, besides all those points above I had to:

Implement content_type
Implement matchers on before and after hooks
Change add_to_head
erb implementation was trickier than I thought, as it was used for templates with and without layouts depending if it was called inside a template or not
Add settings

These web extensions are passing the tests and are looking fine:

Needs further testing and adjust:

sidekiq-scheduler Uses redirect to
sidekiq_status Uses redirect to
sidekiq-enqueuer

Not supported (requires extensive changes):

sidekiq-benchmark Uses sinatra-assetpack

Not tested yet:

sidekiq-prioritized_queues

mperham · 2016-07-29T15:54:28Z

Wow, you're incredibly thorough. I'm impressed. 🍰

I'll review soon and see what else needs to be done to make it work with Sidekiq Pro and Enterprise's UI extensions.

mperham · 2016-07-29T15:57:48Z

Can you benchmark a few different pages so we can see how performance looks vs Sinatra?

badosu · 2016-07-29T16:17:04Z

@mperham I don't know of It'll be a great difference as there aren't any optimizations and also I've sacrificed performance in favour of compatibility with sinatra (see #params implementation)

mperham · 2016-07-29T19:11:12Z

Updated PR with changes to make Sidekiq Pro and Sidekiq Enterprise run green. Notably it took me about an hour to figure out this little tidbit, necessary to run multiple Sidekiq::Web instances in the same process:

http://stackoverflow.com/questions/5643907/what-is-x-cascade-header

badosu · 2016-07-29T19:15:57Z

Surprisingly enough, peformance is worse after this change.

Using a config.ru just mounting the application and thin, with ab -l -c 4 -n 2000 http://0.0.0.0:3000/

~400 req/s with Sinatra
~200 req/s new impl.
~300 req/s with a simple ERB cache

Possible causes:

The Router uses a regexp match even when it is not needed
Tilt can have a better template caching system than the one I am implementing
Rack::Builder

badosu · 2016-07-29T19:25:15Z

I just updated the PR with 2 changes:

Set content-length header, as I could not even test the application with ab without it and it is a tiny implementation
Implement simple caching of erb templates

mperham · 2016-07-29T19:25:32Z

An easy optimization to make is to split the routes based on method.

On Jul 29, 2016, at 12:15, Amadeus Folego notifications@github.com wrote:

Surprisingly enough, peformance is worse after this change.

Using a config.ru just mounting the application and thin, with ab -l -c 4 -n 2000 http://0.0.0.0:3000/

~400 req/s with Sinatra
~200 req/s new impl.
~300 req/s with a simple ERB cache
Possible causes:

The Router uses a regexp match for even when it is not needed
Tilt can have a better template caching system than the one I am implementing
Rack::Builder
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

badosu · 2016-08-02T15:39:57Z

With the last change, sidekiq-statistic should work as well now

badosu · 2016-08-02T18:46:51Z

@mperham I am pretty confident on the status of this implementation.

Let me know if there are any pending issues.

mperham · 2016-08-02T18:55:17Z

I'm not aware of any issues. It'll take me a week or two to get everything tested as this is a major release so be patient.

mperham · 2016-08-18T16:15:08Z

To test this branch, put this in your Gemfile:

gem 'sidekiq', github: 'mperham/sidekiq', branch: 'badosu-remove-sinatra'

badosu · 2016-08-18T19:04:42Z

lib/sidekiq/web.rb

-    get '/stats/queues' do
-      queue_stats = Sidekiq::Stats::Queues.new
+      unless using? ::Rack::Session::Cookie
+        unless secret = Web.session_secret


@mperham I guess you wanted to comment here?

I am not sure what is exactly the use case, but if sharing the session is required I think this will not work at all, because since Rails 4 it uses it's own middleware.

Using Sidekiq::Web::use users can follow the approach used in these tutorials: https://robots.thoughtbot.com/how-to-share-a-session-between-sinatra-and-rails http://stderr.timfischbach.de/2013/09/14/rails-4-and-sinatra-session-sharing.html

However, they should have a way to prevent usage of the ::Rack::Session::Cookie middleware, that is not implemented today.

If just reusing the same session secret is the use case, then I guess this would be a more straightforward way to configure it, as it enables setting more options directly:

### condiftion if Rails Sidekiq::Web.use Rack::Session::Cookie, secret: 'v3rys3cr31', host: 'nicehost.org' ###

We might use this other approach as well: http://ap4y.github.io/2013/11/03/integrating-multiple-ruby-web-applications.html

Let me know what are the use cases for this that I'll take a look at what we can do and how to best integrate with Rails.

Thanks for those links, that's really helpful. mounting in config/routes.rb is the main use case I wanted to handle; sounds like this is automatically handled by Rails.

Oh, maybe we shouldn't set up any session middleware at all by default. Rails is the 90% use case and should provide one to us. We can document that config.ru users need to set up a session handler and we could provide a simple method to do it for them, e.g. Sidekiq::Web.default_session_handler!(secret).

johnathanludwig · 2016-08-18T19:15:09Z

I'm finding two issues in my testing. This is mounted in a rails 5 application.

Passing the redis pool doesn't seem to be working. We mount two web instances. In development each instance points at a different DB on redis. Using this example:

mount Sidekiq::Pro::Web.with(redis_pool: ConnectionPool.new { Redis.new(url: 'redis://localhost:6379/1') }), at: "/sidekiq_primary", as: "/sidekiq_primary"
mount Sidekiq::Pro::Web.with(redis_pool: ConnectionPool.new { Redis.new(url: 'redis://localhost:6379/2') }), at: "/sidekiq_secondary", as: "/sidekiq_secondary"

When I go to sidekiq_primary or sidekiq_secondary, both show the redis instance is redis://localhost:6379/0 instead of 1 or 2.

Chrome is caching the pages sometimes displaying an old response. I think this is because the ETag header is coming back the same even after making changes. In some cases the ETag is being refreshed though. Here are the steps I went through:
1. Go to Queues tab.
2. Queue job in one queue.
3. Refresh page. Queue should show 1 job queued.
4. Click on queue to view job.
5. Delete job. Page should be refreshed and show job removed.
6. Go back to queues. Should say 0 jobs.
7. Delete queue. Page should be refreshed with queue removed.
Performing those steps should result in at least one of those expectations being incorrect due to the caching.

mperham · 2016-08-18T19:27:08Z

The sharding issue is fixed on Pro master. I'll coordinate the gem releases to make sure this isn't a problem.

On Aug 18, 2016, at 12:15, Johnathan Ludwig notifications@github.com wrote:

I'm finding two issues in my testing.

Passing the redis pool doesn't seem to be working. We mount two web instances. In development each instance points at a different DB on redis. Using this example:

mount Sidekiq::Pro::Web.with(redis_pool: ConnectionPool.new { Redis.new(url: 'redis://localhost:6379/1') }), at: "/sidekiq_primary", as: "/sidekiq_primary"
mount Sidekiq::Pro::Web.with(redis_pool: ConnectionPool.new { Redis.new(url: 'redis://localhost:6379/2') }), at: "/sidekiq_secondary", as: "/sidekiq_secondary"
When I go to sidekiq_primary or sidekiq_secondary, both show the redis instance is redis://localhost:6379/0 instead of 1 or 2.

Chrome is caching the pages sometimes displaying an old response. I think this is because the ETag header is coming back the same even after making changes. In some cases the ETag is being refreshed though. Here are the steps I went through:

Go to Queues tab.
Queue job in one queue.
Refresh page. Queue should show 1 job queued.
Click on queue to view job.
Delete job. Page should be refreshed and show job removed.
Go back to queues. Should say 0 jobs.
Delete queue. Page should be refreshed with queue removed.
Performing those steps resulted should result in at least one of those expectations being incorrect due to the caching.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

mperham · 2016-08-20T21:52:49Z

@badosu Any comments on (2) above?

mperham · 2016-08-20T21:54:14Z

We shouldn't be loading any ETag middleware at all. I don't really see much benefit for this specific webapp.

badosu · 2016-08-22T15:35:52Z

@mperham If you're asking about the ETag issue, I don't know how it can happen if we control all headers and we're not passing any ETag response headers at all.

Only if this is something being setup by Pro or a middleware customization.

On a side note, as soon as this PR lands in master I am gonna send a PR to to each plugin that relied on (now) unsupported Sinatra functionality.

mperham · 2016-08-22T15:51:06Z

Could this be part of the rails middleware stack when mounted in a rails app?

On Aug 22, 2016, at 08:35, Amadeus Folego notifications@github.com wrote:

@mperham If you're asking about the ETag issue, I don't know how it can happen if we control all headers and we're not passing any ETag response headers at all.

Only if this is something being setup by Pro or a middleware customization.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

mperham · 2016-08-24T16:41:17Z

Ok, gonna merge this as is and let it sit on master for a few weeks.

mperham · 2016-08-24T16:42:23Z

Thank you @badosu for your incredible work on this PR. It's a huge change and you've put an amazing amount of work into polishing it. I hope it proves stable!

badosu · 2016-08-26T14:01:14Z

@johnathanludwig Do you have any idea how we could reproduce this problem (the Etag one)?

johnathanludwig · 2016-08-26T14:08:19Z

@badosu These were the steps I ran. Along the way something would end up being cached by the browser and not display the expected results:

Go to Queues tab.
Queue one job.
Refresh page. Queues should show 1 job queued.
Click on queue to view job.
Delete job. Page should be refreshed and show job removed.
Go back to queues. Should say 0 jobs.
Delete queue. Page should be refreshed with queue removed.

badosu and others added 4 commits July 26, 2016 11:43

Merge branch 'remove-sinatra' of https://github.com/badosu/sidekiq in…

821abb2

…to badosu-remove-sinatra

Implement several extension points

20e1035

Minor cleanups and improve extension compatibility

91fd77d

badosu changed the title ~~Migrate Sinatra to a Rack application~~ Migrate from Sinatra to a Rack application Jul 27, 2016

badosu force-pushed the badosu-remove-sinatra branch 5 times, most recently from 1328286 to 5c3a6bd Compare July 29, 2016 06:07

Implement compatibility interface with sinatra

d05066d

badosu force-pushed the badosu-remove-sinatra branch from 5c3a6bd to d05066d Compare July 29, 2016 07:52

Add necessary web infrastructure to support Pro/Ent

e8cdbd8

Send Content-Length header

9ae228c

Cache ERB templates

cb59b5b

Remove unnecessary constraints attribute

64fa207

badosu force-pushed the badosu-remove-sinatra branch 2 times, most recently from 0faf8b6 to 29b3577 Compare August 2, 2016 01:44

Capture only param in route :param.extension

a742e96

Unescape PATH_INFO

a781492

spaquet mentioned this pull request Aug 3, 2016

Conflict in Rake version when using Rails 5.0.0 #3089

Closed

badosu reviewed Aug 18, 2016
View reviewed changes

notes, bump

6caed02

merge master

2c31448

mperham merged commit 3f157e2 into master Aug 24, 2016

mperham deleted the badosu-remove-sinatra branch August 24, 2016 16:41

zoras mentioned this pull request Sep 15, 2016

NoMethodError: undefined method `set' for Sidekiq::Web:Class #3139

Closed

tiagoamaro mentioned this pull request Sep 29, 2016

[Feat] Allow devs using local gemfile fameandpartners/website#2051

Merged

6 tasks

mperham mentioned this pull request Nov 21, 2016

Sidekiq Web UI does not starts #3248

Closed

EtienneDepaulis mentioned this pull request Dec 13, 2016

Migrate (back) from Sinatra to a Rack application ? refile/refile#520

Closed

nneufeld mentioned this pull request May 23, 2017

No route matches [POST] "/jobs/grouping/{group}/delete" gzigzigzeo/sidekiq-grouping#28

Closed

brendanshean mentioned this pull request May 10, 2021

LMS Rails 5.0 Prework - Remove sinatra dependency empirical-org/Empirical-Core#7613

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate from Sinatra to a Rack application #3075

Migrate from Sinatra to a Rack application #3075

badosu commented Jul 27, 2016 •

edited

Loading

mperham commented Jul 27, 2016

mperham commented Jul 27, 2016

badosu commented Jul 27, 2016

mperham commented Jul 27, 2016

badosu commented Jul 28, 2016 •

edited

Loading

mperham commented Jul 28, 2016

badosu commented Jul 29, 2016 •

edited

Loading

mperham commented Jul 29, 2016

mperham commented Jul 29, 2016

badosu commented Jul 29, 2016

mperham commented Jul 29, 2016

badosu commented Jul 29, 2016 •

edited

Loading

badosu commented Jul 29, 2016 •

edited

Loading

mperham commented Jul 29, 2016

badosu commented Aug 2, 2016

badosu commented Aug 2, 2016 •

edited

Loading

mperham commented Aug 2, 2016

mperham commented Aug 18, 2016

badosu Aug 18, 2016 •

edited

Loading

mperham Aug 18, 2016

mperham Aug 18, 2016

johnathanludwig commented Aug 18, 2016 •

edited

Loading

mperham commented Aug 18, 2016

mperham commented Aug 20, 2016

mperham commented Aug 20, 2016

badosu commented Aug 22, 2016 •

edited

Loading

mperham commented Aug 22, 2016

mperham commented Aug 24, 2016

mperham commented Aug 24, 2016

badosu commented Aug 26, 2016

johnathanludwig commented Aug 26, 2016

Migrate from Sinatra to a Rack application #3075

Migrate from Sinatra to a Rack application #3075

Conversation

badosu commented Jul 27, 2016 • edited Loading

mperham commented Jul 27, 2016

mperham commented Jul 27, 2016

badosu commented Jul 27, 2016

mperham commented Jul 27, 2016

badosu commented Jul 28, 2016 • edited Loading

mperham commented Jul 28, 2016

badosu commented Jul 29, 2016 • edited Loading

mperham commented Jul 29, 2016

mperham commented Jul 29, 2016

badosu commented Jul 29, 2016

mperham commented Jul 29, 2016

badosu commented Jul 29, 2016 • edited Loading

badosu commented Jul 29, 2016 • edited Loading

mperham commented Jul 29, 2016

badosu commented Aug 2, 2016

badosu commented Aug 2, 2016 • edited Loading

mperham commented Aug 2, 2016

mperham commented Aug 18, 2016

badosu Aug 18, 2016 • edited Loading

Choose a reason for hiding this comment

mperham Aug 18, 2016

Choose a reason for hiding this comment

mperham Aug 18, 2016

Choose a reason for hiding this comment

johnathanludwig commented Aug 18, 2016 • edited Loading

mperham commented Aug 18, 2016

mperham commented Aug 20, 2016

mperham commented Aug 20, 2016

badosu commented Aug 22, 2016 • edited Loading

mperham commented Aug 22, 2016

mperham commented Aug 24, 2016

mperham commented Aug 24, 2016

badosu commented Aug 26, 2016

johnathanludwig commented Aug 26, 2016

badosu commented Jul 27, 2016 •

edited

Loading

badosu commented Jul 28, 2016 •

edited

Loading

badosu commented Jul 29, 2016 •

edited

Loading

badosu commented Jul 29, 2016 •

edited

Loading

badosu commented Jul 29, 2016 •

edited

Loading

badosu commented Aug 2, 2016 •

edited

Loading

badosu Aug 18, 2016 •

edited

Loading

johnathanludwig commented Aug 18, 2016 •

edited

Loading

badosu commented Aug 22, 2016 •

edited

Loading