-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Migrate from Sinatra to a Rack application #3075
Conversation
Migrate Sidekiq::Web a pure Rack application to avoid sinatra as dependency. rack-protection is still needed. The application is mounted on top of Rack::Builder, mantaining all of the previous http interface. Rack apps being used: - Rack::File to serve assets - Rack::Session::Cookie, the secret can be configured via Sidekiq::Web.session_secret - Rack::Protection, same as before when using sinatra - Sidekiq::WebApplication, described below. Sidekiq::WebApplication is a very simple rack application composed of a Sidekiq::WebRouter and a Sidekiq::WebAction dispatcher. This terminology was adopted to be able to mantain Sidekiq::Web as a Rack app. The Router is heavily inspired on Rack::Router[0] (and in many parts identical), however not being retrocompatible. The Action is a wrapper to provide convenience, DRY code and maintain the old interface. I tried to mantain most of the old application structures so that customizations and monkey-patches are easily adjustable or even further work be done to enforce retrocompatibility. Testing welcome! 0: https://github.com/pjb3/rack-router
…to badosu-remove-sinatra
|
I believe Sinatra uses throw/catch to implement halt. Would it be possible to keep the same pattern? |
Do you have a list of known Sidekiq extensions so that I can take a look at what I need to do to support them all? |
You could look thru the Related Projects wiki page.
|
@mperham After some time thinking about it, I`ve decided to make this change totally compatible with the previous sinatra implementation, these are the major points I've identified so far:
I'll take a look at this on this weekend. Feel free to push forward if you want as well. |
That's great, do what you feel is right. A change this big deserves a major version bump, since it'll for sure break someone, but do what you can to minimize the upgrade pain. |
1328286
to
5c3a6bd
Compare
Okay, I am almost finished now, besides all those points above I had to:
These web extensions are passing the tests and are looking fine:
Needs further testing and adjust:
Not supported (requires extensive changes):
Not tested yet: |
5c3a6bd
to
d05066d
Compare
Wow, you're incredibly thorough. I'm impressed. 🍰 I'll review soon and see what else needs to be done to make it work with Sidekiq Pro and Enterprise's UI extensions. |
Can you benchmark a few different pages so we can see how performance looks vs Sinatra? |
@mperham I don't know of It'll be a great difference as there aren't any optimizations and also I've sacrificed performance in favour of compatibility with |
Updated PR with changes to make Sidekiq Pro and Sidekiq Enterprise run green. Notably it took me about an hour to figure out this little tidbit, necessary to run multiple Sidekiq::Web instances in the same process: http://stackoverflow.com/questions/5643907/what-is-x-cascade-header |
Surprisingly enough, peformance is worse after this change. Using a config.ru just mounting the application and thin, with
Possible causes:
|
I just updated the PR with 2 changes:
|
An easy optimization to make is to split the routes based on method.
|
0faf8b6
to
29b3577
Compare
With the last change, |
@mperham I am pretty confident on the status of this implementation. Let me know if there are any pending issues. |
I'm not aware of any issues. It'll take me a week or two to get everything tested as this is a major release so be patient. |
To test this branch, put this in your Gemfile: gem 'sidekiq', github: 'mperham/sidekiq', branch: 'badosu-remove-sinatra' |
get '/stats/queues' do | ||
queue_stats = Sidekiq::Stats::Queues.new | ||
unless using? ::Rack::Session::Cookie | ||
unless secret = Web.session_secret |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@mperham I guess you wanted to comment here?
I am not sure what is exactly the use case, but if sharing the session is required I think this will not work at all, because since Rails 4 it uses it's own middleware.
Using Sidekiq::Web::use
users can follow the approach used in these tutorials: https://robots.thoughtbot.com/how-to-share-a-session-between-sinatra-and-rails http://stderr.timfischbach.de/2013/09/14/rails-4-and-sinatra-session-sharing.html
However, they should have a way to prevent usage of the ::Rack::Session::Cookie
middleware, that is not implemented today.
If just reusing the same session secret is the use case, then I guess this would be a more straightforward way to configure it, as it enables setting more options directly:
### condiftion if Rails
Sidekiq::Web.use Rack::Session::Cookie, secret: 'v3rys3cr31', host: 'nicehost.org'
###
We might use this other approach as well: http://ap4y.github.io/2013/11/03/integrating-multiple-ruby-web-applications.html
Let me know what are the use cases for this that I'll take a look at what we can do and how to best integrate with Rails.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for those links, that's really helpful. mounting in config/routes.rb is the main use case I wanted to handle; sounds like this is automatically handled by Rails.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh, maybe we shouldn't set up any session middleware at all by default. Rails is the 90% use case and should provide one to us. We can document that config.ru
users need to set up a session handler and we could provide a simple method to do it for them, e.g. Sidekiq::Web.default_session_handler!(secret)
.
I'm finding two issues in my testing. This is mounted in a rails 5 application.
|
The sharding issue is fixed on Pro master. I'll coordinate the gem releases to make sure this isn't a problem.
|
@badosu Any comments on (2) above? |
We shouldn't be loading any ETag middleware at all. I don't really see much benefit for this specific webapp. |
@mperham If you're asking about the ETag issue, I don't know how it can happen if we control all headers and we're not passing any ETag response headers at all. Only if this is something being setup by Pro or a middleware customization. On a side note, as soon as this PR lands in master I am gonna send a PR to to each plugin that relied on (now) unsupported Sinatra functionality. |
Could this be part of the rails middleware stack when mounted in a rails app?
|
Ok, gonna merge this as is and let it sit on master for a few weeks. |
Thank you @badosu for your incredible work on this PR. It's a huge change and you've put an amazing amount of work into polishing it. I hope it proves stable! |
@johnathanludwig Do you have any idea how we could reproduce this problem (the Etag one)? |
@badosu These were the steps I ran. Along the way something would end up being cached by the browser and not display the expected results:
|
Continuing discussion from #3069
About extensions:
home_path
on redirect calls, as seen hereeval
.params
includes only the querystring parameters accessible via String keys instead of symbolsroute_params
includes those that come from the route, accessible via symbolshalt
call should be replaced tonext
, e.g.halt 404 -> next 404
We will have to decide what to deprecate and what not, fortunately it looks pretty straightforward to upgrade old extensions to the new model.