Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Move background jobs to the 'jobs' branch until fully baked. Not ship…
…ping with Rails 4.0.
- Loading branch information
Showing
25 changed files
with
4 additions
and
724 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -44,5 +44,4 @@ module ActionMailer | |
autoload :MailHelper | ||
autoload :TestCase | ||
autoload :TestHelper | ||
autoload :QueuedMessage | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What constitutes “fully baked”?
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree with @Aupajo on this. Was the queue system intended to be the underpinning to resque and sidekiq as rack is to rails or something else entirely? Wy bother with writing a new queue system when resque and sidekiq are the defaults ?
Personally, I'm switching to sidekiq.
/cc @hone @mperham
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Aupajo when it reaches a point that we think using it is natural, it still need a lot of work and it is a blocker for the release. We think is better to pospone.
@samgranieri nobody is writing a new queue system, Rails queue is a API can be used with sidekiq, resque, or any other queue system. Think about it as what Active Model is for ORMs.
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Both Resque and Sidekiq already had integration with this interface, but there were some issues that were causing problems, and the feature isn't worth holding up the release of Rails 4 for.
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@rafaelfranca I get that it doesn't feel fully baked, but why doesn't it? What was the awkwardness/what were the issues? (Is there an existing thread on this, maybe?)
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is no thread anywhere. The decision is based on discussion between the core members and experiments. We fell this is not ready and we still have a lot of doubts with the usage.
It need more experiment to get released.
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Slipped a knife in to test, and the API was still gooey in the center 😁
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Love them secret discussions on open source projects.
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's how people get shit done without having trolls spoil their appetite for the work.
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You have the code and the rationale, that is open source. While one is always open and thankful for feedback, the project is run by a team, not by a universal house.
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I trust you guys to made the right decision on this, but I also empathize with the sentiment expressed above. Perhaps some failing tests would make clear exactly what needs work and define "fully baked"?
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We were surprised to see this get cut, although we had heard a little ahead of time. My buddy @phlipper and I had gotten the queue stuff backported to Rails 3.2+ and we're close to releasing the gem for it. https://github.com/probablywrong/rails-queue
Perhaps we can take this discussion and passion over there, get it working for Rails 4 & 3.2+, and work out some of the implementation details/wrinkles outside of the main Rails codebase. That would give us a focused place to discuss specific issues. Interested?
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ElsKid - it's not cut; it's on the
jobs
branch. By all means continue work! (Releasing a gem named "rails queue" = confusion, though.)@mikegee - if I could write a test for this API doesn't feel great in my app, I'd be first in line with 100% coverage ;)
Some quick background: I developed a new app that heavily relies on job processing. I used the Rails.queue API from end to end and, ultimately, I felt it's a regression over writing my Resque jobs directly.
Rails.queue[:name] << job
API style reads as "push this job onto this specific queue." What most calling code needs is "run this job later." The calling code shouldn't be concerned with where the job is headed or who executes it, so picking a queue by name is a poor distribution of responsibility (not to mention repetitive).Rails.database
wouldn't make sense and neither doesRails.queue
. If anything, that serves in a backend/adapter role to jobs themselves.On the plus side, as we coped with this issues, we did come up with some nice abstractions for application jobs. Notably, we'll jettison "queueing" from our vocabulary entirely. We want to present an API for submitting and executing jobs. We have more in common with a process scheduler than a task queue.
And, even more promisingly, we saw a couple of key spots where we'd often like to be able to defer work. I'm optimistic that these will have a greater payoff, overall, than a unified job submission/execution API.
before_commit
callback. Russian Doll caching relies on chainingtouch: true
so changing a child record automatically touches its parent, which touches its grandparent, and so on. That means changing N child objects fires off N parent and grandparent touches when only the last touch was needed. So we'd like to batch up the touches and perform the updates in abefore_commit
.TL;DR we did an experiment with Rails.queue and rejected the API yet learned a lot about what we Really Need. Stay tuned!
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jeremy, thank you for the detailed write-up. Our name was to help ease confusion about what it was that we backported. When we wrote it we thought it as around to stay. heh. Still might have some use as a place to kick around ideas.
We found similar strangeness when testing and using the code as well. Specifically the syntax didn't really "fit"with what we wanted either. And the marshaling broke down for us too.
I think the biggest benefit of the Rails.queue idea was that you always had one and areas that need background execution, no matter the size/scope/expense, you could just fire them. You could then say, do this in dev, do something else in test, and something completely different in production but I have one API to just throw jobs into. That said, a better API to do that would be amazing.
Thanks again!
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jeremy Wow, it sounds like the direction this is heading is great. Thanks for the write-up.
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jeremy if Rails provides the adapter changing the API should not be a big deal. See https://gist.github.com/351550
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For the people who wants an abstraction over the available queues, take a look on https://github.com/fnando/qe
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jeremy Thanks for the write up. "rejected the API yet learned a lot about what we Really Need" I liked the idea of the Rails.queue, but love the before_commit and post request ideas.
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@sobrinho qe looks great - this is precisely where the Job backend are heading.
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jeremy thanks SO MUCH for that writeup, explains why it was decided to be not fully baked, with a convincing and reasonable argument, and also explain what you are thinking about future directions.
It is VERY useful for us Rails-using developers to have this, thanks for realizing such and providing.
(Also, thanks very much to Rails core for not releasing an API as part of Rails when you realize the design is not right, we indeed count on your active judgement of these things to keep Rails great. Throwing in half-baked things will help nobody.)
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jeremy thanks for the write up, that reasoning is really convincing and enlightening.
Post-request processing is clever - I've been using
EM.next_tick
when I use an eventmachine-based web server like thin, which works great. No marshaling is necessary, and binding objects in the local scope are available - which simplifies things tremendously. I'd be very happy if we'd get the utility ofEM.next_tick
without depending on EM.Anonymous jobs FTW!
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm looking forward to submitting job work orders. If you combine this with the power of something like hstore to store arbitrary variables for later when the job executes... that's what I'm talking about.
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Russian Doll :touch issue is something which has been giving me troubles today - look forward to something coming of this discussion.
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@dhh Hi David - it's mentioned in Jeremy's comment above f9da785#commitcomment-2370489 I was also referred to this post discussion from another issue recoded here: #8759
I don't think there is a specific link between the two challenges, but Jeremy seems to have come up against the :touch challenges at some point in his adventure here.
f9da785
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@SirRawlins Any update?