Permalink
Browse files

Move background jobs to the 'jobs' branch until fully baked. Not ship…

…ping with Rails 4.0.
  • Loading branch information...
1 parent 10c0a3b commit f9da785d0b1b22317cfca25c15fb555e9016accb @jeremy jeremy committed Dec 21, 2012
@@ -28,8 +28,6 @@
* Raise an `ActionView::MissingTemplate` exception when no implicit template could be found. *Damien Mathieu*
-* Asynchronously send messages via the Rails Queue *Brian Cardarella*
-
* Allow callbacks to be defined in mailers similar to `ActionController::Base`. You can configure default
settings, headers, attachments, delivery settings or change delivery using
`before_filter`, `after_filter` etc. *Justin S. Leitgeb*
@@ -44,5 +44,4 @@ module ActionMailer
autoload :MailHelper
autoload :TestCase
autoload :TestHelper
- autoload :QueuedMessage
end
@@ -1,10 +1,8 @@
require 'mail'
-require 'action_mailer/queued_message'
require 'action_mailer/collector'
require 'active_support/core_ext/string/inflections'
require 'active_support/core_ext/hash/except'
require 'active_support/core_ext/module/anonymous'
-require 'active_support/queueing'
require 'action_mailer/log_subscriber'
module ActionMailer
@@ -361,8 +359,6 @@ module ActionMailer
#
# * <tt>deliveries</tt> - Keeps an array of all the emails sent out through the Action Mailer with
# <tt>delivery_method :test</tt>. Most useful for unit and functional testing.
- #
- # * <tt>queue</> - The queue that will be used to deliver the mail. The queue should expect a job that responds to <tt>run</tt>.
class Base < AbstractController::Base
include DeliveryMethods
abstract!
@@ -389,9 +385,6 @@ class Base < AbstractController::Base
parts_order: [ "text/plain", "text/enriched", "text/html" ]
}.freeze
- class_attribute :queue
- self.queue = ActiveSupport::SynchronousQueue.new
-
class << self
# Register one or more Observers which will be notified when mail is delivered.
def register_observers(*observers)
@@ -483,8 +476,8 @@ def set_payload_for_mail(payload, mail) #:nodoc:
end
def method_missing(method_name, *args)
- if action_methods.include?(method_name.to_s)
- QueuedMessage.new(queue, self, method_name, *args)
+ if respond_to?(method_name)
+ new(method_name, *args).message
else
super
end
@@ -1,37 +0,0 @@
-require 'delegate'
-
-module ActionMailer
- class QueuedMessage < ::Delegator
- attr_reader :queue
-
- def initialize(queue, mailer_class, method_name, *args)
- @queue = queue
- @job = DeliveryJob.new(mailer_class, method_name, args)
- end
-
- def __getobj__
- @job.message
- end
-
- # Queues the message for delivery.
- def deliver
- tap { @queue.push @job }
- end
-
- class DeliveryJob
- def initialize(mailer_class, method_name, args)
- @mailer_class = mailer_class
- @method_name = method_name
- @args = args
- end
-
- def message
- @message ||= @mailer_class.send(:new, @method_name, *@args).message
- end
-
- def run
- message.deliver
- end
- end
- end
-end
@@ -19,8 +19,6 @@ class Railtie < Rails::Railtie # :nodoc:
options.javascripts_dir ||= paths["public/javascripts"].first
options.stylesheets_dir ||= paths["public/stylesheets"].first
- options.queue ||= app.queue
-
# make sure readers methods get compiled
options.asset_host ||= app.config.asset_host
options.relative_url_root ||= app.config.relative_url_root
@@ -11,7 +11,6 @@
require 'minitest/autorun'
require 'action_mailer'
require 'action_mailer/test_case'
-require 'active_support/queueing'
silence_warnings do
# These external dependencies have warnings :/
@@ -3,13 +3,11 @@
require 'set'
require 'action_dispatch'
-require 'active_support/queueing'
require 'active_support/time'
require 'mailers/base_mailer'
require 'mailers/proc_mailer'
require 'mailers/asset_mailer'
-require 'mailers/async_mailer'
class BaseTest < ActiveSupport::TestCase
def teardown
@@ -422,17 +420,6 @@ def teardown
assert_equal(1, BaseMailer.deliveries.length)
end
- test "delivering message asynchronously" do
- AsyncMailer.delivery_method = :test
- AsyncMailer.deliveries.clear
-
- AsyncMailer.welcome.deliver
- assert_equal 0, AsyncMailer.deliveries.length
-
- AsyncMailer.queue.drain
- assert_equal 1, AsyncMailer.deliveries.length
- end
-
test "calling deliver, ActionMailer should yield back to mail to let it call :do_delivery on itself" do
mail = Mail::Message.new
mail.expects(:do_delivery).once
@@ -1,3 +0,0 @@
-class AsyncMailer < BaseMailer
- self.queue = ActiveSupport::TestQueue.new
-end
@@ -10,16 +10,6 @@
require 'action_mailer'
ActionMailer::Base.view_paths = FIXTURE_LOAD_PATH
-class SynchronousQueue < Queue
- def push(job)
- job.run
- end
- alias << push
- alias enq push
-end
-
-ActionMailer::Base.queue = SynchronousQueue.new
-
class AssertSelectTest < ActionController::TestCase
Assertion = ActiveSupport::TestCase::Assertion
@@ -1,105 +0,0 @@
-require 'delegate'
-require 'thread'
-
-module ActiveSupport
- # A Queue that simply inherits from STDLIB's Queue. When this
- # queue is used, Rails automatically starts a job runner in a
- # background thread.
- class Queue < ::Queue
- attr_writer :consumer
-
- def initialize(consumer_options = {})
- super()
- @consumer_options = consumer_options
- end
-
- def consumer
- @consumer ||= ThreadedQueueConsumer.new(self, @consumer_options)
- end
-
- # Drain the queue, running all jobs in a different thread. This method
- # may not be available on production queues.
- def drain
- # run the jobs in a separate thread so assumptions of synchronous
- # jobs are caught in test mode.
- consumer.drain
- end
- end
-
- class SynchronousQueue < Queue
- def push(job)
- super.tap { drain }
- end
- alias << push
- alias enq push
- end
-
- # In test mode, the Rails queue is backed by an Array so that assertions
- # can be made about its contents. The test queue provides a +jobs+
- # method to make assertions about the queue's contents and a +drain+
- # method to drain the queue and run the jobs.
- #
- # Jobs are run in a separate thread to catch mistakes where code
- # assumes that the job is run in the same thread.
- class TestQueue < Queue
- # Get a list of the jobs off this queue. This method may not be
- # available on production queues.
- def jobs
- @que.dup
- end
-
- # Marshal and unmarshal job before pushing it onto the queue. This will
- # raise an exception on any attempts in tests to push jobs that can't (or
- # shouldn't) be marshalled.
- def push(job)
- super Marshal.load(Marshal.dump(job))
- end
- end
-
- # The threaded consumer will run jobs in a background thread in
- # development mode or in a VM where running jobs on a thread in
- # production mode makes sense.
- #
- # When the process exits, the consumer pushes a nil onto the
- # queue and joins the thread, which will ensure that all jobs
- # are executed before the process finally dies.
- class ThreadedQueueConsumer
- attr_accessor :logger
-
- def initialize(queue, options = {})
- @queue = queue
- @logger = options[:logger]
- @fallback_logger = Logger.new($stderr)
- end
-
- def start
- @thread = Thread.new { consume }
- self
- end
-
- def shutdown
- @queue.push nil
- @thread.join
- end
-
- def drain
- @queue.pop.run until @queue.empty?
- end
-
- def consume
- while job = @queue.pop
- run job
- end
- end
-
- def run(job)
- job.run
- rescue Exception => exception
- handle_exception job, exception
- end
-
- def handle_exception(job, exception)
- (logger || @fallback_logger).error "Job Error: #{job.inspect}\n#{exception.message}\n#{exception.backtrace.join("\n")}"
- end
- end
-end
@@ -1,27 +0,0 @@
-require 'abstract_unit'
-require 'active_support/queueing'
-
-class SynchronousQueueTest < ActiveSupport::TestCase
- class Job
- attr_reader :ran
- def run; @ran = true end
- end
-
- class ExceptionRaisingJob
- def run; raise end
- end
-
- def setup
- @queue = ActiveSupport::SynchronousQueue.new
- end
-
- def test_runs_jobs_immediately
- job = Job.new
- @queue.push job
- assert job.ran
-
- assert_raises RuntimeError do
- @queue.push ExceptionRaisingJob.new
- end
- end
-end
Oops, something went wrong.

27 comments on commit f9da785

What constitutes “fully baked”?

Contributor

samgranieri replied Dec 22, 2012

I agree with @Aupajo on this. Was the queue system intended to be the underpinning to resque and sidekiq as rack is to rails or something else entirely? Wy bother with writing a new queue system when resque and sidekiq are the defaults ?
Personally, I'm switching to sidekiq.
/cc @hone @mperham

Owner

rafaelfranca replied Dec 22, 2012

@Aupajo when it reaches a point that we think using it is natural, it still need a lot of work and it is a blocker for the release. We think is better to pospone.

@samgranieri nobody is writing a new queue system, Rails queue is a API can be used with sidekiq, resque, or any other queue system. Think about it as what Active Model is for ORMs.

Member

steveklabnik replied Dec 22, 2012

Both Resque and Sidekiq already had integration with this interface, but there were some issues that were causing problems, and the feature isn't worth holding up the release of Rails 4 for.

Contributor

samgranieri replied Dec 22, 2012

@rafaelfranca I get that it doesn't feel fully baked, but why doesn't it? What was the awkwardness/what were the issues? (Is there an existing thread on this, maybe?)

Owner

rafaelfranca replied Dec 22, 2012

There is no thread anywhere. The decision is based on discussion between the core members and experiments. We fell this is not ready and we still have a lot of doubts with the usage.

It need more experiment to get released.

Owner

jeremy replied Dec 22, 2012

Slipped a knife in to test, and the API was still gooey in the center 😁

Love them secret discussions on open source projects.

Owner

dhh replied Dec 31, 2012

It's how people get shit done without having trolls spoil their appetite for the work.

Owner

fxn replied Dec 31, 2012

You have the code and the rationale, that is open source. While one is always open and thankful for feedback, the project is run by a team, not by a universal house.

I trust you guys to made the right decision on this, but I also empathize with the sentiment expressed above. Perhaps some failing tests would make clear exactly what needs work and define "fully baked"?

We were surprised to see this get cut, although we had heard a little ahead of time. My buddy @phlipper and I had gotten the queue stuff backported to Rails 3.2+ and we're close to releasing the gem for it. https://github.com/probablywrong/rails-queue

Perhaps we can take this discussion and passion over there, get it working for Rails 4 & 3.2+, and work out some of the implementation details/wrinkles outside of the main Rails codebase. That would give us a focused place to discuss specific issues. Interested?

Owner

jeremy replied Jan 1, 2013

@elskid - it's not cut; it's on the jobs branch. By all means continue work! (Releasing a gem named "rails queue" = confusion, though.)

@mikegee - if I could write a test for this API doesn't feel great in my app, I'd be first in line with 100% coverage ;)

Some quick background: I developed a new app that heavily relies on job processing. I used the Rails.queue API from end to end and, ultimately, I felt it's a regression over writing my Resque jobs directly.

  1. Rails.queue[:name] << job API style reads as "push this job onto this specific queue." What most calling code needs is "run this job later." The calling code shouldn't be concerned with where the job is headed or who executes it, so picking a queue by name is a poor distribution of responsibility (not to mention repetitive).
  2. Marshaling complete Job instances sounds great, but it's error-prone. It's too easy to suck in unexpected references to huge objects or procs that can't be marshaled. The API shouldn't attempt to make this marshaling boundary transparent since the wins are so quickly lost as soon as you track down a single marshaling bug. Passing record ids to a job rather than record instances should be the way we think about firing off a job. We're writing a job work order not packaging up a runnable job.
  3. A Rails.database wouldn't make sense and neither does Rails.queue. If anything, that serves in a backend/adapter role to jobs themselves.

On the plus side, as we coped with this issues, we did come up with some nice abstractions for application jobs. Notably, we'll jettison "queueing" from our vocabulary entirely. We want to present an API for submitting and executing jobs. We have more in common with a process scheduler than a task queue.

And, even more promisingly, we saw a couple of key spots where we'd often like to be able to defer work. I'm optimistic that these will have a greater payoff, overall, than a unified job submission/execution API.

  1. A before_commit callback. Russian Doll caching relies on chaining touch: true so changing a child record automatically touches its parent, which touches its grandparent, and so on. That means changing N child objects fires off N parent and grandparent touches when only the last touch was needed. So we'd like to batch up the touches and perform the updates in a before_commit.
  2. Post-request processing: the Merb-style perform_later. After the HTTP response has been sent, perform any work that didn't need to be completed before sending the response. Many tasks are short-running jobs like "send a notification email" and "update the search index" that needn't block a response, yet barely justify a full-on Job class. By performing this work immediately after a response, we sidestep the need to marshal jobs at all, write Job classes, or even run extra worker boxes. We shift those servers back to normal HTTP duties, leaving worker boxes for heavyweight long-running jobs that can get backed up. Much more efficient resource utilization from a total-cluster point of view. This effort hinges on some deep plumbing into our Rack interaction, since the Rack request handling style is not amenable to this currently.

TL;DR we did an experiment with Rails.queue and rejected the API yet learned a lot about what we Really Need. Stay tuned!

@jeremy, thank you for the detailed write-up. Our name was to help ease confusion about what it was that we backported. When we wrote it we thought it as around to stay. heh. Still might have some use as a place to kick around ideas.

We found similar strangeness when testing and using the code as well. Specifically the syntax didn't really "fit"with what we wanted either. And the marshaling broke down for us too.

I think the biggest benefit of the Rails.queue idea was that you always had one and areas that need background execution, no matter the size/scope/expense, you could just fire them. You could then say, do this in dev, do something else in test, and something completely different in production but I have one API to just throw jobs into. That said, a better API to do that would be amazing.

Thanks again!

@jeremy Wow, it sounds like the direction this is heading is great. Thanks for the write-up.

@jeremy if Rails provides the adapter changing the API should not be a big deal. See https://gist.github.com/351550

Contributor

sobrinho replied Jan 1, 2013

For the people who wants an abstraction over the available queues, take a look on https://github.com/fnando/qe

Contributor

erickrause replied Jan 3, 2013

@jeremy Thanks for the write up. "rejected the API yet learned a lot about what we Really Need" I liked the idea of the Rails.queue, but love the before_commit and post request ideas.

Owner

jeremy replied Jan 3, 2013

@sobrinho qe looks great - this is precisely where the Job backend are heading.

Contributor

jrochkind replied Jan 4, 2013

@jeremy thanks SO MUCH for that writeup, explains why it was decided to be not fully baked, with a convincing and reasonable argument, and also explain what you are thinking about future directions.

It is VERY useful for us Rails-using developers to have this, thanks for realizing such and providing.

(Also, thanks very much to Rails core for not releasing an API as part of Rails when you realize the design is not right, we indeed count on your active judgement of these things to keep Rails great. Throwing in half-baked things will help nobody.)

Contributor

kenn replied Jan 5, 2013

@jeremy thanks for the write up, that reasoning is really convincing and enlightening.

Post-request processing is clever - I've been using EM.next_tick when I use an eventmachine-based web server like thin, which works great. No marshaling is necessary, and binding objects in the local scope are available - which simplifies things tremendously. I'd be very happy if we'd get the utility of EM.next_tick without depending on EM.

Anonymous jobs FTW!

I'm looking forward to submitting job work orders. If you combine this with the power of something like hstore to store arbitrary variables for later when the job executes... that's what I'm talking about.

The Russian Doll :touch issue is something which has been giving me troubles today - look forward to something coming of this discussion.

Owner

dhh replied Aug 5, 2013

@dhh Hi David - it's mentioned in Jeremy's comment above f9da785#commitcomment-2370489 I was also referred to this post discussion from another issue recoded here: #8759

I don't think there is a specific link between the two challenges, but Jeremy seems to have come up against the :touch challenges at some point in his adventure here.

Please sign in to comment.