Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ActiveJob: queue_adapter can be inherited #16992

Merged
merged 1 commit into from Mar 12, 2015

Conversation

@tamird
Copy link
Contributor

tamird commented Sep 20, 2014

@tamird tamird force-pushed the square:configurable-job-queue-adapter branch Sep 24, 2014
@tamird tamird force-pushed the square:configurable-job-queue-adapter branch 2 times, most recently Sep 24, 2014
@tamird tamird changed the title allow overriding sub job queue_adapter without affecting the base job or... AJ::Base.queue_adapter improvements Sep 24, 2014
@tamird tamird force-pushed the square:configurable-job-queue-adapter branch Sep 24, 2014
@lukaszx0
lukaszx0 reviewed Sep 24, 2014
View changes
activejob/lib/active_job/queue_adapters/test_adapter.rb Outdated
performed_jobs << {job: job.class, args: job.arguments, queue: job.queue_name}
job.perform_now
else
enqueued_jobs << {job: job.class, args: job.arguments, queue: job.queue_name}

This comment has been minimized.

Copy link
@lukaszx0

lukaszx0 Sep 24, 2014

Member

We have 4 lines like this. Maybe let's return it from a method like job_hash(job) ?

This comment has been minimized.

Copy link
@lukaszx0

lukaszx0 Sep 24, 2014

Member

Actually even more natural would be job.to_hash

This comment has been minimized.

Copy link
@tamird

tamird Sep 24, 2014

Author Contributor

done

@lukaszx0
lukaszx0 reviewed Sep 24, 2014
View changes
activejob/lib/active_job/queue_adapters/test_adapter.rb Outdated
performed_jobs << {job: job.class, args: job.arguments, queue: job.queue_name, at: timestamp}
job.perform_now
else
enqueued_jobs << {job: job.class, args: job.arguments, queue: job.queue_name, at: timestamp}

This comment has been minimized.

Copy link
@lukaszx0

lukaszx0 Sep 24, 2014

Member

job_hash(job, at: timestamp)

@lukaszx0 lukaszx0 added the activejob label Sep 24, 2014
@tamird tamird force-pushed the square:configurable-job-queue-adapter branch Sep 24, 2014
@cristianbica
Copy link
Member

cristianbica commented Sep 24, 2014

The test adapter was made like this so we can run tests in parallel. @seuros you worked on this
Anyway if you are going to do this make a separate PR

@cristianbica
cristianbica reviewed Sep 24, 2014
View changes
activejob/lib/active_job/queue_adapter.rb Outdated
require 'active_support/core_ext/string/inflections'

module ActiveJob
module QueueAdapter
extend ActiveSupport::Concern

included do
class_attribute :_queue_adapter, instance_accessor: false, instance_predicate: false

This comment has been minimized.

Copy link
@cristianbica

cristianbica Sep 24, 2014

Member

Please rename _queue_adapter to something else _queue_adapter_class or even queue_adapter_class to be more clear

This comment has been minimized.

Copy link
@tamird

tamird Sep 24, 2014

Author Contributor

It's not always a class, though. It may be a callable. I'll change the name to reflect that.

@cristianbica
cristianbica reviewed Sep 24, 2014
View changes
activejob/lib/active_job/queue_adapter.rb Outdated
module ClassMethods
mattr_reader(:queue_adapter) { ActiveJob::QueueAdapters::InlineAdapter }
def queue_adapter
if _queue_adapter.respond_to?(:call)

This comment has been minimized.

Copy link
@cristianbica

cristianbica Sep 24, 2014

Member

Can you provide a use case where you need to determine the adapter in a proc? We should implement features when there are clear use cases for them (I think DHH's rule is at least 2)

This comment has been minimized.

Copy link
@tamird

tamird Sep 24, 2014

Author Contributor

Sure, I needed this today in production actually. In the case of switching from one job queue to another, it may be desirable to enqueue a percentage of jobs to say, sidekiq, while the rest go to resque. We're running a similar thing right now where the percentage is stored in a redis key, allowing us to ramp up dynamically, and roll back if problems are detected (without a deploy).

@cristianbica
Copy link
Member

cristianbica commented Sep 24, 2014

Here's the PR when we switch the test adapter to instance methods #16724 (diff)

@tamird tamird force-pushed the square:configurable-job-queue-adapter branch Sep 24, 2014
@tamird tamird mentioned this pull request Sep 24, 2014
@seuros
seuros reviewed Sep 24, 2014
View changes
activejob/test/cases/test_helper_test.rb Outdated
@@ -7,6 +7,7 @@

class EnqueuedJobsTest < ActiveJob::TestCase
setup { queue_adapter.perform_enqueued_at_jobs = true }
teardown { queue_adapter.perform_enqueued_at_jobs = false }

This comment has been minimized.

Copy link
@seuros

seuros Sep 24, 2014

Member

We don't need these teardowns. The test adapter sould to allow parallel test instantiated for every test

This comment has been minimized.

Copy link
@tamird

tamird Sep 24, 2014

Author Contributor

Not anymore, it's a singleton now. See existing discussion in this PR.

@tamird tamird force-pushed the square:configurable-job-queue-adapter branch 4 times, most recently to 6f851ec Sep 24, 2014
@cristianbica
Copy link
Member

cristianbica commented Sep 24, 2014

Ok. So ... as discussed the ability to pass a proc to queue_adapter is still under discussion so I suggest to leave this PR with the initial scope of being able to override sub-jobs adapters and start a new PR with the proc thing.
On the other hand we need to document the current change so we need to add documentation to queue_adapter= and queue_adapter. What would be nicer is if you have some time to update the AJ guide :)

@cristianbica
Copy link
Member

cristianbica commented Sep 24, 2014

Also you need to rebase before we're ready to merge

@cristianbica
Copy link
Member

cristianbica commented Sep 24, 2014

I meant to rebase so you can squash your commits

@seuros
Copy link
Member

seuros commented Sep 24, 2014

Then you should split them into multiple PRs.
I don't like the singleton conversion. we had it like that and changed it to instance . cc @jeremy

@lukaszx0
Copy link
Member

lukaszx0 commented Sep 24, 2014

No need to squash or split it into multiple PRs.

Looks good to me 👍

@jeremy
Copy link
Member

jeremy commented Sep 25, 2014

I'd keep TestAdapter as a class with a single instance rather than switching to using a class object as the adapter. Nice cleanup of the adapter internals in either case.

@cristianbica
Copy link
Member

cristianbica commented Oct 8, 2014

@tamir this is the same as #16977 so we can close #16977, right?

@tamir
Copy link

tamir commented Oct 8, 2014

I was erroneously ccd. the mail should have gone to @tamird i think.

On 08 Oct 2014, at 09:02, Cristian Bica notifications@github.com wrote:

@tamir this is the same as #16977 so we can close #16977, right?


Reply to this email directly or view it on GitHub.

@tamird tamird force-pushed the square:configurable-job-queue-adapter branch from 6f851ec Feb 23, 2015
test_adapter = ActiveJob::QueueAdapters::TestAdapter.new

@old_queue_adapters = (ActiveJob::Base.subclasses << ActiveJob::Base).select do |klass|
klass.respond_to?(:queue_adapter)

This comment has been minimized.

Copy link
@jeremy

jeremy Mar 12, 2015

Member

Every ActiveJob::Base subclass will respond to :queue_adapter since it's a singleton method on Base

This comment has been minimized.

Copy link
@tamird

tamird Mar 12, 2015

Author Contributor

derp, you're right. fixed

@tamird tamird force-pushed the square:configurable-job-queue-adapter branch from 9515e8a Mar 12, 2015
@jeremy
jeremy reviewed Mar 12, 2015
View changes
activejob/lib/active_job/test_helper.rb Outdated
[klass, klass.queue_adapter].tap do
klass.queue_adapter = test_adapter
end
end

This comment has been minimized.

Copy link
@jeremy

jeremy Mar 12, 2015

Member

Since class attributes no longer inherit from superclass once they're written, assigning #queue_adapter on classes the didn't have it explicitly set will leak side effects between test cases (since changing a superclass attr won't be inherited by its subclass afterward)

This comment has been minimized.

Copy link
@tamird

tamird Mar 12, 2015

Author Contributor

great catch, fixed

@tamird tamird force-pushed the square:configurable-job-queue-adapter branch 3 times, most recently from fabae1a Mar 12, 2015
@jeremy
jeremy reviewed Mar 12, 2015
View changes
activejob/lib/active_job/test_helper.rb Outdated
# only override explicitly set adapters, a quirk of `class_attribute`
klass.singleton_class.public_instance_methods(false).include?(:_queue_adapter)
end.map do |klass|
[klass, klass._queue_adapter].tap do

This comment has been minimized.

Copy link
@jeremy

jeremy Mar 12, 2015

Member

Prefer to use the public-API reader?

This comment has been minimized.

Copy link
@tamird

tamird Mar 12, 2015

Author Contributor

fixed

@jeremy
jeremy reviewed Mar 12, 2015
View changes
activejob/lib/active_job/test_helper.rb Outdated
@@ -16,7 +26,9 @@ def before_setup

def after_teardown
super
ActiveJob::Base.queue_adapter = @old_queue_adapter
@old_queue_adapters.each do |(klass, adapter)|
klass._queue_adapter = adapter

This comment has been minimized.

Copy link
@jeremy

jeremy Mar 12, 2015

Member

Can use the public writer method too

This comment has been minimized.

Copy link
@tamird

tamird Mar 12, 2015

Author Contributor

derp, fixed

This allows different `queue_adapters` to be used in each `ActiveJob`
class heirarchy. Previously, all subclasses used a single global queue
adapter.
@tamird tamird force-pushed the square:configurable-job-queue-adapter branch to 1f8558f Mar 12, 2015
jeremy added a commit that referenced this pull request Mar 12, 2015
ActiveJob: queue_adapter can be inherited
@jeremy jeremy merged commit 202c68c into rails:master Mar 12, 2015
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@tamird tamird deleted the square:configurable-job-queue-adapter branch Mar 12, 2015
@cristianbica
Copy link
Member

cristianbica commented Mar 12, 2015

❤️ took a few months to get this merged :)

@trungpham
Copy link

trungpham commented Mar 12, 2015

👍

@rebyn
Copy link
Contributor

rebyn commented Apr 13, 2015

I'm not seeing this feature being mentioned in the Basics guide. Or we have an Advanced guide somewhere that I don't know :)?

I think this is cool and deserves a mention. Should I submit a PR or has anyone worked on this already?

@rafaelfranca rafaelfranca modified the milestones: 5.0.0 [temp], 5.0.0 Dec 30, 2015
@benoitongit
Copy link

benoitongit commented Mar 7, 2016

👍
Really cool, thanks! Would be great to add it in the basic guide, it took some time to know it was possible.

@jrochkind
Copy link
Contributor

jrochkind commented Jul 21, 2017

Is code merged in such that it's possible to have different queue adapters for different jobs? (and if so beginning in what Rails version)

I got here following a trail of links between issues and PRs here and in ActiveJob. And the disussion and statuses here make me think this is now possible. But I have no idea how to do it!

If I'm on the right track, could anyone post an example here? If I get an example, I will try to play with it and understand it and do a doc PR demo'ing it in the right place. (If it's already in the docs but I just haven't found it, would very much appreciate if you wanted to point me there!).

@jrochkind
Copy link
Contributor

jrochkind commented Jul 21, 2017

Nevermind, just found it in the guide! "You can also configure your backend on a per job basis." Okay! Great!

@steveburkett
Copy link

steveburkett commented Oct 4, 2017

does anyone have a "patch" i can use in my rails 4.2 app to get this to work?

sharshenov added a commit to veeqo/advanced-sneakers-activejob that referenced this pull request Apr 6, 2020
Rails 5+ allows to set queue adapter per job class
rails/rails#16992
sharshenov added a commit to veeqo/advanced-sneakers-activejob that referenced this pull request Apr 6, 2020
Rails 5+ allows to set queue adapter per job class
rails/rails#16992
sharshenov added a commit to veeqo/advanced-sneakers-activejob that referenced this pull request Apr 6, 2020
Rails 5+ allows to set queue adapter per job class
rails/rails#16992
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.