-
Notifications
You must be signed in to change notification settings - Fork 21.8k
Initial implementation of ActiveJob AsyncAdapter. #21257
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
NOTE: I think my benchmarks are uninformative. Since I used the test helpers to setup ActiveJob, it looks like both adapters are performing synchronously, not asynchronously. I will gather better benchmarks as soon as I figure out how to setup the test properly. |
The benchmark script and associated data have been updated. They now represent asynchronous behavior. Performance is roughly the same for |
Thank you for the pull request but I don't think we should maintain a queue On Sun, Aug 16, 2015, 15:46 jdantonio notifications@github.com wrote:
|
@dhh @tenderlove @matthewd @senny @chancancode what are your thoughts about a default async adapter in Rails itself? |
I actually would like to see a default async queue for ActiveJob, mostly for testing. No, you shouldn't use that in production unless you don't care about losing jobs if a process crashes, but it would be nice for testing, so you wouldn't have to install Sucker Punch. |
If that is the goal 👍 too. It should be explicit that it should not be used on production. |
Agree. Or only used in production for things with no criticality. On Sun, Aug 16, 2015 at 5:21 PM, Rafael Mendonça França <
|
I'll begin working on the full feature set tomorrow (Monday). Scheduled tasks ( |
Thanks! Yeah, would be really great to have scheduled tasks work for On Sun, Aug 16, 2015 at 11:23 PM, jdantonio notifications@github.com
|
@dhh @rafaelfranca I'm sorry it took me a few days to get back to this, but it's ready now. I've added job scheduling and support for custom queues. I've also updated the documentation to briefly explain why this adapter is for dev/test and not prod. |
Really nice. I do have 2 suggestions:
|
@carllerche I'll start work on both of those changes this evening. |
@cristianbica This update implements both of your suggestions. Now that |
In fact we might not need to expose that |
}.freeze | ||
|
||
QUEUES = ThreadSafe::Cache.new do |hash, queue_name| | ||
hash[queue_name] = case queue_name |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Lets do:
queue_name = queue_name.to_sym
hash[queue_name] = \
if queue_name == :default
ActiveJob::AsyncJob.default_executor
else
ActiveJob::AsyncJob.create_executor
end
@kaspth I was definitely overthinking things. Running the tests requires that the job runner perform synchronously. The original version was a proof-of-concept that didn't support queues or scheduled tasks, so I simply injected an ImmediateExecutor when running the tests. When I started adding new features I lost sight of the original intent and made things more complicated than necessary. This update has the same features yet is simpler:
This update also includes a set of unit tests for the |
class QueueCreationError < ArgumentError; end | ||
|
||
class << self | ||
# Force all jobs to run synchronously when testing the activejob gem. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's Active Job 😁
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also why do we need to run async adapter jobs synchronously in our own tests? Won't that hurt ourselves down the line?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My apologies regarding the gem name. I've fixed that.
I must have misunderstood what was happening in the test helpers. Some of the test helpers set a "test" backend or mode. When I run the tests without a test mode one of the serialization tests fails. I'll try to figure out what's happening there.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correction--all the adapter tests fail when there is no test mode.
test/adapters/delayed_job.rb:
Delayed::Worker.delay_jobs = false
Delayed::Worker.backend = :test
test/adapters/qu.rb:
require 'qu-immediate'
test/adapters/resque.rb:
Resque.inline = true
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No worries about the name 😁
I'm not sure I follow you. Perhaps we're talking about different things?
My confusion is how the async adapter implements a test mode by not doing what it says on the tin. I can understand if we need the adapter to run synchronously for its own tests, but shouldn't we also have integration tests that mimic how an actual user tests with the adapter?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh. I think I see what you are saying. If I follow, you are suggesting another set of tests that actually post background jobs then verify that the jobs actually run in the background. Is that correct? That's not a problem. I'll add a set of tests for that.
The latest update is a rebase against the latest Rails master and also incorporates the other suggestions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Exactly that 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kaspth I apologize if I'm missing something, but it appears that the file test/integration/queuing_test.rb
currently tests the asynchronous behavior of all adapters. It posts jobs with perform_later
then uses timers (such as wait_for_jobs_to_finish_for(2.seconds)
) to verify that the jobs post. It looks like the adapter tests put all the adapters in synchronous mode, the existing integration tests put the adapters in asynchronous mode, and the Rakefile controls the environment appropriately for each set of tests. However, I didn't notice this earlier so I have not setup the necessary test helper for making the integration tests work with the Async Job. The integration tests (which do not run with rake test
) are failing. I'll fix that.
This just keeps getting better 👏 |
@kaspth The test cases in
All of the tests unit tests ( |
require 'jobs/queue_as_job' | ||
|
||
class AsyncJobTest < ActiveSupport::TestCase | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
✂️ this line
# Raises +QueueCreationError+ when the queue already exists. | ||
def create_queue(name, thread_pool) | ||
raise QueueCreationError.new('queue already exists') if QUEUES.key? name | ||
# possible race condition here but the use case is very narrow |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the use case is so very narrow should we really add this method? Generally we'd like to keep as closed off a public interface as possible.
@robin850 can you check the documentation? @cristianbica what do you think about this? I'm not too familiar with Active Job's tests, so how are these looking? |
# pool directly then call the +create_queue+ method passing the thread | ||
# pool as the second parameter: | ||
# | ||
# thread_pool = Concurrent::FixedThreadPool.new(10, max_queue: 100, fallback_polcy: :caller_runs) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's a missing "i" in fallback_policy
.
Great job @jdantonio! This is looking good! 👏 |
def create_queue(name, thread_pool) | ||
raise QueueCreationError.new('queue already exists') if QUEUES.key? name | ||
# possible race condition here but the use case is very narrow | ||
QUEUES[name] = thread_pool |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not synchronize this method?
Initial implementation of ActiveJob AsyncAdapter.
Thanks @jdantonio, this is great 👍 |
@kaspth Thank you very much for your feedback! This PR is much better because of your help. |
❤️ |
Now that
activesupport
has a runtime dependency onconcurrent-ruby
, we can begin taking advantage of those tools in more ways. This PR creates a simple asynchronous ActiveJob adapter that posts jobs to a concurrent-ruby thread pool. Within the context of ActiveJob it provides functionality comparable to sucker_punch. Rails 5 users will now be able to create simple asynchronous jobs without installing additional gems simply by setting the new adapter:A simple benchmark script which compares enqueue performance vs. sucker_punch can be found here. Performance is comparable on both Ruby 2.2.2 and JRuby 9000.
If this PR is accepted I can add more features such as job prioritization and per-queue thread pools.