New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Initial implementation of ActiveJob AsyncAdapter. #21257

Merged
merged 1 commit into from Aug 25, 2015

Conversation

Projects
None yet
8 participants
@jdantonio
Contributor

jdantonio commented Aug 16, 2015

Now that activesupport has a runtime dependency on concurrent-ruby, we can begin taking advantage of those tools in more ways. This PR creates a simple asynchronous ActiveJob adapter that posts jobs to a concurrent-ruby thread pool. Within the context of ActiveJob it provides functionality comparable to sucker_punch. Rails 5 users will now be able to create simple asynchronous jobs without installing additional gems simply by setting the new adapter:

# config/application.rb
module YourApp
  class Application < Rails::Application
    config.active_job.queue_adapter = :async
  end
end

A simple benchmark script which compares enqueue performance vs. sucker_punch can be found here. Performance is comparable on both Ruby 2.2.2 and JRuby 9000.

If this PR is accepted I can add more features such as job prioritization and per-queue thread pools.

@jdantonio

This comment has been minimized.

Show comment
Hide comment
@jdantonio

jdantonio Aug 16, 2015

Contributor

NOTE: I think my benchmarks are uninformative. Since I used the test helpers to setup ActiveJob, it looks like both adapters are performing synchronously, not asynchronously. I will gather better benchmarks as soon as I figure out how to setup the test properly.

Contributor

jdantonio commented Aug 16, 2015

NOTE: I think my benchmarks are uninformative. Since I used the test helpers to setup ActiveJob, it looks like both adapters are performing synchronously, not asynchronously. I will gather better benchmarks as soon as I figure out how to setup the test properly.

@jdantonio

This comment has been minimized.

Show comment
Hide comment
@jdantonio

jdantonio Aug 16, 2015

Contributor

The benchmark script and associated data have been updated. They now represent asynchronous behavior. Performance is roughly the same for AsyncAdapter and SuckerPunchAdapter.

Contributor

jdantonio commented Aug 16, 2015

The benchmark script and associated data have been updated. They now represent asynchronous behavior. Performance is roughly the same for AsyncAdapter and SuckerPunchAdapter.

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Aug 16, 2015

Member

Thank you for the pull request but I don't think we should maintain a queue
system inside Rails. There are a lot of good implementations out there that
I think we just end up with a poor implementation inside the framework, to
not say the maintenance overhead that it will give us.

On Sun, Aug 16, 2015, 15:46 jdantonio notifications@github.com wrote:

The benchmark script and associated data have been updated. They now
represent asynchronous behavior. Performance is roughly the same for
AsyncAdapter and SuckerPunchAdapter.


Reply to this email directly or view it on GitHub
#21257 (comment).

Member

rafaelfranca commented Aug 16, 2015

Thank you for the pull request but I don't think we should maintain a queue
system inside Rails. There are a lot of good implementations out there that
I think we just end up with a poor implementation inside the framework, to
not say the maintenance overhead that it will give us.

On Sun, Aug 16, 2015, 15:46 jdantonio notifications@github.com wrote:

The benchmark script and associated data have been updated. They now
represent asynchronous behavior. Performance is roughly the same for
AsyncAdapter and SuckerPunchAdapter.


Reply to this email directly or view it on GitHub
#21257 (comment).

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Aug 16, 2015

Member

@dhh @tenderlove @matthewd @senny @chancancode what are your thoughts about a default async adapter in Rails itself?

Member

rafaelfranca commented Aug 16, 2015

@dhh @tenderlove @matthewd @senny @chancancode what are your thoughts about a default async adapter in Rails itself?

@dhh

This comment has been minimized.

Show comment
Hide comment
@dhh

dhh Aug 16, 2015

Member

I actually would like to see a default async queue for ActiveJob, mostly for testing. No, you shouldn't use that in production unless you don't care about losing jobs if a process crashes, but it would be nice for testing, so you wouldn't have to install Sucker Punch.

Member

dhh commented Aug 16, 2015

I actually would like to see a default async queue for ActiveJob, mostly for testing. No, you shouldn't use that in production unless you don't care about losing jobs if a process crashes, but it would be nice for testing, so you wouldn't have to install Sucker Punch.

@rafaelfranca

This comment has been minimized.

Show comment
Hide comment
@rafaelfranca

rafaelfranca Aug 16, 2015

Member

If that is the goal 👍 too. It should be explicit that it should not be used on production.

Member

rafaelfranca commented Aug 16, 2015

If that is the goal 👍 too. It should be explicit that it should not be used on production.

@dhh

This comment has been minimized.

Show comment
Hide comment
@dhh

dhh Aug 16, 2015

Member

Agree. Or only used in production for things with no criticality.

On Sun, Aug 16, 2015 at 5:21 PM, Rafael Mendonça França <
notifications@github.com> wrote:

If that is the goal [image: 👍] too. It should be explicit that it
should not be used on production.


Reply to this email directly or view it on GitHub
#21257 (comment).

Member

dhh commented Aug 16, 2015

Agree. Or only used in production for things with no criticality.

On Sun, Aug 16, 2015 at 5:21 PM, Rafael Mendonça França <
notifications@github.com> wrote:

If that is the goal [image: 👍] too. It should be explicit that it
should not be used on production.


Reply to this email directly or view it on GitHub
#21257 (comment).

@jdantonio

This comment has been minimized.

Show comment
Hide comment
@jdantonio

jdantonio Aug 17, 2015

Contributor

I'll begin working on the full feature set tomorrow (Monday). Scheduled tasks (#set(wait: 1.week)) won't survive a restart, but that should be fine for testing. I'll update the guide to be abundantly clear that the :async adapter is for testing only. I'll update this PR when it is feature-complete.

Contributor

jdantonio commented Aug 17, 2015

I'll begin working on the full feature set tomorrow (Monday). Scheduled tasks (#set(wait: 1.week)) won't survive a restart, but that should be fine for testing. I'll update the guide to be abundantly clear that the :async adapter is for testing only. I'll update this PR when it is feature-complete.

@dhh

This comment has been minimized.

Show comment
Hide comment
@dhh

dhh Aug 17, 2015

Member

Thanks! Yeah, would be really great to have scheduled tasks work for
testing. Sucker Punch doesn't support those. I had that issue with some
tasks that had a 15s delay that I couldn't test.

On Sun, Aug 16, 2015 at 11:23 PM, jdantonio notifications@github.com
wrote:

I'll begin working on the full feature set tomorrow (Monday). Scheduled
tasks (#set(wait: 1.week)) won't survive a restart, but that should be
fine for testing. I'll update the guide to be abundantly clear that the
:async adapter is for testing only. I'll update this PR when it is
feature-complete.


Reply to this email directly or view it on GitHub
#21257 (comment).

Member

dhh commented Aug 17, 2015

Thanks! Yeah, would be really great to have scheduled tasks work for
testing. Sucker Punch doesn't support those. I had that issue with some
tasks that had a 15s delay that I couldn't test.

On Sun, Aug 16, 2015 at 11:23 PM, jdantonio notifications@github.com
wrote:

I'll begin working on the full feature set tomorrow (Monday). Scheduled
tasks (#set(wait: 1.week)) won't survive a restart, but that should be
fine for testing. I'll update the guide to be abundantly clear that the
:async adapter is for testing only. I'll update this PR when it is
feature-complete.


Reply to this email directly or view it on GitHub
#21257 (comment).

@jdantonio

This comment has been minimized.

Show comment
Hide comment
@jdantonio

jdantonio Aug 20, 2015

Contributor

@dhh @rafaelfranca I'm sorry it took me a few days to get back to this, but it's ready now. I've added job scheduling and support for custom queues. I've also updated the documentation to briefly explain why this adapter is for dev/test and not prod.

Contributor

jdantonio commented Aug 20, 2015

@dhh @rafaelfranca I'm sorry it took me a few days to get back to this, but it's ready now. I've added job scheduling and support for custom queues. I've also updated the documentation to briefly explain why this adapter is for dev/test and not prod.

@cristianbica

This comment has been minimized.

Show comment
Hide comment
@cristianbica

cristianbica Aug 20, 2015

Member

Really nice. I do have 2 suggestions:

  • Follow the model of all the adapters: have a different class (ex: ActiveJob::AsyncJob) that handles the queueing, execution, queue creating and the configurations (ex: executor) and the adapter should just call enqueue / enqueue_at on the AsyncJob class.
  • it seems a bit complicated to have to create the queues upfront. I would rather see the async class create the queue if not exists
Member

cristianbica commented Aug 20, 2015

Really nice. I do have 2 suggestions:

  • Follow the model of all the adapters: have a different class (ex: ActiveJob::AsyncJob) that handles the queueing, execution, queue creating and the configurations (ex: executor) and the adapter should just call enqueue / enqueue_at on the AsyncJob class.
  • it seems a bit complicated to have to create the queues upfront. I would rather see the async class create the queue if not exists
@jdantonio

This comment has been minimized.

Show comment
Hide comment
@jdantonio

jdantonio Aug 20, 2015

Contributor

@carllerche I'll start work on both of those changes this evening.

Contributor

jdantonio commented Aug 20, 2015

@carllerche I'll start work on both of those changes this evening.

@jdantonio

This comment has been minimized.

Show comment
Hide comment
@jdantonio

jdantonio Aug 20, 2015

Contributor

@cristianbica This update implements both of your suggestions. Now that ActiveJob::AsyncJob is a separate class I would like to add a set of tests that test it directly, but I'd like to get your feedback on this update before I write those. (The adapter tests still pass.)

Contributor

jdantonio commented Aug 20, 2015

@cristianbica This update implements both of your suggestions. Now that ActiveJob::AsyncJob is a separate class I would like to add a set of tests that test it directly, but I'd like to get your feedback on this update before I write those. (The adapter tests still pass.)

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Aug 20, 2015

Member

executor might be close to the concurrent-ruby terminology, but it feels very removed from Active Job. It would be great to find a term closer to home.

In fact we might not need to expose that executor information at all. As the adapter seems to have a lot of configuration for something that's mostly for testing. What need is there to change the pool size in testing as long as we're using a minimum of 2?

Member

kaspth commented Aug 20, 2015

executor might be close to the concurrent-ruby terminology, but it feels very removed from Active Job. It would be great to find a term closer to home.

In fact we might not need to expose that executor information at all. As the adapter seems to have a lot of configuration for something that's mostly for testing. What need is there to change the pool size in testing as long as we're using a minimum of 2?

@kaspth

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@kaspth

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@kaspth

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@kaspth

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@jdantonio

This comment has been minimized.

Show comment
Hide comment
@jdantonio

jdantonio Aug 21, 2015

Contributor

@kaspth I was definitely overthinking things. Running the tests requires that the job runner perform synchronously. The original version was a proof-of-concept that didn't support queues or scheduled tasks, so I simply injected an ImmediateExecutor when running the tests. When I started adding new features I lost sight of the original intent and made things more complicated than necessary.

This update has the same features yet is simpler:

  • ActiveJob::AsyncJob has a set_test_mode! class method that the test helper calls. It is undocumented and irreversible. I expect that it will only be called from the test helper in this repo.
  • All queues, including the default queue, are created with the same configuration.
  • Queues are automatically created the first time a job is post to the queue.
  • Although it's a narrow use case, queues can still be manually created and configured. The only option in this case is to pass a manually created thread pool as an argument. I think we can assume that anyone choosing to manually configure a queue is also capable of manually configuring the thread pool.

This update also includes a set of unit tests for the AsyncJob class.

Contributor

jdantonio commented Aug 21, 2015

@kaspth I was definitely overthinking things. Running the tests requires that the job runner perform synchronously. The original version was a proof-of-concept that didn't support queues or scheduled tasks, so I simply injected an ImmediateExecutor when running the tests. When I started adding new features I lost sight of the original intent and made things more complicated than necessary.

This update has the same features yet is simpler:

  • ActiveJob::AsyncJob has a set_test_mode! class method that the test helper calls. It is undocumented and irreversible. I expect that it will only be called from the test helper in this repo.
  • All queues, including the default queue, are created with the same configuration.
  • Queues are automatically created the first time a job is post to the queue.
  • Although it's a narrow use case, queues can still be manually created and configured. The only option in this case is to pass a manually created thread pool as an argument. I think we can assume that anyone choosing to manually configure a queue is also capable of manually configuring the thread pool.

This update also includes a set of unit tests for the AsyncJob class.

@kaspth

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@kaspth

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@kaspth

View changes

Show outdated Hide outdated activejob/test/cases/async_job_test.rb
@kaspth

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Aug 23, 2015

Member

This just keeps getting better 👏

Member

kaspth commented Aug 23, 2015

This just keeps getting better 👏

@jdantonio

This comment has been minimized.

Show comment
Hide comment
@jdantonio

jdantonio Aug 24, 2015

Contributor

@kaspth The test cases in test/integration/queuing_test.rb are precisely the type of test you suggested. When I created an appropriate test helper and ran those tests I discovered a localization bug. The latest version has the following changes:

  • Fix the aforementioned locale (serialization) bug
  • Follows the pattern of the other adapters and serializes the job within the adapter
  • Follows the pattern of the other adapters and explicitly passes the queue name from the adapter
  • Adds a test job and AsyncJob unit test case for queue_as
  • Adds a set_normal_mode! method for consistent unit test setup/teardown.

All of the tests unit tests (rake test:async) and integration tests (rake test:integration:async) pass.

Contributor

jdantonio commented Aug 24, 2015

@kaspth The test cases in test/integration/queuing_test.rb are precisely the type of test you suggested. When I created an appropriate test helper and ran those tests I discovered a localization bug. The latest version has the following changes:

  • Fix the aforementioned locale (serialization) bug
  • Follows the pattern of the other adapters and serializes the job within the adapter
  • Follows the pattern of the other adapters and explicitly passes the queue name from the adapter
  • Adds a test job and AsyncJob unit test case for queue_as
  • Adds a set_normal_mode! method for consistent unit test setup/teardown.

All of the tests unit tests (rake test:async) and integration tests (rake test:integration:async) pass.

@kaspth

View changes

Show outdated Hide outdated activejob/test/cases/async_job_test.rb
@kaspth

View changes

Show outdated Hide outdated activejob/lib/active_job/queue_adapters/async_adapter.rb
@kaspth

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@kaspth

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@kaspth

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@kaspth

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@kaspth

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@kaspth

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Aug 24, 2015

Member

@robin850 can you check the documentation?

@cristianbica what do you think about this? I'm not too familiar with Active Job's tests, so how are these looking?

Member

kaspth commented Aug 24, 2015

@robin850 can you check the documentation?

@cristianbica what do you think about this? I'm not too familiar with Active Job's tests, so how are these looking?

@robin850

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@robin850

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@robin850

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@robin850

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@robin850

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@robin850

View changes

Show outdated Hide outdated activejob/lib/active_job/queue_adapters.rb
@robin850

View changes

Show outdated Hide outdated activejob/lib/active_job/queue_adapters.rb
@robin850

View changes

Show outdated Hide outdated activejob/lib/active_job/queue_adapters/async_adapter.rb
@robin850

View changes

Show outdated Hide outdated activejob/test/cases/async_job_test.rb
@robin850

This comment has been minimized.

Show comment
Hide comment
@robin850

robin850 Aug 24, 2015

Member

Great job @jdantonio! This is looking good! 👏

Member

robin850 commented Aug 24, 2015

Great job @jdantonio! This is looking good! 👏

@ianks

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@ianks

View changes

Show outdated Hide outdated activejob/lib/active_job/async_job.rb
@jdantonio

This comment has been minimized.

Show comment
Hide comment
@jdantonio

jdantonio Aug 25, 2015

Contributor

@kaspth @robin850 Updated based on latest feedback and rebased against latest master.

Contributor

jdantonio commented Aug 25, 2015

@kaspth @robin850 Updated based on latest feedback and rebased against latest master.

kaspth added a commit that referenced this pull request Aug 25, 2015

Merge pull request #21257 from jdantonio/async-job
Initial implementation of ActiveJob AsyncAdapter.

@kaspth kaspth merged commit c5a88e5 into rails:master Aug 25, 2015

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Aug 25, 2015

Member

Thanks @jdantonio, this is great 👍

Member

kaspth commented Aug 25, 2015

Thanks @jdantonio, this is great 👍

@jdantonio

This comment has been minimized.

Show comment
Hide comment
@jdantonio

jdantonio Aug 25, 2015

Contributor

@kaspth Thank you very much for your feedback! This PR is much better because of your help.

Contributor

jdantonio commented Aug 25, 2015

@kaspth Thank you very much for your feedback! This PR is much better because of your help.

@kaspth

This comment has been minimized.

Show comment
Hide comment
@kaspth

kaspth Aug 25, 2015

Member

❤️

Member

kaspth commented Aug 25, 2015

❤️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment