-
Notifications
You must be signed in to change notification settings - Fork 43
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Implement background worker for big uploaded files
- Loading branch information
Showing
15 changed files
with
103 additions
and
8 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,22 @@ | ||
# Base class from which other Resque workers inherit. It makes sure that all DB | ||
# connections are active (they could have timed out). | ||
# | ||
# See: | ||
# http://axonflux.com/resque-to-the-rescue-but-a-gotcha-dont-forget | ||
# | ||
|
||
class BaseWorker | ||
# All our workers will make use of the resque-status plugin | ||
include Resque::Plugins::Status | ||
|
||
# Main method for this class, it re-connects any stale AR connections and | ||
# delegates to the #perform_delegate() method to perform the actual work. | ||
def perform(*args) | ||
ActiveRecord::Base.clear_active_connections! | ||
perform_delegate(*args) | ||
end | ||
|
||
# Implementing workers will override this method. | ||
def perform_delegate(*args) | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,38 @@ | ||
# Resque background worker to process files uploaded via the Upload Manager. | ||
# | ||
# This worker requires the following options: | ||
# plugin: Upload Plugin that needs to process the file, | ||
# file: Full path to the file to be processed, | ||
# uid: uuid for this job | ||
|
||
class UploadProcessor < BaseWorker | ||
@queue = :dradis_upload | ||
|
||
def perform_delegate | ||
file = options['file'] | ||
plugin_name = options['plugin'] | ||
uid = options['uid'] | ||
|
||
logger = Dradis::Core::Log.new(uid: uid) | ||
|
||
logger.write{ "Running Ruby version %s" % RUBY_VERSION } | ||
logger.write{ 'Worker process starting background task.' } | ||
|
||
plugin = plugin_name.constantize | ||
|
||
# Detect new-style gemified plugins | ||
if plugin::constants::include?(:Importer) | ||
content_service = Dradis::Plugins::ContentService.new(plugin: plugin) | ||
template_service = Dradis::Plugins::TemplateService.new(plugin: plugin) | ||
|
||
importer = plugin::Importer.new( | ||
logger: logger, | ||
content_service: content_service, | ||
template_service: template_service | ||
) | ||
|
||
importer.import(file: file) | ||
end | ||
logger.write{ 'Worker process completed.' } | ||
end | ||
end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,5 @@ | ||
Resque::Plugins::Status::Hash.expire_in = (24 * 60 * 60) # 24hrs in seconds | ||
|
||
# Resque::Server.use(Rack::Auth::Basic) do |user, password| | ||
# user == 'admin' && password == 'dradispro' | ||
# end |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
require 'resque/tasks' | ||
|
||
# We may want to include only the stuff we need, instead of loading the full | ||
# See: | ||
# http://railscasts.com/episodes/271-resque | ||
task "resque:setup" => :environment do | ||
ENV['QUEUE'] = 'dradis_upload,dradis_export' | ||
end |
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.