Issues - collectiveidea/delayed_job

collectiveidea / delayed_job

123

306

Fork of tobi/delayed_job
Description:	Database based asynchronously priority queue system -- Extracted from Shopify edit

Sort by: Priority | Votes | Last Updated

Loading…

3.

1 vote

./script/delayed_job restart bounces processes, but no new PIDs
18 comments Created 3 months ago by chrisfinne

Title

Body
the PID files are never created for the new process(es), so subsequent restarts and stops won't work. Not sure if this is a problem the the daemons gem (i'm on the latest 1.0.10) or delayed_job. I see this on my Mac and Ubuntu boxes.
or cancel
the PID files are never created for the new process(es), so subsequent restarts and stops won't work.

Not sure if this is a problem the the daemons gem (i'm on the latest 1.0.10) or delayed_job.

I see this on my Mac and Ubuntu boxes.

Comments
peoplebase Sun Aug 30 19:47:22 -0700 2009 | link

I've had this problem too. I haven't been able to track it down, but it seems like an issue with the daemons library

Comment
I've had this problem too. I haven't been able to track it down, but it seems like an issue with the daemons library
or cancel

chrisfinne Mon Aug 31 03:59:30 -0700 2009 | link

Here's my hack in my cap script to restart it. (I hard-code to launch 3 delayed_job processes).
http://gist.github.com/178397

Comment
Here's my hack in my cap script to restart it. (I hard-code to launch 3 delayed_job processes). http://gist.github.com/178397
or cancel

sant0sk1 Tue Sep 22 18:37:39 -0700 2009 | link

I have this problem as well. Will try chrisfinne's hack until somebody comes up with a fix.

Comment
I have this problem as well. Will try chrisfinne's hack until somebody comes up with a fix.
or cancel

rmm5t Wed Sep 23 10:41:53 -0700 2009 | link

I just ran into this problem as well. It appears to be a problem with the daemons gem. When the daemons gem "restarts" it stops, sleeps for 1 sec, and then starts. Pid file cleanup happens all over the place (I think incorrectly), and it looks like a race condition occurs. The daemons gem deletes pid files when you call stop, when the daemon exits normally, and when it traps a kill signal. It can take delayed_job a few seconds to respond to a kill signal because it first finishes what it was doing. Meanwhile, a new delayed_job daemon is forked (even after daemons 1 sec delay) and a new pid file is written. After the restart, the cleanup tasks of the original daemon clean up the pid files again -- incorrectly blowing away the new pid file.

In summary, the problem looks to be the daemons gem, and that gem should wait for the original process to stop running before restarting a new process.

Comment
I just ran into this problem as well. It appears to be a problem with the daemons gem. When the daemons gem "restarts" it stops, sleeps for 1 sec, and then starts. Pid file cleanup happens all over the place (I think incorrectly), and it looks like a race condition occurs. The daemons gem deletes pid files when you call stop, when the daemon exits normally, and when it traps a kill signal. It can take delayed_job a few seconds to respond to a kill signal because it first finishes what it was doing. Meanwhile, a new delayed_job daemon is forked (even after daemons 1 sec delay) and a new pid file is written. After the restart, the cleanup tasks of the original daemon clean up the pid files again -- incorrectly blowing away the new pid file. In summary, the problem looks to be the daemons gem, and that gem should wait for the original process to stop running before restarting a new process.
or cancel

rmm5t Wed Sep 23 10:55:54 -0700 2009 | link

Turns out this is a known problem with the daemons gem. Too bad the simple fix hasn't been applied.
http://rubyforge.org/tracker/index.php?func=detail&aid=21050&group_id=524&atid=2084

Comment
Turns out this is a known problem with the daemons gem. Too bad the simple fix hasn't been applied. http://rubyforge.org/tracker/index.php?func=detail&aid=21050&group_id=524&atid=2084
or cancel
rmm5t Wed Sep 23 11:07:46 -0700 2009 | link
There's a fork of daemons that fixes the problem. http://github.com/ghazel/daemons

$ sudo gem uninstall daemons $ sudo gem install ghazel-daemons

v1.0.11 works well for me and I'm going to start freezing my apps that use delayed_job to the ghazel-daemons gem.
Comment
There's a fork of daemons that fixes the problem. http://github.com/ghazel/daemons $ sudo gem uninstall daemons $ sudo gem install ghazel-daemons v1.0.11 works well for me and I'm going to start freezing my apps that use delayed_job to the ghazel-daemons gem.
or cancel
chrisfinne Wed Sep 23 21:36:34 -0700 2009 | link

closing

Comment
closing
or cancel

rmm5t Thu Sep 24 06:02:02 -0700 2009 | link

I'm not sure this is worth closing until either daemons is fixed, delayed_job puts an explicit dependency on ghazel-daemons, or delayed_job implements a workaround.

Comment
I'm not sure this is worth closing until either daemons is fixed, delayed_job puts an explicit dependency on ghazel-daemons, or delayed_job implements a workaround.
or cancel

sant0sk1 Thu Sep 24 06:12:43 -0700 2009 | link

I agree, Its much easier for others to find if it is open and technically the issue has not been resolved.

Comment
I agree, Its much easier for others to find if it is open and technically the issue has not been resolved.
or cancel

chrisfinne Thu Sep 24 10:16:46 -0700 2009 | link

I figured that since it was definitively proven to be another package's bug and a solid workaround beyond my ugly hack was detailed, I'd close it, but you make some good points, so I'll leave it open.

Comment
I figured that since it was definitively proven to be another package's bug and a solid workaround beyond my ugly hack was detailed, I'd close it, but you make some good points, so I'll leave it open.
or cancel

brandon Mon Sep 28 09:16:05 -0700 2009 | link

So is everyone still having this issue with the latest version?

Comment
So is everyone still having this issue with the latest version?
or cancel

rmm5t Mon Sep 28 09:58:37 -0700 2009 | link

Brandon, Yes, the restart problem is in daemons-1.0.10 and has been there for about a year.

Edit: To clarify, ghazel-daemons-1.0.11 fixes the problem, but that fork is not well known and most people running delayed_job have daemons-1.0.10 installed.

Comment
Brandon, Yes, the restart problem is in daemons-1.0.10 and has been there for about a year. Edit: To clarify, ghazel-daemons-1.0.11 fixes the problem, but that fork is not well known and most people running delayed_job have daemons-1.0.10 installed.
or cancel

brandon Mon Sep 28 18:17:15 -0700 2009 | link

Has anyone tried to contact the maintainer of daemons? I'm thinking about just ditching it altogether and trying to figure out a different solution.

Comment
Has anyone tried to contact the maintainer of daemons? I'm thinking about just ditching it altogether and trying to figure out a different solution.
or cancel

rmm5t Mon Sep 28 19:31:29 -0700 2009 | link

Brandon, The maintainer of daemons (Thomas Uehlinger) responded to the associated ticket about a year ago, but nothing since. I haven't tried to contact him myself. It doesn't look like there's been any activity in daemons since either.
http://rubyforge.org/tracker/index.php?func=detail&aid=21050&group_id=524&atid=2084

Perhaps daemon-spawn helps, though the gem on rubyforge is either not published yet or missing.
http://github.com/alexvollmer/daemon-spawn

Comment
Brandon, The maintainer of daemons (Thomas Uehlinger) responded to the associated ticket about a year ago, but nothing since. I haven't tried to contact him myself. It doesn't look like there's been any activity in daemons since either. http://rubyforge.org/tracker/index.php?func=detail&aid=21050&group_id=524&atid=2084 Perhaps daemon-spawn helps, though the gem on rubyforge is either not published yet or missing. http://github.com/alexvollmer/daemon-spawn
or cancel

tcocca Tue Oct 06 05:11:24 -0700 2009 | link

chrisfinne's hack worked perfect for me after struggling with this for so long.

Can this be overriddden in the config/deploy.rb instead of editing the plugin or gem?

Comment
chrisfinne's hack worked perfect for me after struggling with this for so long. Can this be overriddden in the config/deploy.rb instead of editing the plugin or gem?
or cancel

dlegg Thu Oct 08 06:19:23 -0700 2009 | link

One thing I have noticed out of all of this is that restart option definitely doesn't work regardless. The pidfile gets blown away but the process isn't actually stopped or started which means that you can start another process with the zombie hanging around. You have to do an explicit stop and then start if you want to restart delayed_job, and that seems to work. I haven't done enough testing yet, but with chrisfinne's script above are we saying that we can have a similar thing happen with an explicit stop/start, hence waiting for the process to actually end?

Comment
One thing I have noticed out of all of this is that restart option definitely doesn't work regardless. The pidfile gets blown away but the process isn't actually stopped or started which means that you can start another process with the zombie hanging around. You have to do an explicit stop and then start if you want to restart delayed_job, and that seems to work. I haven't done enough testing yet, but with chrisfinne's script above are we saying that we can have a similar thing happen with an explicit stop/start, hence waiting for the process to actually end?
or cancel

tcocca Thu Oct 08 13:55:55 -0700 2009 | link

dlegg, correct, the restart doesn't work. chrisfinne's script does an explicit stop, then keeps checking for the process to actually stop and then not call start again till that process has stopped and the pid has dissapeared. This worked great for me. In the cap deploy you will see how long it takes for the process to actually stop with the "waiting for process to stop ..." text.

I would recommend trying this.

I haven't tried the other version of the daemons gem though so I can't speak to that.

~ tom

Comment
dlegg, correct, the restart doesn't work. chrisfinne's script does an explicit stop, then keeps checking for the process to actually stop and then not call start again till that process has stopped and the pid has dissapeared. This worked great for me. In the cap deploy you will see how long it takes for the process to actually stop with the "waiting for process to stop ..." text. I would recommend trying this. I haven't tried the other version of the daemons gem though so I can't speak to that. ~ tom
or cancel

scottj97 Thu Oct 29 18:57:11 -0700 2009 | link

chrisfinne's recipe works great, unless your deploy server is also your development server, in which case it will wait forever because it sees the 'cap delayed_job:restart' task in the process list. (I took care of that with another grep -v.)

What I don't understand is why an idle delayed_job server should take 20 seconds or more to exit??

Comment
chrisfinne's recipe works great, unless your deploy server is also your development server, in which case it will wait forever because it sees the 'cap delayed_job:restart' task in the process list. (I took care of that with another grep -v.) What I don't understand is why an idle delayed_job server should take 20 seconds or more to exit??
or cancel

Parsed with GitHub Flavored Markdown
6.

0 votes

starting multiple delayed_jobs via script/delayed_job
0 comments Created 2 months ago by skippy

Title

Body
being able to start multiple script/delayed_job's would be nice, specifically to allow this: script/delayed_job start -e Production -n 2 --min-priority 1 script/delayed_job start -e Production -n 8 ie, to be able to have multiple delayed_job processes going, but some have the minimum priority flag set. I find this desirable behavior. I can do this using rake jobs:work, and something like god, but if I want to use monit I have to use a script like delayed_job because monit doesn't grab and handle pid files like god does. if you would like to do this, you need to replace the lines: ` def daemonize worker_count.times do |worker_index| process_name = worker_count == 1 ? "delayed_job" : "delayed_job.#{worker_index}" Daemons.run_proc(process_name, :dir => "#{RAILS_ROOT}/tmp/pids", :dir_mode => :normal, :ARGV => @args) do |*args| run process_name end end end` with ` def daemonize worker_count.times do |worker_index| process_name = "delayed_job_#{@options.zip.join('_')}" process_name += ".#{worker_index}" if worker_count > 1 Daemons.run_proc(process_name, :dir => "#{RAILS_ROOT}/tmp/pids", :dir_mode => :normal, :ARGV => @args, :multiple => true) do |*args| run process_name end end end` thanks, Adam
or cancel
being able to start multiple script/delayed_job's would be nice, specifically to allow this:

script/delayed_job start -e Production -n 2 --min-priority 1
script/delayed_job start -e Production -n 8

ie, to be able to have multiple delayed_job processes going, but some have the minimum priority flag set.

I find this desirable behavior. I can do this using rake jobs:work, and something like god, but if I want to use monit I have to use a script like delayed_job because monit doesn't grab and handle pid files like god does.

if you would like to do this, you need to replace the lines:

` def daemonize
```
  worker_count.times do |worker_index|
    process_name = worker_count == 1 ? "delayed_job" : "delayed_job.#{worker_index}"
    Daemons.run_proc(process_name, :dir => "#{RAILS_ROOT}/tmp/pids", :dir_mode => :normal, :ARGV => @args) do |*args|
      run process_name
    end
  end
end`
```
with

` def daemonize
```
  worker_count.times do |worker_index|
    process_name = "delayed_job_#{@options.zip.join('_')}"
    process_name += ".#{worker_index}" if worker_count > 1
    Daemons.run_proc(process_name, :dir => "#{RAILS_ROOT}/tmp/pids", :dir_mode => :normal, :ARGV => @args, :multiple => true) do |*args|
      run process_name
    end
  end
end`
```
thanks,
Adam
Comments

Parsed with GitHub Flavored Markdown
14.

1 vote

Worker name improperly set when using script/delayed_job
3 comments Created about 1 month ago by albus522

Title

Body
The pid is set in the worker name before the daemon splits off the worker processes which means the worker name has the pid of the original process and not the worker process. I have a fix here xspond/delayed_job@8f9a321cf6e50617e286398c8f4087a5bf2f781d
or cancel

The pid is set in the worker name before the daemon splits off the worker processes which means the worker name has the pid of the original process and not the worker process.

I have a fix here
xspond/delayed_job@8f9a321

Comments

jerryvos Fri Nov 20 07:04:58 -0800 2009 | link

fix looks pretty good IMO

Comment
fix looks pretty good IMO
or cancel

albus522 Fri Nov 20 07:13:49 -0800 2009 | link

commit has already been merged

Comment
commit has already been merged
or cancel

jerryvos Fri Nov 20 07:15:24 -0800 2009 | link

Thanks for the update. Can someone please close this issue.

Comment
Thanks for the update. Can someone please close this issue.
or cancel

Parsed with GitHub Flavored Markdown
16.

0 votes

Rake tasks fail with: uninitialized constant Delayed
1 comment Created about 1 month ago by ebababi

Title

Body
A quick remedy to the issue above would be to include in project's Rakefile the line: autoload :Delayed, 'delayed_job' Another place for the line above could be the delayed/tasks.rb itself. Please, comment on the above.
or cancel
A quick remedy to the issue above would be to include in project's Rakefile the line:

autoload :Delayed, 'delayed_job'

Another place for the line above could be the delayed/tasks.rb itself. Please, comment on the above.

Comments
weppos Tue Nov 03 05:51:02 -0800 2009 | link
Be sure to include the environment dependency in your rake task.

task :foo => [:environment] do # ... end
Comment
Be sure to include the environment dependency in your rake task. task :foo => [:environment] do # ... end
or cancel
Parsed with GitHub Flavored Markdown
17.

0 votes

undefined method ***** for #<YAML::Object:****>
3 comments Created 29 days ago by jerefrer

Title

Body
Hi there, Here's a bit of code, followed by explanations. So here's my Job class : class ImportJobs < Struct.new(:type, :data_file, :user) def perform Import.tickets(data_file, user) if type == "tickets" Import.bl_bre(data_file) if type == "bl" end end And here's the call : Delayed::Job.enqueue(ImportJobs.new(type, data_file, @current_user)) As you can see, I use the ImportJob class to call 2 different methods of the same Import class, depending on the type argument given. The first method doesn't actually get a "data_file" object, it gets an Array, and a "user" object. And it works fine. BUT, the second function which only gets a "data_file" object fails throwing this error : "undefined method `path' for #<YAML::Object:0x7f16cf40a488>" ("path" being a method of my DataFile class) So I can't manage to figure out why it works when serializing a "user" object, while failing when serializing a "data_file" object. Obviously, when I call the Import.bl_bre(data_file) method directly (without using DelayedJobs), everythings works well. Any idea ?
or cancel
Hi there,

Here's a bit of code, followed by explanations.

So here's my Job class :
```
class ImportJobs < Struct.new(:type, :data_file, :user)
  def perform
       Import.tickets(data_file, user) if type == "tickets"
       Import.bl_bre(data_file) if type == "bl"
  end
end
```
And here's the call :
```
 Delayed::Job.enqueue(ImportJobs.new(type, data_file, @current_user))
```
As you can see, I use the ImportJob class to call 2 different methods of the same Import class, depending on the type argument given. The first method doesn't actually get a "data_file" object, it gets an Array, and a "user" object. And it works fine.

BUT, the second function which only gets a "data_file" object fails throwing this error :
"undefined method `path' for #<YAML::Object:0x7f16cf40a488>" ("path" being a method of my DataFile class)

So I can't manage to figure out why it works when serializing a "user" object, while failing when serializing a "data_file" object.

Obviously, when I call the Import.bl_bre(data_file) method directly (without using DelayedJobs), everythings works well.

Any idea ?
Comments

laserlemon Thu Oct 29 07:46:24 -0700 2009 | link

I ran across this issue as well. It's because whatever class of which data_file is an instance has not been initialized yet. Until delayed_job starts trying to constantize the class before loading the YAML in, there's not a clean way to get around this (that I know of).

Comment
I ran across this issue as well. It's because whatever class of which `data_file` is an instance has not been initialized yet. Until delayed_job starts trying to constantize the class before loading the YAML in, there's not a clean way to get around this (that I know of).
or cancel

jerefrer Mon Nov 02 06:26:47 -0800 2009 | link

So what's the workaround ? Don't give a DataFile object but the path as a string until they make changes ?

Comment
So what's the workaround ? Don't give a DataFile object but the path as a string until they make changes ?
or cancel

hojberg Wed Nov 04 05:21:11 -0800 2009 | link

thats what i did to solve this atleast

Comment
thats what i did to solve this atleast
or cancel

Parsed with GitHub Flavored Markdown
18.

0 votes

'Delayed::Job when another worker is already performing an task, it should be able to get access to the task if it was started more then max_age ago' FAILED
0 comments Created 24 days ago by weppos

Title

Body
1) 'Delayed::Job when another worker is already performing an task, it should be able to get access to the task if it was started more then max_age ago' FAILED expected: "worker2", got: "worker1" (using ==) ./spec/job_spec.rb:228: Finished in 2.131937 seconds 54 examples, 1 failure Tested against cbb8bb06a4c487724bc0b732ff37b33e31ec8f08
or cancel
```
1)
'Delayed::Job when another worker is already performing an task, it should be able to get access to the task if it was started more then max_age ago' FAILED
expected: "worker2",
     got: "worker1" (using ==)
./spec/job_spec.rb:228:

Finished in 2.131937 seconds

54 examples, 1 failure
```
Tested against cbb8bb0
Comments

Parsed with GitHub Flavored Markdown
19.

0 votes

Does restarting allow current workers to finish first?
0 comments Created 22 days ago by barmstrong

Title

Body
When "script/delayed_job restart production" gets run on a deploy, does it let the existing worker(s) finish or are they killed right away? I'm concerned about a job getting killed in the middle. By the way, I'm using the daemon-spawn gem. Thanks! Brian
or cancel

When "script/delayed_job restart production" gets run on a deploy, does it let the existing worker(s) finish or are they killed right away? I'm concerned about a job getting killed in the middle.

By the way, I'm using the daemon-spawn gem. Thanks!
Brian

Comments

Parsed with GitHub Flavored Markdown
20.

0 votes

You have a nil object when you didn't expect it
7 comments Created 20 days ago by kakra

Title

Body
Since upgrading I get the following backtrace in "rake jobs:work" and a similar exception when trying to start script/delayed_job: You have a nil object when you didn't expect it! The error occurred while evaluating nil.now .../vendor/plugins/delayed_job/lib/delayed/job.rb:253:in `db_time_now' .../vendor/plugins/delayed_job/lib/delayed/job.rb:126:in `find_available' .../vendor/plugins/delayed_job/lib/delayed/job.rb:155:in `reserve_and_run_one_job' .../vendor/plugins/delayed_job/lib/delayed/job.rb:202:in `work_off' .../vendor/plugins/delayed_job/lib/delayed/job.rb:201:in `times' .../vendor/plugins/delayed_job/lib/delayed/job.rb:201:in `work_off' .../vendor/plugins/delayed_job/lib/delayed/worker.rb:30:in `start' /usr/lib/ruby/1.8/benchmark.rb:308:in `realtime' .../vendor/plugins/delayed_job/lib/delayed/worker.rb:29:in `start' .../vendor/plugins/delayed_job/lib/delayed/worker.rb:26:in `loop' .../vendor/plugins/delayed_job/lib/delayed/worker.rb:26:in `start' .../vendor/plugins/delayed_job/lib/delayed/tasks.rb:13
or cancel
Since upgrading I get the following backtrace in "rake jobs:work" and a similar exception when trying to start script/delayed_job:

You have a nil object when you didn't expect it!
The error occurred while evaluating nil.now
.../vendor/plugins/delayed_job/lib/delayed/job.rb:253:in db_time_now' .../vendor/plugins/delayed_job/lib/delayed/job.rb:126:infind_available' .../vendor/plugins/delayed_job/lib/delayed/job.rb:155:in reserve_and_run_one_job' .../vendor/plugins/delayed_job/lib/delayed/job.rb:202:inwork_off' .../vendor/plugins/delayed_job/lib/delayed/job.rb:201:in times' .../vendor/plugins/delayed_job/lib/delayed/job.rb:201:inwork_off' .../vendor/plugins/delayed_job/lib/delayed/worker.rb:30:in start' /usr/lib/ruby/1.8/benchmark.rb:308:inrealtime' .../vendor/plugins/delayed_job/lib/delayed/worker.rb:29:in start' .../vendor/plugins/delayed_job/lib/delayed/worker.rb:26:inloop' .../vendor/plugins/delayed_job/lib/delayed/worker.rb:26:in `start' .../vendor/plugins/delayed_job/lib/delayed/tasks.rb:13

Comments
hojberg Thu Nov 12 02:19:46 -0800 2009 | link

i have this same problem

Comment
i have this same problem
or cancel

hojberg Thu Nov 12 02:36:10 -0800 2009 | link

hax in config.active_record.default_timezone = :utc in you config/envoriment.rb file and it should work again.

Comment
hax in config.active_record.default_timezone = :utc in you config/envoriment.rb file and it should work again.
or cancel
hojberg Thu Nov 12 02:38:58 -0800 2009 | link
So the problem is this method in job.rb

def self.db_time_now (ActiveRecord::Base.default_timezone == :utc) ? Time.now.utc : Time.zone.now end

reason being that if you havent set your default_time zone it tries to find Time.zone.now and that doesnt not exist :(
Comment
So the problem is this method in job.rb def self.db_time_now (ActiveRecord::Base.default_timezone == :utc) ? Time.now.utc : Time.zone.now end reason being that if you havent set your default_time zone it tries to find Time.zone.now and that doesnt not exist :(
or cancel
kakra Thu Nov 12 02:53:20 -0800 2009 | link

Yep, that helped. I already figured that out myself. But delayed_job shouldn't insist on a non-mandatory configuration option - at least not silently.

Comment
Yep, [that](http://github.com/collectiveidea/delayed_job/issues/#issue/20/comment/74319) helped. I already figured that out myself. But delayed_job shouldn't insist on a non-mandatory configuration option - at least not silently.
or cancel

hojberg Thu Nov 12 03:00:24 -0800 2009 | link

ye totally agree!

Comment
ye totally agree!
or cancel

bmarini Thu Nov 12 21:42:14 -0800 2009 | link

This seems to only be a problem when delayed_job is installed as a plugin. I don't know why that is.

Comment
This seems to only be a problem when delayed_job is installed as a plugin. I don't know why that is.
or cancel

kakra Fri Nov 13 07:44:12 -0800 2009 | link

I temporarily fixed that by putting the following monkey patch in my lib dir and requiring it in the environment: http://gist.github.com/233905

Comment
I temporarily fixed that by putting the following monkey patch in my lib dir and requiring it in the environment: http://gist.github.com/233905
or cancel

Parsed with GitHub Flavored Markdown

Labels

Comments

Comments

Comments

Comments

Comments

Comments

Comments

Comments