Scheduled jobs not shown under Schedule tab

[benthorner]

After upgrading to 4.2.0 from 4.1.0, our scheduled jobs (loaded statically) are no longer visible under the Schedule tab, but still seem to work (i.e. get scheduled!). Some code (part of a Rails app):

config/routes.rb

require 'resque/scheduler/server'
require 'resque/server'
Rails.application.routes.draw do
...

lib/tasks/resque.rake

namespace :resque do
  task setup_schedule: :setup do
    require 'resque-scheduler'
    Resque.schedule = YAML.load_file('config/schedule.yml')
  end

  task scheduler: :setup_schedule
end

I tried switching to a dynamic schedule, but that doesn't seem to help. Downgrading resque-scheduler (only) to 4.1.0 and everything works again...

[carsonreinke]

Is resque-scheduler running when you looked at the schedule? Any odd Redis/Resque configuration or initializer settings?

[benthorner]

I did a grep for 'resque' and 'redis' in config/. The only result was the following (in an initializer).

ActiveJob::Base.queue_adapter = :resque

I removed this line and restarted everything, but the tab was still bare. Hope that helps :-)

[Janther]

I can confirm this.
My jobs are being scheduled if I but not displayed on the tab look at the log, but are not displayed on the tab.
What Can I provide to help you debugging this?

[hkdsun]

I cannot reproduce this issue. I've never used the web interface but I tried to load the web server using examples/run-resque-web and a dummy schedule. The dummy schedule shows up in the schedule tab.

I'm a bit busy these days but if I can get quick steps to reproduce I'm willing to help

[carsonreinke]

In resque-web go to Stats -> keys, is there a schedules or resque_scheduler_master_lock listed?

[Janther]

It's listed but the type is none and the size is []

[benthorner]

Hello all (sorry for the delay in replying). I can confirm these same:

 Keys owned by Resque Client connected to redis://127.0.0.1:6379/0
 (All keys are actually prefixed with "resque:")
 key                                            type    size
 delayed:last_enqueued_at                       hash    
 queue:default                                  list    4
 queues                                         set     1
 resque_scheduler_master_lock                   none    []
 stat:failed                                    string  1
 stat:processed                                 string  1

[hkdsun]

In resque-web go to Stats -> keys, is there a schedules ... listed?

The schedules hash has been deprecated in favour of in-memory storage (see #535). However, that's not how the schedule tab is populated (relevant part of the code).

My guess is that @benthorner and @Janther (and most deployments really) run the web server independently of resque-scheduler (for example through their rails app). Resque::Scheduler#schedule doesn't talk to redis anymore for non-persistent schedules, so the schedule tab appears empty.

[carsonreinke]

...so basically you just need to ensure you have something like Resque.schedule = YAML.load_file('config/schedule.yml') in an initializer so that resque-scheduler can know of the schedule.

[benthorner]

@hkdsun is correct in my case: running the web server and scheduler in separate processes. As @carsonreinke says, I am doing a YAML.load_file(...) as part of the initialisation for resque-scheduler; since it doesn't talk to redis anymore, there's no way for the web server process to know the schedule.

Any clues for a suggested fix for this (I agree with @hkdsun that this probably affects a lot of users)? The readme is confusing me here: it looks like I need to set the persist flag for each job to get it to persist with redis, and perhaps enable dynamic schedule (haven't tried it yet, just putting it out there :-).

[Janther]

Yes, moving this line

Resque.schedule = YAML.load_file Rails.root.join('config', 'resque_schedule.yml')

from lib/tasks/resque.rake to config/initializers/resque.rb did the trick.
The tasks are scheduled by the scheduler and displayed by the web server

Maybe this should be added to the Readme

[benthorner]

Can confirm this works too 👍, although it does mean the schedule loaded by the scheduler could diverge from the one on the web server. I agree with @Janther the Readme should be updated in the meantime.

[carsonreinke]

@benthorner The dynamic and persisted schedule is different, unless you are dynamically changing your schedule, you won't be using those options. The persist option simply just allows a dynamic schedule to be persisted if the scheduler daemon was restarted.

[benthorner]

You're right @carsonreinke, I wouldn't normally be using those options. Unfortunately I can't put Resque.schedule = ... in an initializer, as this requires redis to be running whenever I run any task in the Rails environment, including assets:precompile (which I run as part of an isolated Docker build).

Adding 'persist:true' for each job in the schedule forces it to be stored in redis, which means it's visible in the UI. I agree that's not what it should be used for - I was just trying to look for a solution that worked for me. Since I don't really need the Schedules tab, I think I'll just get rid of it :-)

The Readme is still wrong, though - https://github.com/resque/resque-scheduler#changes-as-of-200.

[ciaoben]

My guess is that @benthorner and @Janther (and most deployments really) run the web server independently of resque-scheduler (for example through their rails app). Resque::Scheduler#schedule doesn't talk to redis anymore for non-persistent schedules, so the schedule tab appears empty.

Can someone explain me how to not launch the web-server ndependently of resque-scheduler like he said?

I have managed to make it works, and here what I launch to make it run:

rake resque:scheduler
resque-web ./path/to/a/config/file.rb

What is the other method?

[carsonreinke]

@ciaoben, the file ./path/to/a/config/file.rb would have to load the static schedule from the YAML file.

There are three types of schedules: static, dynamic, and both. A dynamic schedule that is persisted will not require loading of schedule for resque-web, since this will come from Redis. If you are using either static or both, resque-web must be initialized with the static schedule (e.g. Resque.schedule = YAML.load_file).

The docs make things confusing it should be improved.

[ciaoben]

@carsonreinke Thanks, but it is not what I meant. I was asking about this:

My guess is that @benthorner and @Janther (and most deployments really) run the web server independently of resque-scheduler (for example through their rails app).

I would like to know if there are another method to run resque-scheduler DEPENDENTLY with the web server.

If I test in one of the controller of my Rails app for the Resque::Scheduler.dynamic, I get false even if in my resque.rake file I have configured to true.

The scheduling works dynamically, and it is al right, but the I am having problem with the gem resque-scheduler-web because I am having trouble to let know my app of the current configuration of Resque::Scheduler.

Any advice would be really appreciated.

@benthorner says to move Resque.schedule=. I understand this, but I would like to know how can I do it if I don't use static schedule, but a dynamic one.

[carsonreinke]

@ciaoben, my bad. Maybe @hkdsun is referring to running the resque-scheduler standalone executable. You can't run it in web/app server, unless it was forked.

Resque::Scheduler.dynamic must be set for both your Rails app and resque.rake, it is not persisted within Redis. Probably the simplest is just have something like this in a resque.rb initializer:

require 'resque-scheduler'

Resque::Scheduler.dynamic = true #Only enable if you use a dynamic schedule
Resque.schedule = YAML.load_file(...) #This is for a static schedule only

To load a dynamic schedule, you don't have to do this each time, but you would use Resque.set_schedule(...).

I think the docs need to be updated!

[ciaoben]

@carsonreinke thanks. It is the solution I came up with.

Basically, if I've understood right, you can start the process as a rake task, or as a standalone. In BOTH cases your rails app will be unaware of it, so it will be unable to read the configuration with which has been started the scheduler. To avoid this is necessary to specify the perks of the configuration in some rails initializers so the app (and its gems) can work well with the scheduler.

[carsonreinke]

Correct, the resque-web must have some resque-scheduler configuration done for it to work correctly.

[toomanyjoes]

Just want to make sure I'm understanding this correctly, in order to see the schedule in resque-web you must create an initializer as described. However if you follow the documentation and initialize your background worker with rake resque:scheduler the scheduler will not find the schedule unless it is also created as a rake task (as described in the documentation)? So one must define the schedule in two different places to have a worker running the schedule as well as resque-web knowing about the schedule? Ugh... I hope I'm understanding this incorrectly.

I'm mounting resque-web in my routes.rb file if that matters, not running it from the command line. The former is not described in this documentation as far as I found.

[carsonreinke]

@toomanyjoes you do NOT need to be running the Rake task for resque-web to display the schedule. You do need an initializer to initialize the configuration and load the static schedule. The schedule is not defined is two places, there is two different types of schedules, a static one, usually load from a YAML file, and a dynamic one that is defined through code. Take a look at this example!