with Rails 8.1.0.alpha & friends my recurring jobs are not enqueued

[wdiechmann]

I've had my fair share of issues with background/recurring jobs but this (new) issue really takes the prize - nothing gets enqueued and neither /mission_control_jobs nor Beekeeper Studio is able to shed any light on the subject 😢

I hurried to upgrade everything to 8.1.0.alpha (solid_queue to 1.0.2) and believe to have keep the important stuff in a pristine condition, like:

• bin/jobs sits in my
• /Procfile.dev which gets called from
• bin/dev and I'd expect it to read
• config/recurring.ymlwhich looks a lot like:

development:
  background_job_scheduler:
    class: BackgroundManagerJob
    enabled: true
    schedule: every 60.seconds

and gets configured with config/environments/development.rb that holds - amongst other stuff

  # Highlight code that enqueued background job in logs.
  config.active_job.queue_adapter = :solid_queue
  config.solid_queue.connects_to = { database: { writing: :queue, reading: :queue } }

(on purpose an exact copy of production in order to verify that production does the right thing - I know that you recommend setting the adapter to :async on development)

and the config/database.yml has a proper filling of

development:
  writer:
    <<: *default
    pool: 1
    database: storage/development.sqlite3
  reader:
    <<: *default
    readonly: true
    database: storage/development.sqlite3
  queue:
    <<: *default
    database: storage/development_queue.sqlite3
    migrations_paths: db/queue_migrate

and the tables are in place at storage/development_queue.sqlite* but, alas, empty (the *_recurring* ones that is)

[wdiechmann]

oh - my issue is (should it not be 'obvious'): how do I tap the recurring jobs on the back and say: go do your stuff (when you should)?

[rosa]

They should be enqueued automatically every 60 seconds. What do you get from ps ax | grep solid after you've started everything via bin/dev?

[wdiechmann]

I get exactly what Beekeeper Studio tells me when looking at solid_queue_processes

ps ax | grep solid
34642 s000  S+     0:01.09 solid-queue-supervisor(1.0.2): supervising 34664, 34665 
34664 s000  S+     0:00.13 solid-queue-dispatcher(1.0.2): dispatching every 1 seconds 
34665 s000  S+     0:00.51 solid-queue-worker(1.0.2): waiting for jobs in * 
34666 s000  S+     0:00.13 solid-queue-supervisor(1.0.2): supervising 34671, 34672      
34671 s000  S+     0:00.15 solid-queue-dispatcher(1.0.2): dispatching every 1 seconds      
34672 s000  S+     0:00.53 solid-queue-worker(1.0.2): waiting for jobs in *      
34834 s002  S+     0:00.00 grep solid

id	kind	last_heartbeat_at	supervisor_id	pid	hostname	metadata	created_at	name
43	Supervisor	2024-11-27 13:35:54.995775		29869	Walthers-MBAir-2.local		2024-11-27 13:34:54.965570	supervisor-3861a24fcab8b33ba273
44	Dispatcher	2024-11-27 13:35:55.055088	43	29871	Walthers-MBAir-2.local	{"polling_interval":1,"batch_size":500,"concurrency_maintenance_interval":600}	2024-11-27 13:34:55.010314	dispatcher-16cdac2a8f55b5c2b5d8
45	Worker	2024-11-27 13:35:55.070660	43	29872	Walthers-MBAir-2.local	{"polling_interval":0.1,"queues":"*","thread_pool_size":3}	2024-11-27 13:34:55.014622	worker-92242585595f07d3b92a

[rosa]

Aha, yes, no scheduler there for the recurring jobs. Does BackgroundManagerJob exist?

[wdiechmann]

- and I get exactly 1 line in my log from the jobs process(es):

16:48:53 web.1  | started with pid 34640
16:48:53 css.1  | started with pid 34641
16:48:53 job.1  | started with pid 34642
16:48:54 web.1  | => Booting Puma
16:48:54 web.1  | => Rails 8.1.0.alpha application starting in development 
16:48:54 web.1  | => Run `bin/rails server --help` for more startup options
16:48:54 web.1  | Watching: /Users/walther/Deling/Walther/src/mortimer/redbox/bellis
16:48:54 web.1  |   (?-mix:app\/views\/.+\.(erb|haml|slim|rb)$) => on_change
16:48:54 web.1  | Puma starting in single mode...
16:48:54 web.1  | * Puma version: 6.5.0 ("Sky's Version")
16:48:54 web.1  | * Ruby version: ruby 3.3.3 (2024-06-12 revision f1c7b6f435) +YJIT [arm64-darwin23]
16:48:54 web.1  | *  Min threads: 3
16:48:54 web.1  | *  Max threads: 3
16:48:54 web.1  | *  Environment: development
16:48:54 web.1  | *          PID: 34640
16:48:54 web.1  | * Listening on ssl://0.0.0.0:3000?key=config/box.mortimer_3-key.pem&cert=config/box.mortimer_3.pem&verify_mode=peer&ca=config/box.mortimer_3.pem
16:48:54 web.1  | Use Ctrl-C to stop

[wdiechmann]

Aha, yes, no scheduler there for the recurring jobs. Does BackgroundManagerJob exist?

# app/jobs/background_manager_job.rb
class BackgroundManagerJob < ApplicationJob
  queue_as :default

  #
  # step 1 background jobs will be running 3 jobs
  #
  #   1: user_state_job
  #   2: user_eu_state_job
  #   3: user_auto_punch_job
  #
  def perform
    UserMailer.error_report("testing!", "BackgroundManagerJob.perform").deliver_later
    dt = DateTime.current
    prepare_state_job if dt.hour == 5 and dt.min == 0
    prepare_eu_state_job if dt.hour == 5 and dt.min == 5
    prepare_auto_punch_job if dt.hour == 23 and dt.min == 50
  rescue => error
    UserMailer.error_report(error.to_s, "BackgroundManagerJob.perform").deliver_later
  end

  def prepare_state_job
    Tenant.all.each do |tenant|
      if tenant.users.any?
        UserStateJob.perform_later tenant: tenant
      end
    end
  end

  def prepare_eu_state_job
    Tenant.all.each do |tenant|
      if tenant.users.any?
        UserEuStateJob.perform_later tenant: tenant
      end
    end
  end

  def prepare_auto_punch_job
    Tenant.all.each do |tenant|
      if tenant.users.any?
        UserAutoPunchJob.perform_now tenant: tenant
      end
    end
    true
  end
end

[wdiechmann]

FYI - it did work back at Rails 7-something with Redis et al

(if - against all odds - everything should have buckled under the pressure, I'd expect at least a failed job to turn up in mission_control_job, right?

[rosa]

Thanks! And just to rule this out, you don't have SOLID_QUEUE_RECURRING_SCHEDULE env var set, pointing to a different location than config/recurring.yml, I suppose?

It seems as if something was up with the recurring configuration 😕

[wdiechmann]

unknown ENV

[wdiechmann]

and I've just finished setting up a fresh r810 with the exact same definition - with only DHH's popular Post table 😆

It does exactly likewise - so I'm not fiddling there

[rosa]

Can you open a rails console and copy the output from the following code?

SolidQueue::Configuration.new.send(:recurring_tasks)

config = SolidQueue::Configuration.new.send(:recurring_tasks_config)
tasks = config.map do |id, options|
  SolidQueue::RecurringTask.from_configuration(id, **options)
end

puts tasks.map(&:valid?)
pp tasks

[wdiechmann]

Loading development environment (Rails 8.1.0.alpha)
mortimer(dev)> SolidQueue::Configuration.new.send(:recurring_tasks)
=> []
mortimer(dev)> config = SolidQueue::Configuration.new.send(:recurring_tasks_config)
mortimer(dev)* tasks = config.map do |id, options|
mortimer(dev)*   SolidQueue::RecurringTask.from_configuration(id, **options)
mortimer(dev)> end
mortimer(dev)> 
mortimer(dev)> puts tasks.map(&:valid?)
mortimer(dev)> pp tasks
false
[#<SolidQueue::RecurringTask:0x0000000129276a58
  id: nil,
  key: "background_job_scheduler",
  schedule: "every 60.seconds",
  command: nil,
  class_name: "BackgroundManagerJob",
  arguments: [],
  queue_name: nil,
  priority: nil,
  static: true,
  description: nil,
  created_at: nil,
  updated_at: nil>]
=> 
[#<SolidQueue::RecurringTask:0x0000000129276a58
  id: nil,
  key: "background_job_scheduler",
  schedule: "every 60.seconds",
  command: nil,
  class_name: "BackgroundManagerJob",
  arguments: [],
  queue_name: nil,
  priority: nil,
  static: true,
  description: nil,
  created_at: nil,
  updated_at: nil>]

[wdiechmann]

would you like me to attach a zip with a 'clean' R8.1.0 setup for solid_queue and a single recurring job?

[wdiechmann]

If you are "at home" (in lovely Madrid - one of my favorite cities to visit ) I guess you are about to call it a day - so I'll attach that R8.1.0 default app and extend a buenos noche 🫶
r810.zip

[rosa]

That's super kind of you! ❤️ And I think I just spotted the problem!

every 60.seconds

This should be

every 60 seconds

with a space instead of ..

The schedule ends up being invalid and it fails to load silently. So sorry about that! This is basically #414 🙈 A PR was just opened for it yesterday (#427) but I haven't had time to review it yet. Hopefully I'll get to it this week and this won't happen again 😅