Upgrading to Que 2.x from 1.4 will cause the Que scheduler migration to fail on prior migrations

[rclimepoint]

Description

The Que scheduler migration calls Que::Scheduler::VersionSupport.enqueue_a_job(Que::Scheduler::SchedulerJob) when applying the first version of Que scheduler. This is not a problem if you run this migration after you have migrated Que, or if your database schema is already up-to-date and doesn't need to run all the previous migrations.

However, when starting off with an empty database, the db:migrate will start migrating from the first version. Since this is an upgrade, the Que migration will be migrated last and this results in the Que scheduler migration failing due to the 'kwargs' column not existing in the que_jobs table at this point.

Versions

ruby version: 2.7.6
que version: 2.2
que-scheduler version: 4.4
postgres version: 14.5

Messages

An example below is if you have following migration files:

• 20211201000000_create_que_scheduler_schema.rb

  class CreateQueSchedulerSchema < ActiveRecord::Migration[6.0]
    def change
      Que::Scheduler::Migrations.migrate!(version: 6)
    end
  end

• 20221201000000_update_que7.rb

  class UpdateQue7 < ActiveRecord::Migration[7.0]
    def change
      Que.migrate!(version: 7)
    end
  end

Caused by:
PG::UndefinedColumn: ERROR: column "kwargs" of relation "que_jobs" does not exist
LINE 1: ...e_jobs (queue, priority, run_at, job_class, args, kwargs, da...

(eval):9:in `run_sql_middleware'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/connection.rb:60:in `execute'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/connection_pool.rb:57:in `block in execute'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/connection_pool.rb:39:in `block in checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/active_record/connection.rb:13:in `block (2 levels) in checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/active_record/connection.rb:12:in `block in checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/activesupport-7.0.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/active_record/connection.rb:23:in `wrap_in_rails_executor'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/active_record/connection.rb:11:in `checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/connection_pool.rb:18:in `checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/connection_pool.rb:57:in `execute'
/usr/local/rbenv/versions/2.7.6/lib/ruby/2.7.0/forwardable.rb:235:in `execute'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/job.rb:131:in `enqueue'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-scheduler-4.4.0/lib/que/scheduler/version_support.rb:57:in `enqueue_a_job'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-scheduler-4.4.0/lib/que/scheduler/migrations.rb:55:in `migrate_up'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-scheduler-4.4.0/lib/que/scheduler/migrations.rb:22:in `block in migrate!'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-scheduler-4.4.0/lib/que/scheduler/db.rb:23:in `block in transaction'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/utils/transactions.rb:14:in `block in transaction'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/connection_pool.rb:39:in `block in checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/active_record/connection.rb:13:in `block (2 levels) in checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/activerecord-7.0.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/active_record/connection.rb:12:in `block in checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/activesupport-7.0.4/lib/active_support/execution_wrapper.rb:92:in `wrap'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/active_record/connection.rb:23:in `wrap_in_rails_executor'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/active_record/connection.rb:11:in `checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/connection_pool.rb:18:in `checkout'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-2.2.0/lib/que/utils/transactions.rb:12:in `transaction'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-scheduler-4.4.0/lib/que/scheduler/db.rb:23:in `call'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-scheduler-4.4.0/lib/que/scheduler/db.rb:23:in `transaction'
/usr/local/rbenv/versions/2.7.6/lib/ruby/gems/2.7.0/gems/que-scheduler-4.4.0/lib/que/scheduler/migrations.rb:19:in `migrate!'
/opt/xxx/db/migrate/20211201000000_create_que_scheduler_schema.rb:3:in `change'

Workaround

We've added a monkey patch to skip the job enqueue when doing a migration, and then re-enqueue the scheduler once we've migrated Que to the latest version:

module Que
  module Scheduler
    module Migrations
      class << self
        private

        def migrate_up(current, version)
          execute_step((current += 1), :up) until current == version
        end
      end
    end
  end
end

• 20221201000000_update_que7.rb

  class UpdateQue7 < ActiveRecord::Migration[7.0]
    def change
      Que.migrate!(version: 7)
      Que::Scheduler::Migrations.reenqueue_scheduler_if_missing
    end
  end

[hlascelles]

Thank you for the excellent report @rclimepoint!

Yes, I can see how that would be an issue. Hmm, this is a tough one - how to know if there are "more migrations to come". I guess maybe for this exact error I can make it give a precise error message with instructions of what to do.

Will have a look...

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hlascelles]

Will have a look at this one still. I think it isn't urgent, but a better error message would be nice.

[blunckr-aj]

It seems like the only reliable way to solve this might be to not attempt to enqueue the job during migrations at all. Instead you could add a step to the instructions to enqueue it after migrations, either in seeds or a rake task.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[hlascelles]

I will look at this...

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.