Removing jobs from the solid_queue_jobs table?

[wheee]

Hi,

Hopefully a simple question - what would happen if you remove a job intentionally that was just picked up for processing?

If you apply a lock on the row before removing it, would that prevent solid_queue from working on it?

[rosa]

If you apply a lock on the row before removing it, would that prevent solid_queue from working on it?

No, as the polling and claiming applies to ready_executions and claimed_executions records, not jobs.

About what would happen, it'd depend on when it gets deleted and whether you have this foreign key in your DB:

    add_foreign_key :solid_queue_claimed_executions, :solid_queue_jobs, column: :job_id, on_delete: :cascade

which is added by the default migration.

Here's the relevant code that handles job processing, part of ClaimedExecution:

  def perform
    result = execute

    if result.success?
      finished
    else
      failed_with(result.error)
    end
  ensure
    job.unblock_next_blocked_job
  end

  private
    def execute
      ActiveJob::Base.execute(job.arguments)
      Result.new(true, nil)
    rescue Exception => e
      Result.new(false, e)
    end

    def finished
      transaction do
        job.finished!
        destroy!
      end
    end

    def failed_with(error)
      transaction do
        job.failed_with(error)
        destroy!
      end
    end

If the deletion is committed before the job is loaded as part of

ActiveJob::Base.execute(job.arguments)

it won't get to run as job.arguments will raise an exception due to job being nil. That exception will be captured and returned as a result per:

    rescue Exception => e
      Result.new(false, e)
    end

However, another exception will be raised when calling failed_with because job.failed_with will fail due to job being nil. The ensure block will run:

    job.unblock_next_blocked_job

but it'll also fail because job is nil, and that error will get reported via on_thread_error (through handle_thread_error) if you have that configured.

The claimed execution would have been deleted with the job via the foreign key if you have that defined. If not, the claimed execution will be left behind and will be cleared up when the process that owned it is terminated.

If the deletion is committed after the job is loaded in claimed_execution but before it finishes running, the job will run and finish (ActiveJob::Base.execute(job.arguments)). Then, we'll try to run:

    def finished
      transaction do
        job.finished!
        destroy!
      end

But the two queries from job.finished! and destroy!; that would be something like:

UPDATE `solid_queue_jobs` SET `solid_queue_jobs`.`updated_at` = '...', `solid_queue_jobs`.`finished_at` = '...' WHERE `solid_queue_jobs`.`id` =..
DELETE FROM `solid_queue_claimed_executions` WHERE `solid_queue_claimed_executions`.`id` = ..

won't do anything because the job and the claimed execution would be gone from the database (the claimed execution would have been deleted with the job via the foreign key). If you didn't have the foreign key, then the claimed execution will get deleted there.
Then all will finish just fine, and it'd be similar to if the job had finished fine, and then it'd been deleted after being performed.

If the deletion is committed after the job is loaded in claimed_execution and after it finishes running, everything will be run normally. Your deletion will have to wait until the job is marked as finished or failed, as that'd take a lock on the job, and then it'll be deleted.