Fix to isolate the sidekiq process that runs the scheduler job

[noellabo]

Fix #14764

Introduce a scheduler queue that runs the scheduler job so that only the sidekiq process that processes this queue runs the schedule.

In the case of a single sidekiq process, everything is processed, including the schedule, as before.

bundle exec sidekiq -c 25

If you have multiple sidekiq processes running, only the process that handles the scheduler queue handles the schedule.

bundle exec sidekiq -c 25 -q defalut, 6 -q mailers, 2
bundle exec sidekiq -c 25 -q push, 4 -q pull -q scheduler

For existing configurations with multiple sidekiq processes running explicitly in the queue, this change prevents scheduled execution because there is no process to handle the scheduler queue.

bundle exec sidekiq -c 25 -q defalut, 6 -q mailers, 2
bundle exec sidekiq -c 25 -q push, 4 -q pull

The solution is to have one of the processes process the scheduler queue, or add a dedicated process for scheduled execution.

bundle exec sidekiq -c 5 -q scheduler

[Gargron]

With this change I believe locks can be taken out of the scheduler classes

[RobPurcellUK]

Just upgrading to 3.3.0 from 3.1.3 and I noticed the new queue. How do you see this working if running multiple instances in containers (k8s cluster)?

Running multiple instances of the streaming api, web and sidekiq processes for resilience and performance takes a hit here, as we need a single dedicated instance for the scheduler.

Any thoughts?