feat(workers): many small RQ worker features

[ankush]

reference for all of this: https://python-rq.org/docs/workers/

---

Multi-queue consumption

Why?

• servers with insufficient RAM. Using 1 common worker vs 3 workers = ~120-180MB RAM saved.
• Bench with low # of sites which dont truly need 3 separate workers for 3 queues because the workload is very low and wait times are tolerable.
• Lighter development setups. After this change you can use single worker for all queues like this:

- worker_short: bench worker --queue short
- worker_long: bench worker --queue long
- worker_default: bench worker --queue default

+ worker: bench worker --queue short,default,long

If you drop --queue argument altogether then all queues are consumed. So both of these are equievalent if you dont have custom queues.

worker: bench worker --queue short,default,long
worker: bench worker

Note: RQ has inbuilt prioritization, the order of specified queues matters. So in the example above until the short queue is finished default queue won't be worked upon and so on.

---

burst mode

If you only periodically need higher worker count you can do so by calling bench worker --burst this can be useful in high workload scenario.

Worker exits once specified queues are cleared.

---

Dequeue strategies

bench worker --strategy round_robin will give equal priority to all specified queues. random does it randomly. Default strategy (no flags required) gives priority based on order you specify.

docs: https://frappeframework.com/docs/v14/user/en/api/background_jobs

[codecov]

Codecov Report

Merging #18995 (ed28626) into develop (b170311) will decrease coverage by 0.07%.
The diff coverage is 41.66%.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop   #18995      +/-   ##
===========================================
- Coverage    63.46%   63.39%   -0.08%     
===========================================
  Files          750      750              
  Lines        67621    67733     +112     
  Branches      6027     6027              
===========================================
+ Hits         42917    42939      +22     
- Misses       21230    21320      +90     
  Partials      3474     3474              

Flag	Coverage Δ
server-mariadb	67.28% <55.55%> (-0.03%)	⬇️
server-postgres	67.29% <55.55%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.