Split out in multiple queues

[pjaspers]

I added the doc so I had something to push, and we could have a discussion here on exactly what should go where.

I think we should have 3 queues:

• critical
• dhis2-safe
• default

Where critical means 'a user will need this', dhis2-safe means 'we won't hurt the dhis much'?, and 'default' means 'run this when the other two are done'.

If someone or something now enqueues 300 large invoices, we will still have delays, because the workers will pick 16 items of the default queue and we will only process the critical queue once a worker becomes free again.

If we really, really wanted to have critical always working, we would need to start two sidekiq process, I think.

So below, our workers and how I would divide them:

create_dhis2_element_for_formula_mapping_worker.rb
  Fast

create_dhis2_element_worker.rb
  Abstract (but could set queu for create_dhis2 and create_missing_dhis2)
  queue: critical

create_missing_dhis2_element_for_activity_worker.rb
  Fast

dhis2_analytics_worker.rb
  Fast (analytics is slow, but triggering it is fast)
  queues: critical

dhis2_snapshot_compactor.rb
  PORO used in worker
dhis2_snapshot_worker.rb
  Snapshot per project
  queues: critical

discard_invoicing_job_worker.rb
  Fast (cleans up orphaned tasks)

invoice_for_project_anchor_worker.rb
  queue: default

invoice_simulation_worker.rb
  queue is dhis2_safe

output_dataset_worker.rb
  queues: critical

project_coc_aoc_reference_worker.rb
  queues: critical

synchronise_deg_ds_worker.rb
  queues: critical

synchronise_groups_worker.rb
  queues: critical

update_metadata_worker.rb
  queues: critical