Refactor job queue

[dapplion]

Motivation

To develop #2265 we need a queue that is able to choose between FIFO and LIFO. Our current JobQueue does not support that since it-pushable hardcodes a FIFO module inside it.

Description

I've taken this opportunity to refactor the queue with backed-in metrics that will be used on all of it's consumers

• Use a simple enum to track queue state instead on an async iterable.
• Keep jobs in an array in the JobQueue instance to consume them differently in FIFO or LIFO mode
• Yield to the macro queue after every job with a sleep(0)
• Incorporate three metrics into the JobQueue (only if requested)
  • ${prefix}_length: Count of total queue length
  • ${prefix}_dropped_jobs_total: Count of total dropped jobs
  • ${prefix}_job_time_seconds: Time (async) to process queue job

The metrics use a new API from prom-client: the .collect() function available in v13. It allows to de-duplicated state from the prometheus client if that state is already tracked by the application. This is used in the queue length metric, and only computed on demand.

[codeclimate]

Code Climate has analyzed commit 883c0e6 and detected 0 issues on this pull request.

View more on Code Climate.

[mpetrunic]

Looks good to me

[wemeetagain]

💯