Make types of Que jobs Prometheus metrics exclusive

[guicassolato]

Redefines the types of Prometheus metrics on Que jobs (que_jobs_scheduled_total) as the following:

Type	Meaning
ready	Number of jobs enqueued and ready to be executed ASAP (never failed, nor got expired)
scheduled	Number of jobs enqueued to be executed some time in the future, but not now (never failed, nor got expired)
finished	Number of jobs that executed with success
failed	Number of jobs that failed at least once, did not run out of attempts to retry and therefore are scheduled for retry any time soon
expired	Number of jobs that failed and ran out of attempts to retry, therefore won't be retried again

Considering current implementation in Zync, scheduled and finished are expected to be always zero. The former because Zync does not make any job to "wait until" except when handling retries of failed jobs. The latter because Zync only keeps in the database the executed jobs that fail; whenever succeeded, they are deleted.

que_jobs_scheduled_total{type="scheduled"} does not correspond to Que's definition of "scheduled" (as showed in the Que's web console). Que defines "scheduled" as virtually a synonym of "enqueued". Every jobs scheduled to be executed, now or some time in the future, will be listed as "scheduled" by Que. In fact, even "expired" jobs may be listed as "scheduled" by Que. Whereas for the metric, we do a clear distinction between "ready" (now), "scheduled" (to some time in the future) and "expired", with unambiguous meaning to each one of these terms.

Moreover, ready is also expected to be zero most of time. If it's not zero, it's because Zync if nor processing the jobs or not processing them fast enough.

---

Closes THREESCALE-5460

[raelga]

Solves https://github.com/3scale/platform/issues/230

[guicassolato]

Test in OpenShift

[scaled zync-que down to 0]

oc exec -it $(oc get pods | grep zync-que | grep Running | awk '{ print $1 }') -- bash -c 'curl http://localhost:9394/metrics | grep que_jobs_scheduled_total' | grep 'UpdateJob'
que_jobs_scheduled_total{job="UpdateJob"} 0
que_jobs_scheduled_total{job="UpdateJob",type="ready"} 0
que_jobs_scheduled_total{job="UpdateJob",type="scheduled"} 0
que_jobs_scheduled_total{job="UpdateJob",type="finished"} 0
que_jobs_scheduled_total{job="UpdateJob",type="failed"} 0
que_jobs_scheduled_total{job="UpdateJob",type="expired"} 0

oc exec -it $(oc get pods | grep zync-database | grep Running | awk '{ print $1 }') -- bash -c 'psql -U zync zync_production'

select id, run_at from que_jobs;
 id |           run_at
----+-----------------------------------
 76 | 2020-06-25 12:47:22.409502+00

type=scheduled

update que_jobs set run_at='2020-06-26 12:47:22.409502+00' where id=76;

[scaled zync-que up to 1]

oc exec -it $(oc get pods | grep zync-que | grep Running | awk '{ print $1 }') -- bash -c 'curl http://localhost:9394/metrics | grep que_jobs_scheduled_total' | grep 'UpdateJob'
que_jobs_scheduled_total{job="UpdateJob"} 1
que_jobs_scheduled_total{job="UpdateJob",type="ready"} 0
que_jobs_scheduled_total{job="UpdateJob",type="scheduled"} 1
que_jobs_scheduled_total{job="UpdateJob",type="finished"} 0
que_jobs_scheduled_total{job="UpdateJob",type="failed"} 0
que_jobs_scheduled_total{job="UpdateJob",type="expired"} 0

type=failed

update que_jobs set error_count=1 where id=76;

oc exec -it $(oc get pods | grep zync-que | grep Running | awk '{ print $1 }') -- bash -c 'curl http://localhost:9394/metrics | grep que_jobs_scheduled_total' | grep 'UpdateJob'
que_jobs_scheduled_total{job="UpdateJob"} 1
que_jobs_scheduled_total{job="UpdateJob",type="ready"} 0
que_jobs_scheduled_total{job="UpdateJob",type="scheduled"} 0
que_jobs_scheduled_total{job="UpdateJob",type="finished"} 0
que_jobs_scheduled_total{job="UpdateJob",type="failed"} 1
que_jobs_scheduled_total{job="UpdateJob",type="expired"} 0

type=expired

update que_jobs set expired_at=now() where id=76;

oc exec -it $(oc get pods | grep zync-que | grep Running | awk '{ print $1 }') -- bash -c 'curl http://localhost:9394/metrics | grep que_jobs_scheduled_total' | grep 'UpdateJob'
que_jobs_scheduled_total{job="UpdateJob"} 1
que_jobs_scheduled_total{job="UpdateJob",type="ready"} 0
que_jobs_scheduled_total{job="UpdateJob",type="scheduled"} 0
que_jobs_scheduled_total{job="UpdateJob",type="finished"} 0
que_jobs_scheduled_total{job="UpdateJob",type="failed"} 0
que_jobs_scheduled_total{job="UpdateJob",type="expired"} 1

type=finished

update que_jobs set expired_at=null, error_count=0, finished_at=now() where id=76;

oc exec -it $(oc get pods | grep zync-que | grep Running | awk '{ print $1 }') -- bash -c 'curl http://localhost:9394/metrics | grep que_jobs_scheduled_total' | grep 'UpdateJob'
que_jobs_scheduled_total{job="UpdateJob"} 1
que_jobs_scheduled_total{job="UpdateJob",type="ready"} 0
que_jobs_scheduled_total{job="UpdateJob",type="scheduled"} 0
que_jobs_scheduled_total{job="UpdateJob",type="finished"} 1
que_jobs_scheduled_total{job="UpdateJob",type="failed"} 0
que_jobs_scheduled_total{job="UpdateJob",type="expired"} 0

[raelga]

Thanks for solving this that fast!