time_since_last_job_run and last_job_run_duration updated only when the status of job is changing

[mkrvanton]

Hi!
Thank you so much for your pipeline-exporter project it is awesome! :)
After some short research, I was found one issue as I think...
I checked it on 0.3.2 version.
As I guess gitlab_ci_pipeline_time_since_last_job_run_seconds and gitlab_ci_pipeline_last_job_run_duration_seconds updated only when the status of job is changing.
Is it a bug or a feature?

12:52
gitlab_ci_pipeline_time_since_last_job_run_seconds{job_name="tests",kind="branch",project="test/test-alm",ref="custom-field-type-Table",stage="selenium-tests",topics="",variables=""} 8530
gitlab_ci_pipeline_last_job_run_duration_seconds{job_name="tests",kind="branch",project="test/test-alm",ref="custom-field-type-Table",stage="selenium-tests",topics="",variables=""} 962.16907283
gitlab_ci_pipeline_last_job_run_status{job_name="tests",kind="branch",project="test/test-alm",ref="custom-field-type-Table",stage="selenium-tests",status="running",topics="",variables=""} 1

12:59
gitlab_ci_pipeline_time_since_last_job_run_seconds{job_name="tests",kind="branch",project="test/test-alm",ref="custom-field-type-Table",stage="selenium-tests",topics="",variables=""} 8530
gitlab_ci_pipeline_last_job_run_duration_seconds{job_name="tests",kind="branch",project="test/test-alm",ref="custom-field-type-Table",stage="selenium-tests",topics="",variables=""} 962.16907283
gitlab_ci_pipeline_last_job_run_status{job_name="tests",kind="branch",project="test/test-alm",ref="custom-field-type-Table",stage="selenium-tests",status="running",topics="",variables=""} 1

13:05
gitlab_ci_pipeline_time_since_last_job_run_seconds{job_name="tests",kind="branch",project="test/test-alm",ref="custom-field-type-Table",stage="selenium-tests",topics="",variables=""} 9433
gitlab_ci_pipeline_last_job_run_duration_seconds{job_name="tests",kind="branch",project="test/test-alm",ref="custom-field-type-Table",stage="selenium-tests",topics="",variables=""} 1301.050302
gitlab_ci_pipeline_last_job_run_status{job_name="tests",kind="branch",project="test/test-alm",ref="custom-field-type-Table",stage="selenium-tests",status="success",topics="",variables=""} 1

13:15
gitlab_ci_pipeline_time_since_last_job_run_seconds{job_name="tests",kind="branch",project="test/test-alm",ref="custom-field-type-Table",stage="selenium-tests",topics="",variables=""} 9433
gitlab_ci_pipeline_last_job_run_duration_seconds{job_name="tests",kind="branch",project="test/test-alm",ref="custom-field-type-Table",stage="selenium-tests",topics="",variables=""} 1301.050302
gitlab_ci_pipeline_last_job_run_status{job_name="tests",kind="branch",project="test/test-alm",ref="custom-field-type-Table",stage="selenium-tests",status="success",topics="",variables=""} 1

[mvisonneau]

Hi @mkrvanton, indeed, this is a bug related to #106

I fixed through this commit : 5a91ba2

I will release it as part of 0.3.3 now 👍

[mvisonneau]

@mkrvanton can you please try to upgrade to 0.3.3 and let us know if it works properly?

[mkrvanton]

@mkrvanton can you please try to upgrade to 0.3.3 and let us know if it works properly?

gitlab_ci_pipeline_time_since_last_job_run_seconds is updating fine for running jobs, but doesnt update for successful/faled jobs,
gitlab_ci_pipeline_last_job_run_duration_seconds is not updating for running jobs

gitlab_ci_pipeline_last_job_run_status is not updating (stuck in running state) :(

[mvisonneau]

👋 hey @mkrvanton, I have just released 0.3.4 which should be fixing those issues! 🤞

[mkrvanton]

👋 hey @mkrvanton, I have just released 0.3.4 which should be fixing those issues! 🤞

Hey @mvisonneau,
maybe I do something wrong, but it doesn't work for me with 0.3.4
I mean the "gitlab_ci_pipeline_time_since_last_run_seconds" updating, but all things were related to job metrics do not update (gitlab_ci_pipeline_last_job_run_duration_seconds, gitlab_ci_pipeline_last_job_run_status, gitlab_ci_pipeline_time_since_last_job_run_seconds)
Here is some of my logs, but probably it is will be not so helpfull :(

time="2020-08-05T14:43:01+02:00" level=info msg="found pipeline jobs" jobs-count=107 pipeline-id=973 project-id=1 project-ref=Caching-Add-records
time="2020-08-05T14:43:13+02:00" level=info msg="polling metrics from projects refs" total=25
time="2020-08-05T14:43:50+02:00" level=info msg="polling metrics from projects refs" total=25
time="2020-08-05T14:58:54+02:00" level=info msg="refreshed all jobs metrics" jobs-count=0 project-id=1 project-ref=Handle-Deprecation-in-Jira-REST
time="2020-08-05T15:19:30+02:00" level=info msg="refreshed all jobs metrics" jobs-count=0 project-id=1 project-ref=Caching-Add-records````

And one more thing: I am not an expert, but it looks like after some time it is hanging or looping in pollProjectRefJobs function... 
Please let me know if you will have any idea what can be wrong with my environment or how can I debug it,
thank you in advance! :)

[mvisonneau]

can you run it in with --log-level=debug for a few minutes and send me the logs please? 🤔

[mkrvanton]

@mvisonneau I have a logs.. can I send it to you as a private message or by the email?

[DjinNO]

Hi, i have same problem(?), pipeline of my project ran a few seconds ago, but that's what i see:

gitlab_ci_pipeline_time_since_last_run_seconds{kind="branch",project="gitlab/play_with_ci",ref="master",topics="",variables=""} 1.664266e+06

[mvisonneau]

@mvisonneau I have a logs.. can I send it to you as a private message or by the email?

@mkrvanton sure, here is my keybase: https://keybase.io/mvisonneau

[mvisonneau]

Hi, i have same problem(?), pipeline of my project ran a few seconds ago, but that's what i see:

gitlab_ci_pipeline_time_since_last_run_seconds{kind="branch",project="gitlab/play_with_ci",ref="master",topics="",variables=""} 1.664266e+06

@DjinNO I am not sure to understand how it does compare to this issue nor what is wrong with the log line you pasted 🤔

[DjinNO]

Hi, i have same problem(?), pipeline of my project ran a few seconds ago, but that's what i see:
gitlab_ci_pipeline_time_since_last_run_seconds{kind="branch",project="gitlab/play_with_ci",ref="master",topics="",variables=""} 1.664266e+06

@DjinNO I am not sure to understand how it does compare to this issue nor what is wrong with the log line you pasted

Sorry
I will create new issue with more information about my problem

[mvisonneau]

👋 hey @mkrvanton, thanks for the logs and apologies for the late feedback on this!

I managed to reproduce it locally and to sort it out with 48a6778!

it will be part of the next release, in the meantime you can try it out from the source if you want or with the docker.io/mvisonneau/gitlab-ci-pipelines-exporter:latest container image

[mvisonneau]

These changes are part of the 0.4.0 release! Feel free to reopen if necessary @mkrvanton 🙇!