@coqbot could report failures from GitLab CI and restart spurious ones

[Zimmi48]

When a GitLab CI pipeline completes with failures, first check if the pipeline is up-to-date with respect to the PR.

If yes, check if some of the failures are spurious:

• "runner system failure" detected by GitLab, example error message: ERROR: Job failed (system failure): Cannot connect to the Docker daemon at tcp://10.142.0.123:2376. Is the docker daemon running?
• connection trouble:

  error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500 Internal Server Error
  fatal: The remote end hung up unexpectedly
  ERROR: Job failed: exit code 1
  

• uploading artifacts failed:

  Uploading artifacts to coordinator... ok            id=71082452 responseStatus=201 Created token=G7Azf-fN
  ERROR: Job failed: exit code 1
  

If yes, restart the corresponding jobs.
(For the spurious failures we have control upon, we should fix them instead.)

Otherwise, post a message in the PR thread with the last few lines of the failing job logs and direct links to these logs.

[Zimmi48]

This is going to be way easier than I thought thanks to the job webhook, the build trace API endpoint and the build retry API endpoint (cf. bf62f4b and https://docs.gitlab.com/ee/api/jobs.html#retry-a-job).

[Zimmi48]

Actually something is missing from the webhook load or the trace to be able to tell whether the failure is due to a failing runner. Cf. https://gitlab.com/gitlab-org/gitlab-ee/issues/6408

[Zimmi48]

Another unrelated problem to put this in practice would be to stop relying on Heroku's free dynos as GitLab job webhook generates way too many requests to let the bot have the 7 hours of statutory sleep.

[ejgallego]

Please disable the report functionality until the "stale build problem" is fixed as detailed in coq/coq#7871 (comment)

[Zimmi48]

OK, this is fixed now.

[ejgallego]

Thanks!!!

[Zimmi48]

This is basically implemented now and further enhancements can be treated in separate issues.

[ejgallego]

Thanks for this great work.