SCM/CI: Report pending status for multibuilds

[perlpunk]

We are about to use the SCM/CI for openQA projects.
We are using multibuilds, and it is a documented limitation, that multibuilds don't report a pending status, but only report the final fail or success.
https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.scm_ci_workflow_integration.html#sec.obs.obs_scm_ci_workflow_integration.setup.status_reporting

This can be problematic because when looking at a PR, one doesn't even know if there are still running checks, and we have a lot of checks.
A PR might get merged because every check passes.
We also use mergify to automatically merge in certain conditions, and we have to manually add every OBS build/platform to the configuration to be sure that it doesn't merge before those checks are reported.

So it would be nice if the pending status could also be reported.

[dmarcoux]

This isn't true anymore and the documentation needs to be updated. The build status of multibuild flavors is reported. Thank you for reporting this @perlpunk.

[perlpunk]

@dmarcoux oh, since when does that work?
It wasn't working yet last friday:
perlpunk/os-autoinst#12
You can see the checks are all reported, but they only appeared when they were finished, not during the running check.

[dmarcoux]

Hum... I'll need to double-check, but I'm pretty sure the initial report is working.

[dmarcoux]

It's not working somehow, we need to investigate. Thank you again for reporting this.

[vpereira]

Tested locally on master and I couldn't reproduce it.

[perlpunk]

@vpereira I will try again right now and create a PR in https://github.com/os-autoinst/os-autoinst/pulls and ping you directly after it. maybe you can see it

[dmarcoux]

@perlpunk I had a look at the PR and the initial report with the "pending" status weren't reported. It is the same that I experienced in a test I did.

[okurz]

So seems like there seems to a little difference in what you guys, @dmarcoux and @vpereira did? I read that at least there is the difference between "what is deployed+configured to build.opensuse.org" and "locally on master", right?

[vpereira]

@vpereira I will try again right now and create a PR in https://github.com/os-autoinst/os-autoinst/pulls and ping you directly after it. maybe you can see it

As we talked privately, our issue is that we cannot reproduce it in development, but as @dmarcoux said he is able to reproduce in production. We are still investigating the issue.

[dmarcoux]

This was closed by mistake, there's still more to do for this. Reopening.

[perlpunk]

One more thing, I don't know if it's related: builds that are unresolvable are also not reported.
Example: os-autoinst/openQA#4494
I would expect them to be reported as failures.

[dmarcoux]

Yes, this is sadly not implemented. We're tracking this in https://trello.com/c/0N3jHq5M/2257-report-back-to-scm-when-build-results-arent-failed-and-succeeded

[perlpunk]

ok thanks

[JanZerebecki]

This will be important for ALP/Git.

[hennevogel]

As we are tracking this in our backlog, let's close this issue here.