Test gating status should link to which tests were ignored

[tyll]

On https://bodhi.fedoraproject.org/updates/FEDORA-2018-77214e2c70 Bodhi says that tests were ignored in green but it is unclear which tests were ignored. Would all failures block the gating or just some? It would be great if one could see if the update would be pushed without ignoring the tests. Also I guess green is a bad color for ignoring something since it should be a warning IMHO. Maybe yellow or orange would be better (especially when failed tests were ignored).

[bowlofeggs]

The ignored state actually means that the Greenwave policy is to ignore test results, meaning that no matter what it is going to tell Bodhi to let the update go to stable. This is why we went with green, as the color indicates that the update can go to stable. Greenwave is currently configured this way due to the difficulties with the gating UI as discussed in https://pagure.io/fesco/issue/1872.

I personally feel that the color and text are OK, but perhaps we could use some hover text to help explain it. Additionally, we do need documentation about this. Would you be amenable to that as a solution?

[tyll]

I am not completely convinced. For me green implies that everything is good, but if tests failed and it was ignored, then something is broken and IMHO the UI should indicate this and allow to diagnose what is going on by linking to the test results that failed and were ignored. Also when we do not want to ignore all tests results anymore it would be good to know beforehand if updates would fail then.

[bowlofeggs]

The "tests ignored" state doesn't mean that tests failed, it actually means that the tests were not considered in the gating decision. When an update says "tests ignored", it's really that the Greenwave policy is to allow the update to stable no matter what the state of its tests was. I guess I could see an argument for making it orange/yellow as a cautionary message, as the human is the only gating mechanism when this happens. What do you think?

[tyll]

I would prefer orange/yellow. Nevertheless, is there a way to check if there are any failed tests that would have prevented a push in case the Greenwave policy was set to enforcing?

[bowlofeggs]

The Greenwave policy is just a list of tests that must pass, and "test ignored" means there were no tests in its policy. Thus, it's not possible to know which tests those would be, if that makes sense.

I'm not opposed to changing the color for tests ignored to orange/yellow.

[sunn-e]

Is this still open? I might give it a try.Will that be fine.

[bowlofeggs]

@sunn-e sure

[AdamWill]

these days, the text you see in the UI is "no tests are required" (well, in the right bar; you'll still see a note in the stream on the left side that the test gating status is "ignored"). The check mark next to this text is green.

I don't think I've heard anyone else complain since that this is confusing. Do we need to do anything here, or could we close this?