Add quarantine mechanism to test suite

[nebril]

There is a new cli options for our test suite - cilium.runQuarantined, which tells ginkgo whether to skip tests which are marked as quarantined.

Test is marked as being under quarantine when its It(... call is replaced by SkipItIf(helpers.SkipQuarantined, ...

[maintainer-s-little-helper]

Please set the appropriate release note label.

[nebril]

test-runtime

[nebril]

test-4.9

[coveralls]

Coverage increased (+0.03%) to 37.064% when pulling 1131776 on pr/quarantine-jobs into f3b474a on master.

[pchaigno]

Is that for flakes? Just to make it easier to see which tests are disabled compared to PIt, PDescribe, etc.?

It would be nice to have the same for Context and Describe as we sometimes disable whole contexts.

[nebril]

@pchaigno this is for flakes, but also to allow running flaky tests within "quarantine" pipelines. Whole contexts will be skipped by using SkipContextIf, similarly to how It blocks became quarantined in this change.

[pchaigno]

this is for flakes, but also to allow running flaky tests within "quarantine" pipelines.

Ah, nice!

Whole contexts will be skipped by using SkipContextIf, similarly to how It blocks became quarantined in this change.

Of course 🤦 I read the code a bit too quickly.

[aanm]

@nebril I assume this is supposed to be backported?

[nebril]

test-runtime

[nebril]

test-4.9

[qmonnet]

Note: We could maybe quarantine the check disabled in this commit for Linux 4.19, but I can do that in a follow-up if necessary.

[nebril]

@qmonnet interesting case, so you're saying that it should be quarantined only on 4.19? It should be easy enough to do.

[brb]

Don't we need to update other Jenkinsfile as well? Also (unrelated to your PR), but can we somehow share some sections among all Jenkinsfiles (e.g. ENV vars)?

[nebril]

@brb this is only for running https://jenkins.cilium.io/view/Quarantine%20Pipelines/ which for now only use this jenkinsfile.