🌱 Allow overriding gingko args in e2e tests

[killianmuldoon]

Allow overriding the value of GINKGO_ARGS in the CI test script. This will allow us to disable fail-fast and give us full runs of e2e tests in the CI.

/area testing

[killianmuldoon]

/hold

Will keep this until after the 1.4 release to ensure we don't interfere with the signal for now.

[fabriziopandini]

/lgtm

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 8237d7a0e6167c72c90b8fc25060f98ebd7ae162

[oscr]

/lgtm

[killianmuldoon]

/hold remove

[killianmuldoon]

/hold cancel

[killianmuldoon]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: killianmuldoon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [killianmuldoon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sbueringer]

@killianmuldoon looks like something we should cherry-pick? Was there a corresponding test infra pr?

[killianmuldoon]

@killianmuldoon looks like something we should cherry-pick? Was there a corresponding test infra pr?

We didn't cherry-pick as this was pretty much an experiment to see how things would go. There's a PR to implement the "fail fast" behaviour here: kubernetes/test-infra#29135

I'm not sure yet if we want to extend this to other test runs, but cherry-picking doesn't hurt IMO.

[sbueringer]

What I like about --fail-fast false is that we get better and more data. (with --fail-fast we obviously don't run all tests, but also we always get 2 additional tests flagged as failing even though they didn't fail and it's a bit more effort to find the actual failing test in the Prow UI).

I would be in favor of using --fail-fast false everywhere. If we use it everywhere, should we simply set the default to false in the repo (+ release branches) and revert the test-infra PR?

[killianmuldoon]

I would be in favor of using --fail-fast false everywhere

I'm fine with this, but limited to periodic jobs. I think it makes sense to get the fail-fast behaviour for PRs - WDYT?

[sbueringer]

I would be in favor of using --fail-fast false everywhere

I'm fine with this, but limited to periodic jobs. I think it makes sense to get the fail-fast behaviour for PRs - WDYT?

Good question. Depends on if we prefer to get all errors with one run or if we want to fix one after another.

I think on PRs this only really affects e2e-full. All the others run only 1 test case or 2 in parallel (or something similar).

I think I would prefer to keeping it everywhere the same (and thus simple). I.e. I would stop using fail-fast everywhere.

It's still possible to see test failures earlier in e2e-full jobs if someone cares about that by looking at the logs as they are "livestreamed" to the Prow UI.

[killianmuldoon]

Makes sense to me - I'll open a WIP PR to do this and track it. Let's continue any converstaion over there - #8548