[env] Don't run next assess when failNow is called

[Fricounet]

What type of PR is this?

/kind bug

What this PR does / why we need it:

When writing tests, this allows the writer to control whether the test should continue running the next assess when a failure occurs. Basically, they can chose between using t.FailNow() or t.Fail() depending on if the failure should stop the test or not. It works independently from the -fail-fast flag specific to the framework and allows a more granular control on test failures.

Which issue(s) this PR fixes:

Fixes #386

Special notes for your reviewer:

I tried to add a new test for this case but couldn't find a way to make it work. Since failNow is called, the test fails and I couldn't find a way to expect a test to fail in go 😅. Happy to add one if someone has an idea on how to do it though.

Code to reproduce

You can use the following code to illustrate what I'm trying to solve

package envfuncs_test

import (
	"context"
	"os"
	"testing"

	"sigs.k8s.io/e2e-framework/pkg/env"
	"sigs.k8s.io/e2e-framework/pkg/envconf"
	"sigs.k8s.io/e2e-framework/pkg/features"
)

var testenv env.Environment

func TestMain(m *testing.M) {
	testenv = env.New()
	os.Exit(testenv.Run(m))
}

func TestFailNow(t *testing.T) {
	feat1 := features.New("fail now").
		Assess("Assess 1", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
			t.Log("Assess 1 (should be printed)")
			t.FailNow()
			return ctx
		}).
		Assess("Assess 2", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
			t.Log("Assess 2 (should NOT be printed)")
			return ctx
		}).
		Feature()

	feat2 := features.New("succeed").
		Assess("Assess 1", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
			t.Log("Assess 1 (should be printed)")
			return ctx
		}).
		Assess("Assess 2", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
			t.Log("Assess 2 (should be printed)")
			return ctx
		}).
		Feature()

	testenv.Test(t, feat1, feat2)
}

func TestFail(t *testing.T) {
	feat := features.New("fail (not now)").
		Assess("Assess 1", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
			t.Log("Assess 1 (should be printed)")
			t.Fail()
			return ctx
		}).
		Assess("Assess 2", func(ctx context.Context, t *testing.T, cfg *envconf.Config) context.Context {
			t.Log("Assess 2 (should be printed)")
			return ctx
		}).
		Feature()

	testenv.Test(t, feat)
}

• Before the changes, here is the output:

❯ go test ./pkg/envfuncs -test.v
=== RUN   TestFailNow
=== RUN   TestFailNow/fail_now
=== RUN   TestFailNow/fail_now/Assess_1
    ns_funcs_test.go:39: Assess 1 (should be printed)
=== RUN   TestFailNow/fail_now/Assess_2
    ns_funcs_test.go:44: Assess 2 (should NOT be printed)
=== RUN   TestFailNow/succeed
=== RUN   TestFailNow/succeed/Assess_1
    ns_funcs_test.go:51: Assess 1 (should be printed)
=== RUN   TestFailNow/succeed/Assess_2
    ns_funcs_test.go:55: Assess 2 (should be printed)
--- FAIL: TestFailNow (0.00s)
    --- FAIL: TestFailNow/fail_now (0.00s)
        --- FAIL: TestFailNow/fail_now/Assess_1 (0.00s)
        --- PASS: TestFailNow/fail_now/Assess_2 (0.00s)
    --- PASS: TestFailNow/succeed (0.00s)
        --- PASS: TestFailNow/succeed/Assess_1 (0.00s)
        --- PASS: TestFailNow/succeed/Assess_2 (0.00s)
=== RUN   TestFail
=== RUN   TestFail/fail_(not_now)
=== RUN   TestFail/fail_(not_now)/Assess_1
    ns_funcs_test.go:66: Assess 1 (should be printed)
=== RUN   TestFail/fail_(not_now)/Assess_2
    ns_funcs_test.go:71: Assess 2 (should be printed)
--- FAIL: TestFail (0.00s)
    --- FAIL: TestFail/fail_(not_now) (0.00s)
        --- FAIL: TestFail/fail_(not_now)/Assess_1 (0.00s)
        --- PASS: TestFail/fail_(not_now)/Assess_2 (0.00s)
FAIL
FAIL	sigs.k8s.io/e2e-framework/pkg/envfuncs	0.008s
FAIL

• Now it is:

❯ go test ./pkg/envfuncs -test.v
=== RUN   TestFailNow
=== RUN   TestFailNow/fail_now
=== RUN   TestFailNow/fail_now/Assess_1
    ns_funcs_test.go:153: Assess 1 (should be printed)
=== RUN   TestFailNow/succeed
=== RUN   TestFailNow/succeed/Assess_1
    ns_funcs_test.go:165: Assess 1 (should be printed)
=== RUN   TestFailNow/succeed/Assess_2
    ns_funcs_test.go:169: Assess 2 (should be printed)
--- FAIL: TestFailNow (0.00s)
    --- FAIL: TestFailNow/fail_now (0.00s)
        --- FAIL: TestFailNow/fail_now/Assess_1 (0.00s)
    --- PASS: TestFailNow/succeed (0.00s)
        --- PASS: TestFailNow/succeed/Assess_1 (0.00s)
        --- PASS: TestFailNow/succeed/Assess_2 (0.00s)
=== RUN   TestFail
=== RUN   TestFail/fail_(not_now)
=== RUN   TestFail/fail_(not_now)/Assess_1
    ns_funcs_test.go:180: Assess 1 (should be printed)
=== RUN   TestFail/fail_(not_now)/Assess_2
    ns_funcs_test.go:185: Assess 2 (should be printed)
--- FAIL: TestFail (0.00s)
    --- FAIL: TestFail/fail_(not_now) (0.00s)
        --- FAIL: TestFail/fail_(not_now)/Assess_1 (0.00s)
        --- PASS: TestFail/fail_(not_now)/Assess_2 (0.00s)
FAIL
FAIL	sigs.k8s.io/e2e-framework/pkg/envfuncs	0.013s
FAIL

Does this PR introduce a user-facing change?

Stopped running the following assess if FailNow() was called during a test

Additional documentation e.g., Usage docs, etc.:

[k8s-ci-robot]

Hi @Fricounet. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Fricounet]

cc @harshanarayana

[cpanato]

/ok-to-test

[vladimirvivien]

Thank you for this change. Will let @harshanarayana take a look.
/lgtm

[harshanarayana]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Fricounet, harshanarayana

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [harshanarayana]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[harshanarayana]

/retest