Make it easier to handle expected failures

[blampe]

I have a situation where I expect an update to fail and I'd like to assert that it does. Unfortunately errors are always treated as fatal:

providertest/pulumitest/up.go

Lines 19 to 22 in 385bd7f

	result, err := a.currentStack.Up(a.ctx, opts...)
	if err != nil {
	a.fatalf("failed to deploy: %s", err)
	}

It would be really nice if we returned result, err here instead, although that would be a breaking change.

We should at least be able to have an option ExpectFailure and then use that to inform the t.fatal? Although that will need to update the auto API's options.

// Ideal
up, err := test.Up()
assert.Error(t, err)

// Also OK
up := test.Up(optup.ExpectFailure())

@VenelinMartinov points out a workaround: up, err := test.CurrentStack().Up(ctx).

[EronWright]

Suggestion: add an ExpectFailure flag to the Up options, and fail/succeed accordingly.

[danielrbradley]

Returning result, err would reduce the clarity of tests as they'd have to now contain error handling for every step. This would be equivilent to just using the AutomationAPI directly without the test framework.

Another option instead of disabling the assert would be to present an alternative method for operations we expect might fail - perhaps something like upResult, err := test.TryUp(). I'm not sure if Try... is a usual convention in Go, perhaps there's another ideom that would do the same job.