Introduce the Test.expectFailure function

[m-Peter]

Closes onflow/cadence-tools#108

Description

Introduces a new native function for allowing/expecting a test case to fail/panic, with a specified error message.

---

• Targeted PR against master branch
• Linked to Github issue with discussion and accepted design OR link to spec that describes this work
• Code follows the standards mentioned here
• Updated relevant documentation
• Re-reviewed Files changed in the Github PR explorer
• Added appropriate labels

[codecov]

Codecov Report

Merging #2468 (9eb2fac) into master (c6117d1) will increase coverage by 0.01%.
The diff coverage is 93.58%.

@@            Coverage Diff             @@
##           master    #2468      +/-   ##
==========================================
+ Coverage   78.28%   78.30%   +0.01%     
==========================================
  Files         327      327              
  Lines       72773    72851      +78     
==========================================
+ Hits        56971    57044      +73     
- Misses      13707    13710       +3     
- Partials     2095     2097       +2     

Flag	Coverage Δ
unittests	78.30% <93.58%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
runtime/stdlib/test_contract.go	84.46% <93.58%> (+0.93%)	⬆️

[m-Peter]

I will certainly add more tests, I just opened this up in order to bootstrap a conversation around the implementation details and desired functionality 🙏

[turbolent]

Looks great! 👍

[m-Peter]

@turbolent I have added more test cases, and fixed the scenario where a failure is expected, but there is none (946ffae#diff-4709f76365b48c2caca87b859afcc573afb62cb4e6186b9431d2b540972d402fR619)
However, I'm struggling with the test case here (946ffae#diff-74ad93a1435e307865ff8474567598f410879038788126ec263e27c9244edf98R1487). With a local build, the actual tests (in Cadence Testing Framework), works well. Do you have any clue?
P.S: This PR should be merged after #2403, to avoid huge conflicts 😇

[m-Peter]

import Test

pub struct Foo {
    priv let answer: UInt8
    
    init(answer: UInt8) {
        self.answer = answer
    }
    
    pub fun correctAnswer(_ input: UInt8): Bool {
        if self.answer != input {
            panic("wrong answer!")
        }
        return true
    }
}

pub fun testOne() {
    let foo = Foo(answer: 42)
    Test.expectFailure(fun(): Void {
        foo.correctAnswer(42)
    }, errorMessageSubstring: "wrong answer")
}

pub fun testTwo() {
    let foo = Foo(answer: 42)
    Test.expectFailure(fun(): Void {
        foo.correctAnswer(43)
    }, errorMessageSubstring: "wrong answer!")
}

pub fun testThree() {
    let foo = Foo(answer: 42)
    Test.expectFailure(fun(): Void {
        foo.correctAnswer(43)
    }, errorMessageSubstring: "what is wrong?!")
}

And the equivalent test execution results:

~/Dev/forks/flow-cli/cmd/flow/flow-x86_64-linux- test my_expect_failure_tests.cdc

Test results: "my_expect_failure_tests.cdc"
- FAIL: testOne
		Execution failed:
			error: Execution failed:
			error: Expected a failure, but found none.
			  --> 7465737400000000000000000000000000000000000000000000000000000000:14:8
			
			  --> 7465737400000000000000000000000000000000000000000000000000000000:14:8
			
- PASS: testTwo
- FAIL: testThree
		Execution failed:
			error: Expected error message to include: "what is wrong?!". Found: Execution failed:
			error: panic: wrong answer!
			  --> 7465737400000000000000000000000000000000000000000000000000000000:12:12
			
			  --> 7465737400000000000000000000000000000000000000000000000000000000:12:12

[turbolent]

@m-Peter figured out what's wrong and also refactored the recover logic, but can't push the commits.

Could you please allow commits from maintainers by checking the "Allow edits from maintainers" box? (see https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork)

[m-Peter]

@turbolent It seems that it's not possible for organization repositories.. See https://github.com/orgs/community/discussions/5634 I am unable to find this option also..

[m-Peter]

@turbolent I have sent you an invitation as a collaborator, you should be able to do so now 🙏

[turbolent]

@m-Peter Thank you! Pushed up the fix. The issue was that the panic function was declared for the checker, but not for the interpreter

[turbolent]

Nice! Thanks for adding this

[m-Peter]

@m-Peter Thank you! Pushed up the fix. The issue was that the panic function was declared for the checker, but not for the interpreter

Oh, I see, great catch 💯 💯 💯

I tried locally the newest changes, but it seems that they affect the displayed error messages. For example:

I have added a commit to make them less verbose:

What do you think about it? I can drop the changes, if you want.
Otherwise, feel free to merge it, and I will later resolve the conflicts for the other PR (#2403) 🙏

[turbolent]

Looks better, nice!

[turbolent]

@m-Peter #2403 is now merged, so there are some conflicts now. The additions of this PR basically have to be moved to test_contract.go

[m-Peter]

@turbolent It was rather impossible to rebase on master, after #2403 was merged. So I had to reset the commits of this PR, manually apply the changes and then force-push. I have added you as co-author though 🙏

[turbolent]

Nice!