New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unsupported -a in fishtape 2.0.0 #40

Closed
matchai opened this Issue Feb 8, 2019 · 7 comments

Comments

Projects
None yet
2 participants
@matchai
Copy link

matchai commented Feb 8, 2019

Great work with the release of fishtape 2.0.0!

Unfortunately, the new test system doesn't allow for multiple assertions in one test, so the test setup would have to be duplicated for every test case.

E.g:

@test "It should not mock blacklisted elements" (
	mock eval \*
) = "The function \"eval\" is reserved and therefore cannot be mocked."

@test "It should not mock blacklisted elements" (
	mock eval \*
) $status -eq 1

instead of:

@test "It should not mock blacklisted elements" (
	mock eval \*
) = "The function \"eval\" is reserved and therefore cannot be mocked." -a $status -eq 1

In this example, the test setup is short, but a fair bit of duplication would need to be added tests where setup is more complex, without a way to have multiple assertions.

@jorgebucaran

This comment has been minimized.

Copy link
Owner

jorgebucaran commented Feb 8, 2019

@matchai Funny story. Fishtape 1 didn't have this either. Your tests will pass even if the next condition fails.

@matchai

This comment has been minimized.

Copy link
Author

matchai commented Feb 8, 2019

@jorgebucaran Oh... That's a shock. 😳
I guess I'll be quickly updating my tests in that case.

@jorgebucaran

This comment has been minimized.

Copy link
Owner

jorgebucaran commented Feb 8, 2019

@matchai What happens if the expression fails—how do we report that with TAP?


Here is an alternative. Print the $status right after running mock:

@test "It should not mock blacklisted elements" (
    echo (
        mock eval \*
        echo $status
    )
) = "The function \"eval\" is reserved and therefore cannot be mocked. 0"

We need to wrap mock and echo $status in another echo to remove new lines.

Here is another idea: If you check the exit status often, create a utility function to echo stdout if $staus is 0.

function only_if_ok
    and echo "$argv"; or echo
end

@test "It should not mock blacklisted elements" (
    only_if_ok (mock eval \*)
) = "The function \"eval\" is reserved and therefore cannot be mocked."
@matchai

This comment has been minimized.

Copy link
Author

matchai commented Feb 8, 2019

What happens if the expression fails—how do we report that with TAP?

I think the ideal thing would be to support subtests. Though they have never been standardized in TAP, they're commonly used. (TestAnything/Specification#2)


Both solid suggestions! I'll try them out. 👍

@jorgebucaran

This comment has been minimized.

Copy link
Owner

jorgebucaran commented Feb 9, 2019

@matchai I disagree.

I'm not opposed to subtests (see below), but I am positive that using conditional operators (-a or -o) to describe them would be the wrong tool for the job.

Now, how do we describe subtests using our limited syntax (jorgebucaran/fishtape#34)?

Here's an idea that doesn't rely on syntax:

fishtape parent_test.fish
# Parent
    # Child
        # Grandchild
            ok 1 - some foo test
            ok 2 - some bar test
            not ok 3 - some baz test
            not ok 3 - some baz test
              ---
                operator: =
                expected: bar
                actual:   baz
              ...
            1..3
            # tests 3
            # pass  2
            # fail  1
        ok 1 - grandchild_test.fish
        1..1
        # tests 1
        # fail  1
    ok 1 - child_test.fish
    1..1
    # tests 1
    # fail  1
ok 1 - parent_test.fish
1..1
# tests 1
# fail  1
parent_test.fish
@mesg "Parent"
fishtape child_test.fish
child_test.fish
@mesg "Child"
fishtape granchild_test.fish
grandchild_test.fish
@mesg "Grandchild"
test "some foo test" foo = foo
test "some bar test" bar = bar
test "some baz test" bar = baz

Closing as I won't be implementing conditional operators at this time, but we can discuss subtests in a separate issue.

@matchai

This comment has been minimized.

Copy link
Author

matchai commented Feb 9, 2019

I am positive that using conditional operators (-a or -o) to describe them would be the wrong tool for the job

Oh yeah, absolutely. I was just thinking that having multiple test cases share the same setup would be best done with subtests. If it would be possible for tests to inherit their parents' setup, that would make -a or -o unnecessary.

@jorgebucaran

This comment has been minimized.

Copy link
Owner

jorgebucaran commented Feb 9, 2019

@matchai Ah, I see what you mean then. Glad we're clear about that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment