Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to specify minimum number of executed assertions #690

Closed
danielo515 opened this issue Mar 15, 2017 · 10 comments
Assignees
Labels
Milestone

Comments

@danielo515
Copy link

@danielo515 danielo515 commented Mar 15, 2017

Hello,

Just like you can specify the minimum code coverage percentage, I think it will be very useful to specify the minimum number of assertions to be executed per test. It have happened to me that one colleague wrote several tests, with 100% code coverage, but there were a couple of tests where 0 assertions were executed. For me this is a bug on the test, and it should be flagged as failed.

When does this happen? In a situation like this

// Test shortcuts
const lab = exports.lab = Lab.script();
const describe = lab.experiment;
const it = lab.test;
const expect = Code.expect;

describe('********** Example suite **********', () => {

    describe('----- Example test -----', () => {

        it('Should create an instance of a User', (done) => {

            thisReturnsAPromise() 
                        .then(()=>done()) // instead of the expected rejection, we got a valid response
                       .catch( err => { expect(err).to.be.an.insanceOf(Error); done()} )
        });
    });
});

The done is called to avoid timeouts.
The test pass, and they cover certain lines, but it happens that no assertion was executed. This is obviously a mistake, but that is the point of the tests, to catch up errors. If I can set a minimum of assertions to be executed to 2, that mistake would be evident.

If this is already available, there is no documentation about it.

@danielo515

This comment has been minimized.

Copy link
Author

@danielo515 danielo515 commented Mar 15, 2017

@Marsup

This comment has been minimized.

Copy link
Member

@Marsup Marsup commented Mar 15, 2017

You already have plan which is explained in the docs. I can't think of any reason to allow "minimum" of expectations, as your test should be predictable.

@danielo515

This comment has been minimized.

Copy link
Author

@danielo515 danielo515 commented Mar 15, 2017

Hello @Marsup ,

I didn't noticed the plan option, it is not very prominent on the docs. However, it works on a per test basis, I would like to specify it globally.
What do you mean with "your test should be predictable" ? I just want to specify a minimum amount of assertions to avoid lazy programmers to just do the basic tests.

@Marsup

This comment has been minimized.

Copy link
Member

@Marsup Marsup commented Mar 15, 2017

OK I misunderstood your goal on this feature, if it's to enforce best practices, I'm not sure whether lab should take care of it or an eslint rule, both would work. I'll leave the decision to @geek :)

@devinivy

This comment has been minimized.

Copy link
Member

@devinivy devinivy commented Mar 15, 2017

You can ask Code for the total number of assertions it has made by calling Code.count(). Perhaps you could simply add a test for this at the end of your test suite.

@danielo515

This comment has been minimized.

Copy link
Author

@danielo515 danielo515 commented Mar 15, 2017

You can ask Code for the total number of assertions it has made by calling Code.count(). Perhaps you could simply add a test for this at the end of your test suite.

I would consider that a workaround.
I want to enforce this policies from OUTSIDE the code of the tests. That way the developers does not need to take care of this, while I can force it from the outside, without modifying the code at all. I can run the same test code (the command line that executes the tests) for all the projects and be sure that such policies are met.

@geek geek added the request label Mar 16, 2017
@danielo515

This comment has been minimized.

Copy link
Author

@danielo515 danielo515 commented Mar 22, 2017

Dear @geek do you have any idea in mind about how to implement this?

@geek

This comment has been minimized.

Copy link
Member

@geek geek commented Mar 22, 2017

@danielo515 I was thinking of a global rule that states you must have at least 1 assertion in every test. This would add a check if the test doesn't have a plan to verify that it has at least 1 assertion, likely close to where the plan checks exist.

@geek geek closed this in 2380bcf Mar 12, 2018
geek added a commit that referenced this issue Mar 12, 2018
Implement minimum assertions threshold. Closes #690.
@geek geek added feature and removed request labels Mar 12, 2018
@geek geek added this to the 15.4.0 milestone Mar 12, 2018
@danielo515

This comment has been minimized.

Copy link
Author

@danielo515 danielo515 commented Mar 12, 2018

Awesome! May this be back ported to 14.x?

@geek

This comment has been minimized.

Copy link
Member

@geek geek commented Mar 12, 2018

@danielo515 I'm happy to review/merge/release this feature for 14.x if someone wants to do the work of backporting it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.