Allow to rerun flaky tests

[tinganho]

Some test suites are very flaky. Flaky means it can generate false negatives for some tests. An example, is if we query an external web service and the network, one in every 100 times the socket hangs up. It would generate a false positive.

My proposal is have an CLI option:

--max-test-fails [number]

that defaults to 0.

Also you can set this for a specific test suite and all its children suites:

describe('SuiteName', function() {
    this.maxTestFails(3);
    ...
})

I have PR ready, if this sounds interesting enough for you. But for me, I cannot use Mocha without it. My test suite generates false negatives in nearly every run.

[rodoabad]

@tinganho are those false positives intended?

[tinganho]

Sorry should be false negative. NOT false positive. The tests are not meant to fail, but sometimes due to network issues they do.

[rodoabad]

@tinganho Why is network connectivity an issue? Can you post a scenario? Are you not mocking/stubbing modules that aren't SUT?

[tinganho]

What's SUT?

I personally don't think stubbing and mocking things are good. It takes long time and doesn't really represent the real object at all the times.

One scenario I have right now, is a service that uploads images to amazon S3. The upload stalls sometimes.

I also, think Mocha should be agnostic to what testing strategy any developers try to implement.

[rodoabad]

SUT is system under test. So what are you testing here?

it should...? Amazon is third party (as far as you're concerned) and will most likely be something that you cannot control. This is where stubs and mocks come in.

Even if you set a max fail rate of 3...if for some reason S3 is down at the time you ran your test then your suite just failed even though it should have passed (assuming everything else works).

I'm probably getting ahead of myself here since I don't know if you're writing unit tests or e2e, but you get the idea.

[tinganho]

SUT is system under test. So what are you testing here?

I'm testing a service. Where one of them forwards an upload of a picture to S3. I thought I explained it on my last comment?

I have also seen testing locally in webdriver also generates false negatives.

it should...? Amazon is third party (as far as you're concerned) and will most likely be something that you cannot control. This is where stubs and mocks come in.

@rodoabad I have used mocking and stubbing in almost all my projects in many years. Though, I recently discovered that it just better to not mock and stub things, because it is such a big overhead. I think it is up to every developer to choose what kind of testing strategy they choose or not choose to implement. Mocha should thus, be agnostic to which testing strategy you prefer. Even though you change the state of your S3 service you can revert it with an afterEach(...) hook. And really, to stub my S3 upload would require very much brain power to achieve, with dependency injection etc.

Even if you set a max fail rate of 3...if for some reason S3 is down at the time you ran your test then your suite just failed even though it should have passed (assuming everything else works).

Yes, this is one of the downside. It's not a perfect solution I'm suggesting. Though I think this rarely happens.

It's not unit testing, or it depends on your definition, I'm testing the direct output of an HTTP request. But I hope that you are not suggesting me to mock and stub things. I think it is really up to each developer to choose its testing strategy. I have been involved in really complex project where mocking and stubbing things would make you pull your hair off. I have been contributing a lot to the TypeScript project and there they just test the direct compiled output. No need to mock and stub things.

[chenchaoyi]

The retry/rerun is indeed useful. @tinganho How's this compared with #1773 btw ?

[tinganho]

@chenchaoyi maybe it is duplicate. A project owner can close this issue if they want and just put a comment on how they want to proceed. I could submit a PR. But I think the API interface need to be resolved first.

[rodoabad]

@tinganho No, I'm not suggesting that you change your testing methods. Everyone has their own style that they follow. So I'm not sure why you think I'm even trying to do that.

I just wanted to have some context.