Asserting that assertions were made #94

Closed
mstade opened this Issue Aug 9, 2012 · 30 comments

Comments

Projects
None yet

mstade commented Aug 9, 2012

Is there a way to assert whether any chai assertions have been made?

We've been bitten a number of times by tests passing gloriously in Mocha, only to realize that it's because no assertions were made for whatever reason (a common one is the misspelling of assertions, such as calledwith instead of calledWith. This would easily be caught by having a test hook that runs after each test to make sure that assertions were indeed made. It won't help with situations where some assertions are made and others are broken by misspelling or such, but in our experience it's been less common (if occurring at all.)

If not possible, I'm happy to contribute, just let me know!

Owner

logicalparadox commented Aug 9, 2012

This seems a bit tricky as a single line of an assertion can actually have multiple "assertions" (calls to this.assert) nested within it; counting could be misleading. However, open to suggestions/contributions if you have thoughts on how to implement...

QUnit has expect:

Specify how many assertions are expected to run within a test.

mstade commented Aug 21, 2012

I'd have to dig into the code to familiarize myself with it before suggesting implementations. As for the actual assertion count, I'd say it's less important than knowing whether any assertions were made at all. The former is useful for stats geeks I guess, but hardly useful for any type of code quality measurements. However, tests that don't have any assertions what so ever need to be singled out, as they are by definition pretty useless in that case :)

Owner

logicalparadox commented Aug 21, 2012

I like the idea of counts, just cause I'm a stats nerd. But the issue we run into is that QUnit's counting works because (1) the assertions and the test runner are integrated (2) the assertions aren't chain-able.

A cheap and dirty way to do this as a plugin would have this interface...

var chai = require('chai')
  , chaiCount = require('chai-counter')
  , should = chai.should();

chai.use(chaiCount);

var c = chai.counter();

something.should.equal('hello universe').count(c);
otherThing.should.have.property('hello')
  .and.be.a('string').count(c);

c.assert(2);

The count has to be the last in the chain in order for it to count properly. If it isn't last, there is no guarantee that the full thread finished.

I'm sure there might be a better way to do this...

mstade commented Aug 21, 2012

:)

My use case is a little bit different. I want to do this on a very global level, where before I run any test, the assertion counter (or flag) is reset, and after the test has run the counter/flag will be checked to see if it was hit at all. I use chai with mocha, and it's very easy to hook into before and after each test, but well in there I have no idea what to do with chai. I suppose solving that problem will inherently give you a correct counter as well, since you'd have to know whenever any type of assertion were made. Am I making any sort of sense? I feel like I might be confusing things.

Edit: The benefit of having a global counter or flag is of course that I can then make sure to fail any tests that didn't make any assertions at all. It happens a little bit too often that I misspell the code and mocha happily reports these tests as A-OK, which is clearly not right.

Owner

logicalparadox commented Aug 21, 2012

The only way I can think of a global counter/flag is to hook into Assertion.prototype.assert. The issue with this is that a flag/count can be misleading. Some assertions include multiple calls to this.assert. .length(n) is one example. It, internally, calls Assertion.prototype.assert to assert that the subject of the assertions has the property .length before actually checking the value of length. Your count for expect(arr).to.have.length(4) would be 2.

Alternatively, if you were doing something like expect(bln).to.be.a('boolean').and.treu (intentionally misspelled), your global flag would trigger true even though not all of your assertions were invoked. Not that efficient either.

If you know all of chai's internals completely, the counting solution isn't too bad. But I don't think that should be a requisite.

So, rock/hard place...

mstade commented Aug 21, 2012

Rock/hard place indeed. I think though, for my current use case, that hooking into Assertion.prototype.assert will be good enough. All I need is to know whether the test made any assertions at all and if hooking into that method will give me that, then I'm all game, even if it may be called more than once per true (for lack of a better word) assertion. I'll give it a go with my local set up and see if works out, thanks for that!

Owner

logicalparadox commented Aug 21, 2012

There should be sufficient information on writing helper on the docs site to get you started, but feel free to post code for review here if you need further assistance. Also, when you have a solution you like, please post it as a Gist and include your information on the wiki page for helpers. Thank you kindly!

Contributor

domenic commented Nov 25, 2012

You could hook into should and expect and assert. Each of them should only be called once per conceptual "assertion".

nzakas commented Nov 21, 2013

This thread seems to have died, but I have a need for this. I'm more interested in if any assertions occurred than the exact number. If there's any interest, I can take a look and make a PR for discussion. Shall I go ahead?

Owner

logicalparadox commented Nov 21, 2013

Sure, be interested to see what you come up with!

I'm definitely interested in this. I've fallen victim to the problem outlined above where no assertions were made but the test passes.

@nzakas nzakas added a commit to nzakas/chai that referenced this issue Nov 23, 2013

@nzakas nzakas Added assertion counting (fixes #94) e103da6

@nzakas nzakas added a commit to nzakas/chai that referenced this issue Nov 27, 2013

@nzakas nzakas Added assertion counting (fixes #94) 41eb468
Contributor

lo1tuma commented Feb 5, 2014

+1

The issue may be the hidden usage of property getters. Does anyone know of a library that rewrites these into actual functions, so the invocation of these properties will throw an error if misspelled? E.g. expect(foo).to.be.treu() should rightly err.

mstade commented Feb 25, 2014

tpyo is a proof-of-concept to show how proxy can be used to allow misspellings, perhaps there's inspiration to be had there.

By the way - to count assertions (or rather groups of assertions) you can use checkmark - the tiny library I created to overcome problems of that kind. Not the most elegant solution -- a kind of a workaround -- but it's simple, works for me and might also work for you.

I like @radkodinev's way if it were to be written as a Chai plugin. For instance:

var arr = ["array"];

it("should make an assertion", function(done) {
    // After one assertion is made, run `done`
    expect(1).check(done);

    // `.mark` will increment the internal counter above.
    arr.should.have.length(1).mark;
})

mstade commented Nov 19, 2014

FWIW I've since switched to must which solves all of this nonsense quite elegantly.

I made a gist implementing checkmark for chai if anyone's interested. It should accomplish most of what OP wants when using it with the done() callback structure of test frameworks.

Owner

keithamus commented Nov 26, 2014

I'm going to keep this open. I think this is a reasonable enough add-on to the core, if someone comes up with a sensible solution. In the meantime, @sirlancelot if you fancy making your gist an proper plugin (with an npm/bower package), you could add it to our plugins list (just edit the plugins.js file in chai/chai-docs)

I got around to making a full-fledged repository with linting: https://github.com/sirlancelot/chai-checkmark

@sirlancelot chai-checkmark is useful, thanks. I've implemented it in najax tests to make sure all different HTTP request types are covered.

My usage case is somewhat different than the described case. I would be interested to see more usage examples here, to show how test implementations could look.

Ciantic commented Apr 13, 2015

Just got hit by this, this should be fixed.

I can't rely on test framework which can't guarantee all assertions have been run. I'm now trying this must thing.

P.S I just opened an issue in Mocha to implement pluggable assertion counting in Mocha

Owner

keithamus commented Apr 7, 2016

I think I jumped the gun a bit earlier by saying a PR is wanted. It would be better to flesh out a design here before moving to making PRs

Contributor

Turbo87 commented Apr 7, 2016

I've written down a proposal of how it could look like in #670. Feel free to comment.

It would be awesome if this could also work with the ava test runner, which run tests concurrently, but in that case we would need to store the planned/executed variables on some sort of test context.

dtracers commented Jun 6, 2016 edited

I have a quick question.

how does it know what the last link in the chain is?

Because if you just do it by the last link in the chain and that every link in the chain is chainable or is valid then it would solve most of the issues

example:

expect(foo).to.be.true

this will add the number to the assertions when true is called bc its the last in the chain.

This may not fix the typo issue like true but it does get much closer to the problem.

It's like the mark library but built into the last assertion.

Contributor

meeber commented Jun 6, 2016 edited

@dtracers I don't think it's possible for a chain to know what its last link is, or for a link to know that it's the last link in a chain. At least not without some extreme AST parsing.

A consequence of enabling expressive chainable assertions is that there's nothing stopping someone from doing some pretty whacky things, such as splitting each link of a chain onto a separate line and then branching into multiple assertions.

This is a valid test:

it("hurts my soul", function () {                                               
  var link1 = expect(42);                                                       
  var link2 = link1.to;                                                         
  var link3a = link2.equal(42);                                                 
  var link3b = link2.not;                                                       
  var link4b = link3b.equal(43);                                                
});  

dtracers commented Jun 6, 2016

so basically the only way to really know is after each test you build a tree of all of the calls for that specific test then check leaf nodes of that built tree.

But the problem is that chai isnt really integrated on the test level.

Contributor

meeber commented Jul 1, 2016

The biggest issue here (with typos and and non-existent property assertions) is resolved via #721.
Discussion regarding possible future measures (e.g., validating assertions were run, or replacing property assertions with method assertions) can be discussed in #726.

meeber closed this Jul 1, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment