Concerns about not.property and not.throws

[meeber]

I'm having second thoughts about some of the recent changes made to not.property and not.throws.

Consider these examples:

expect(myCat).to.not.have.property("numFleas", 7);
expect(groomCat).to.not.throw(TypeError, "Invalid brush");

There are two ways to interpret these assertions:

New way:

• Expect myCat to not have a property myFleas that's equal to 7
• Expect groomCat to not throw a TypeError with the message "Invalid brush"

Old way:

• Expect myCat to have a property myFleas that's not equal to 7
• Expect groomCat to throw a TypeError with a message that's not "Invalid brush"

On the surface, the new way seems more intuitive. But in actuality it encourages writing bad tests in the exact same manner that adding an or assertion does.

Interpreting the assertions the new way means that expect(myCat).to.not.have.property("numFleas", 7); passes when myCat doesn't have a property numFleas OR when it does have a property numFleas but it's equal to something other than 7.

Likewise, expect(groomCat).to.not.throw(TypeError, "Invalid brush"); passes when groomCat doesn't throw any error, OR when it throws some error other than a TypeError, OR when it throws a TypeError with a message other than "Invalid brush".

When is it ever a good idea to test all of these different possibilities with a single assertion? Of course, you could always revert to the old way of interpretation by protecting your assertions like so:

expect(myCat).to.have.property("numFleas"); // Protection
expect(myCat).to.not.have.property("numFleas", 7);

expect(groomCat).to.throw(); // Protection
expect(groomCat).to.throw(TypeError); // Protection
expect(groomCat).to.not.throw(TypeError, "Invalid brush");

But it seems to me that the above protections should be there by default. I haven't been able to think of a single example when those protections should be removed. After all, without those protections in place, the new way of interpretation is just stringing together multiple assertions via "or".

I'm currently in favor of reverting the behavior of not.property and not.throws to the old way of interpretation. Thoughts?

Edit: Actually I think it was only not.property that used to follow the old way of interpretation. The recent changes made to not.throws were fixing unrelated bugs.

[lucasfcosta]

Hi @meeber, thanks for the detailed explanation, you always have interesting thoughts.

That's something we should definitely talk about.

When reading the first examples you've written the new way of interpretation is the first thing that comes to my mind. So, even though that encourages writing "bad" tests, I think that's the most obvious thing and I think that's our users would expect it to do.

We may be entering the realm of semantically obvious assertions versus educating users about it again. IMO our job is to write a library which does what people expect it to do when using certain methods, we're just giving them the best tool we can but it's their job to use it correctly.

When paying close attention to what happens whenever people use not we will be able to see that in fact the problem is not with the assertion itself, but with not. Whenever someone expects something to not be anotherThing they are implicitly saying that they expect something to be anything else than anotherThing and this means a lot of ORs (maybe infinite possibilities).

IMO we should keep the new behavior since its the most obvious thing and since the root cause of this problem is not really the property or throws assertion, but the not flag.

Please let me know if you disagree or you have any further interesting thoughts 😄

[meeber]

As an aside, it's worth noting that the assertions under discussion are very strange to begin with:

expect(myObj).to.not.have.property('myProp');  // Normal
expect(myObj).to.not.have.property('myProp', unexpectedVal);  // Strange; use expect(myObj).to.have.property('myProp', expectedVal) instead

expect(myFn).to.not.throw();  // Normal
expect(myFn).to.not.throw(unexpectedErrType);  // Strange; use expect(myFn).to.throw(expectedErrType) instead
expect(myFn).to.not.throw(unexpectedErrType, unexpectedMsg);  // Strange; use expect(myFn).to.throw(expectedErrType, expectedMsg) instead
expect(myFn).to.not.throw(unexpectedMsg);  // Strange; use expect(myFn).to.throw(expectedMsg) instead

In almost every case, it's better to use the non-negated version of the assertion.

Anyway, there's actually an issue here that's completely separate and unrelated to the concept you described above regarding a not comparison implicitly meaning infinite ORs.

The separate issue only involves assertions such as property and throws which can be given extra parameters that transform a single assertion into multiple, separate assertions. For example, a single throws assertion can perform up to 3 separate assertions depending on which arguments are given:

1. Assert whether or not an error is thrown
2. Optionally assert whether or not a thrown Error is of a specific type
3. Optionally assert whether or not a thrown Error contains a specific message.

Without the not flag, such a compound assertion isn't a problem. For example, expect(groomCat).to.throw(TypeError, "Invalid brush"); is conceptually equivalent to:

expect(groomCat).to.throwAnError()
  .and.expectTheErrorToBeATypeError()
  .and.expectTheErrorToHaveThisMessage("Invalid brush");

Unfortunately, we run into problems when applying not to a compound assertion. For example, using the new intrepretation, expect(groomCat).to.not.throw(TypeError, "Invalid brush"); is conceptually equivalent to:

expect(groomCat).to.notThrowAnError()
  .or.expect(groomCat).to.throwAnErrorButNotATypeError()
  .or.expect(groomCat).to.throwATypeErrorButNotWithThisMessage("Invalid brush");

This is a horrible assertion, but it's what Chai's new interpretation does by default when not is paired with a compound assertion such as throws and property. Also, it's worth pointing out that the reason it was so tricky to code the new logic for the negated throws assertion is because we were actually coding OR logic straight into the assertion :D

Anyway, it's much safer to instead interpret expect(groomCat).to.not.throw(TypeError, "Invalid brush"); as:

expect(groomCat).to.throwAnError()
  .and.expectTheErrorToBeATypeError()
  .and.expectTheErrorToNotHaveThisMessage("Invalid brush");

That way, only one thing is being asserted, instead of three things combined with OR.

I'm not so sure that this interpretation defies user expectations either. Partially because not.throws(<args>) and not.property(<args>) is such an abnormal thing to do to begin, and partially because I suspect the and version is usually what's actually intended by the user when writing such an assertion.

I feel like someone writing: expect(myFn).to.not.throw(TypeError, "blahblahblah"); is usually expecting a TypeError to be thrown, just not with the message "blahblahblah". But with the new interpretation, this assertion will continue to pass even when myFn stops throwing a TypeError or any error at all.

Also keep in mind that we're already doing something like this with other assertions. For example: expect('blah').to.not.be.above(15);. You could make the argument that this assertion should return true because blah isn't even a number so of course it's not above 15. But instead it throws a TypeError because multiple assertions are being made: 1) Is the value a number?, and 2) Is that number above 15? And more importantly, the two assertions are being tied together with AND instead of OR, with the not only applying to the second assertion ("having already established we have a number, is it above 15?")

[lucasfcosta]

Your argument regarding coding the OR logic into the assertion makes sense, but I still don't think that our user will expect the example you've used to behave that way.

I feel like someone writing: expect(myFn).to.not.throw(TypeError, "blahblahblah"); is usually expecting a TypeError to be thrown, just not with the message "blahblahblah". But with the new interpretation, this assertion will continue to pass even when myFn stops throwing a TypeError or any error at all.

Actually the user is not saying that he wants a TypeError but not with "blahblahblah" as its message. He is just saying that he wants anything that's not a TypeError and if it is it should not have "blahblahblah" as its message.

Perhaps we could analyze the possibility of removing the message check from the throws assertion and moving it to a separate assertion. This way our users would still be able to negate the message only if they wanted to. That new assertion, however, can already be made, since the Error thrown becomes the new subject of the assertion and thus our users could use existing assertions to check that Error object.

Anyway, let's wait to hear what @keithamus, @vieiralucas and @shvaikalesh or anyone else willing to contribute what they have to say about this. 😄

[meeber]

I'm still struggling to think of a single instance in which someone would want to use not.throws(<params>), regardless of Chai's interpretation of not. I can understand the motivation for using not.throws() without any parameters to assert that a function doesn't throw at all, but in what situation would someone assert that a function doesn't throw a specific type of error and/or doesn't throw an error with a specific message, instead of asserting that a function does throw a specific type of error and/or throws an error with a specific message?

Maybe the thing to do here is disallow not from being used in conjunction with Chai's compound assertions (e.g., property with 2 arguments, and throws with any arguments)?

[meeber]

I browsed through all of our assertions, and here are the ones that suffer from this problem when paired with not:

expect(obj).not.include(obj);  // Only a problem when more than one property pair in 2nd obj
expect(obj).not.property(name, val);
expect(obj).not.ownPropertyDescriptor(name, descriptor);
expect(obj).not.keys(keys);  // Only a problem when more than one key and `any` flag isn't set
expect(fn).not.throw(errType);
expect(fn).not.throw(errMsg);
expect(fn).not.throw(errType, errMsg);
expect(array).not.members(array);  // Only a problem when more than one item in 2nd array

In each case above, multiple assertions are being performed internally, and each assertion is connected via or. None of our other assertions seem to exhibit this problem when paired with not.

The questions are:

• Is it okay that these assertions are performing or logic internally?
• If it is okay, then what makes it different than us adding an explicit or assertion?
• If it's not okay, then what do we do about it?

Possible options:

1. Do nothing
2. Add warning to docs but otherwise do nothing
3. Throw error when not is used in situations that lead to or chaining
4. Change logic to use and instead (creates a new problem as noted by @lucasfcosta)

It's worth noting that in general there's very few instances in which it makes sense to use not; most of the time it's better to assert what is expected as opposed to what isn't expected.

Would love to hear more thoughts on this from the team! @lucasfcosta @keithamus @shvaikalesh @vieiralucas

[lucasfcosta]

Hi @meeber! Actually I wrote about it on my blog recently and there might be a few interesting considerations there about it.

As you've said yourself, it's better to assert on what is expected rather than asserting on what is not expected. Whenever using not we're saying that any other result except that one is acceptable, which is logically Inifity - 1 ORs. There might be some cases where it is useful to use it, such as when testing the not assertion itself or working with errors, otherwise I'd always advise that in general people should avoid it whenever they can.

Just by using not itself people are already creating a huge number of ORs and if we take into account that using not means: I want everything except this one result I think that our current behavior is right, because this one thing means this very specific thing (such as an specific type of error with an specific message). I'm not sure this paragraph was clear, but basically what I'm trying to say is that not allows all possibilities but once so by making sure this one possibility allowed meets both requirements then it is consistent with the rest of our logic.

Again, here I feel like we're in a case of what we think is correct versus what is semantically more obvious. Whenever confronted with these two options I would choose the second one. I'd rather allow people to do things that are not the best solution but work as they expect it to than reducing their options.

However, I'm feeling more and more inclined to start adding warnings to our assertions in order to start educating people about it. Maybe we could write a few detailed markdown documents on this repository and add links pointing to them after concise error messages.

[meeber]

@lucasfcosta Good points. If Chai was a young project, a strong argument could be made about reversing the decision to build not logic into every assertion, but at this stage of project maturity, it feels too disruptive to mess with. I think the way forward is to continue improving our documentation, noting common pitfalls and bad practices. Gonna close this issue.