additional unittest type equality methods

[benjaminp]

BPO	2578
Nosy	@smontanaro, @rhettinger, @gpshead, @ncoghlan, @pitrou, @giampaolo, @benjaminp, @ezio-melotti, @voidspace
Files	unittest-new-asserts.diff unittest-new-asserts-gps04.diff.txt unittest-assertis.diff unittest-sameelements.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = <Date 2009-04-05.19:22:41.257>
created_at = <Date 2008-04-08.01:01:25.325>
labels = ['type-feature', 'library']
title = 'additional unittest type equality methods'
updated_at = <Date 2011-03-16.03:42:27.279>
user = 'https://github.com/benjaminp'

bugs.python.org fields:

activity = <Date 2011-03-16.03:42:27.279>
actor = 'ezio.melotti'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2009-04-05.19:22:41.257>
closer = 'michael.foord'
components = ['Library (Lib)']
creation = <Date 2008-04-08.01:01:25.325>
creator = 'benjamin.peterson'
dependencies = []
files = ['13488', '13510', '13613', '13614']
hgrepos = []
issue_num = 2578
keywords = ['patch']
message_count = 68.0
messages = ['65132', '65142', '65145', '65146', '65148', '65150', '65154', '65240', '65244', '70982', '80748', '80773', '80774', '84342', '84353', '84359', '84360', '84379', '84413', '84426', '84427', '84430', '84431', '84432', '84619', '84622', '84625', '84627', '84633', '84650', '84757', '84759', '84795', '84829', '84831', '84836', '84838', '84861', '84866', '84885', '84973', '85047', '85107', '85111', '85116', '85199', '85221', '85242', '85243', '85247', '85249', '85269', '85274', '85275', '85317', '85326', '85327', '85399', '85400', '85401', '85405', '85406', '85416', '85422', '85431', '85442', '85556', '85560']
nosy_count = 11.0
nosy_names = ['skip.montanaro', 'rhettinger', 'gregory.p.smith', 'ncoghlan', 'pitrou', 'giampaolo.rodola', 'pupeno', 'benjamin.peterson', 'gpolo', 'ezio.melotti', 'michael.foord']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue2578'
versions = ['Python 3.1']

[benjaminp]

unittest has many redundant APIs (eg. failIf and assertFalse) which can
be phased out in 3.x. We may also want to change the actually methods so
they really do what they say:

if x == y:
   pass
 else:
   raise AssertionError(...)

rather than

 if x != y:
   raise AssertionError(...)

[gvanrossum]

My plan:

• add deprecations to 3.1
• remove in 3.3

[rhettinger]

In a separate bug report, Steve Purcell indicated that he was taking
the module in the opposite direction and adding more redundancy in an
effort to fully match the J-Unit API.

[gvanrossum]

I'm all for adding new APIs that provide new functionality (we have a
few at Google that we really should contribute).

I'm not for having several aliases for the same functionality; it maybe
the JUnit way but Python's unittest.py is not JUnit not does it strive
for 100% JUnit compatibility -- the languages just are to different.

What's the bug# for Steve's proposal?

[rhettinger]

See http://bugs.python.org/issue2249 for discussion.

Personally, I prefer minimalism and would like the API thinned
considerably. Also, I don't think all of the classes should be
exposed. AFAICT, nobody cares about test suite objects, test result
objects, and test loader objects. These are really artifiacts of an
implementation originally designed to demonstrate how a unittest suite
could be implemented.

In Kent Beck's book on Test Driven Development, he complains that most
unittest implementations spawned from his original work have grown far
too complicated and would be better served by sticking to a simple
framework for writing and running tests.

Some of that may have been lost in a effort to model J-Unit or to
expose all the parts in support of people who want to use subclassing
to write their own unittest variants and extensions (I have seen this
in practice but it is somewhat rare).

If making big changes to the unittest API is on the table, it is worth
considering alternatives like py.test which is more powerful, easier to
learn, much less verbose, and more in line with the way python is
usually written.

[gvanrossum]

Agreed that the various loader classes etc. are mostly a waste. The sad
thing is that everyone who is serious about unittests has to reinvent
the code that finds files ending in _test.py on the filesystem, or
something like that, since that's *not* supported directly.

[purcell]

Hey, I'm open to anything. If I started writing unittest from scratch
knowing what I know now, I'd probably have kept the API a little
slimmer. Oh, and I'd have named everthing according to Python
conventions; my deepest and belated apologies for that.

I think the design has held up pretty well, even if it's arguably not
the most pythonic. Its familiarity to users of other xUnit frameworks
really does help new Pythoneers start writing tests immediately. And as
for the TestLoader stuff, it looks (and perhaps is) a bit overblown, but
I can't count the number of times people have asked me how to do obscure
or unusual things with the module and I've been able to respond with
something like, "just write a custom TestLoader/TestRunner".

I don't intend to take unittest in any particular direction; truth be
told, I'm now only an occasional visitor to the land of Python, and I
don't think I've had commit rights since the move to subversion. My
continued involvement with the unittest tickets is mainly to help
provide input along the lines of "we discussed this years ago, and
decided against it / thought it would be great". Far be it from me to
stand in the way of progress -- I'd be happy to see unittest re-worked
in any way that makes sense.

[gvanrossum]

Steve Purcell <purcell@users.sourceforge.net> added the comment:
Hey, I'm open to anything. If I started writing unittest from scratch
knowing what I know now, I'd probably have kept the API a little
slimmer. Oh, and I'd have named everthing according to Python
conventions; my deepest and belated apologies for that.

I think the current consensus is to start trimming the API in 3.1. We
could start documenting best practices in 2.6 and 3.0 though.

I think the design has held up pretty well, even if it's arguably not
the most pythonic. Its familiarity to users of other xUnit frameworks
really does help new Pythoneers start writing tests immediately.

Though I wonder how common that use case is. Not all new Pythoneers
come from Java, you know... Many come from Perl, PHP, even C++, and
more and more come from not programming at all before.

And as
for the TestLoader stuff, it looks (and perhaps is) a bit overblown, but
I can't count the number of times people have asked me how to do obscure
or unusual things with the module and I've been able to respond with
something like, "just write a custom TestLoader/TestRunner".

I hope we can add more custom TestLoader/TestRunner subclasses for
some of the *common* use cases.

I don't intend to take unittest in any particular direction; truth be
told, I'm now only an occasional visitor to the land of Python, and I
don't think I've had commit rights since the move to subversion. My
continued involvement with the unittest tickets is mainly to help
provide input along the lines of "we discussed this years ago, and
decided against it / thought it would be great". Far be it from me to
stand in the way of progress -- I'd be happy to see unittest re-worked
in any way that makes sense.

And thanks for your continued involvement! I think the clue the
developer community can take from this is not to worry too much about
changing the original design; you don't seem to have a strong sense of
"ownership", which (in this case) sounds good to me.

[purcell]

Hey, I came to Python from Perl, Java *and* C++! I'm pretty sure the
xUnits in all those other languages hold to the same basic API.

Adding more TestLoaders/TestRunners sounds like an excellent move. I'd
expect that once those common conveniences have been provided for, the
core TestSuite/TestCase scheme will continue to be helpful.

I could certainly envisage the provision of convenience decorators that
would remove some of the typical subclassing boilerplate by
transforming plain functions into TestCases.

I was recently sent a fun paper describing a mechanism for transforming
doctests into 'unittest' tests, so there's obviously a lot of scope for
fun additions and changes.

I'll be interested to see what people come up with, and to help a little
if and when I can.

[pupeno]

Is there anything that can be done now about this issue? like renaming
the API and leaving the old names as aliases? If so, let me know and
I'll try to work on it.

[gpshead]

FYI - As a side note to this issue of removing the clutter of alternate
names - I intend to gather up the worthy non-clutter additional unittest
assertFoo APIs that we have at Google to contribute towards 3.1/2.7, my
goal is to do this before PyCon this year.

This thread from last spring summarizes the types of things I'd like
include already.

http://mail.python.org/pipermail/python-dev/2008-April/078716.html

[gpolo]

As an side-side-note, I find much more important to add some kind of
test discovery since it is being duplicated in several packages in Lib/
and will continue being duplicated.

[gvanrossum]

@gpsmith: Way to go!

@gpolo: Alas, test discovery is now a much harder problem because it
depends on using conventions for test naming. Unless all existing
implementations use the same conventions, it's hard to see how to
replace them. Please bring that up on python-ideas@python.org.

[gpshead]

Attaching a patch that adds Google's unittest.TestCase assertFooEqual()
extensions and automatically uses rich comparisons in assertEqual() with
nicely formatted failure messages for list, tuple, dict, set, frozenset.

The following are added by this patch:

+ def assertSequenceEqual(self, seq1, seq2, msg=None, seq_type=None):
+ def assertListEqual(self, list1, list2, msg=None):
+ def assertTupleEqual(self, tuple1, tuple2, msg=None):
+ def assertSetEqual(self, set1, set2, msg=None):
+ def assertIn(self, a, b, msg=None):
+ def assertNotIn(self, a, b, msg=None):
+ def assertDictEqual(self, d1, d2, msg=None):
+ def assertDictContainsSubset(self, expected, actual, msg=None):
+ def assertSameElements(self, expected_seq, actual_seq, msg=None):
+ def assertMultiLineEqual(self, first, second, msg=None):
+ def assertLess(self, a, b, msg=None):
+ def assertLessEqual(self, a, b, msg=None):
+ def assertGreater(self, a, b, msg=None):
+ def assertGreaterEqual(self, a, b, msg=None):
+ def assertIsNone(self, obj, msg=None):
+ def assertIsNotNone(self, obj, msg='unexpectedly None'):
+ def assertRaisesWithRegexpMatch(self, expected_exception,
expected_regexp,

As I said in the dev summit, I'll get some stats on how often each of
these are used relative to others from a large code base which has had
access to them for some time as an indicator of what may or may not be
worth making standard.

I did not include assertCommandSucceeds or assertCommandFails as those
are more difficult to do the right thing for everyone on cross platform
and our internal implementations use some other messy and not
integration-worthy internal libraries at the moment.

[pitrou]

With all due respect, I find the presence of numerous type specific
functions (assertListEqual, etc.) awful. I don't understand why the type
isn't simply cased in the standard assertEqual function. I also don't
think adding so many assert methods makes the API easy to remember and
pleasant to use.

[gpshead]

Some statistics from our large code base as to which of the assert
methods get used relative to the others:

These percentages are relative to the count of assertEqual(s) uses being
100% and cover python code written by a crazy large number of engineers.

assertListEqual 4.61%
assertNotEqual(s) 1.91%
assertDictEqual 1.90%
assertIn 1.75%
assertSameElements 1.08%
assertIsNone 0.47%
assertGreater 0.37%
assertMultiLineEqual 0.36%
assertNotIn 0.36%
assertLess 0.24%
assertRaisesWithRegexpMatch 0.23%
assertIsNotNone 0.20%
assertSetEqual 0.16%
assertGreaterEqual 0.11%
assertLessEqual 0.09%
assertTupleEqual 0.04%
assertSequenceEqual 0.04%
assertDictContainsSubset 0.02%

assertSequenceEqual is used by the List and Tuple equal functions so its
stand alone use isn't that relevant. DictContainsSubset does not appear
to be used much.

I -did- add casing for the type to the main assertEqual method so it may
make sense to make the new type specific methods private so that they're
not part of the public API (ie: I agree with pitrou that it seems
pointless).

[gpshead]

Oh for reference, i left these out but they may interest people for
completeness sake.

assert_ 15%
assertTrue 9%
assertFalse 5%

We don't currently have the auto type checking in assertEqual in our
internal codebase, direct use of the type specific methods has been
encouraged in the past but that doesn't mean it is the right thing for
Python.

[pitrou]

I -did- add casing for the type to the main assertEqual method so it may
make sense to make the new type specific methods private so that they're
not part of the public API (ie: I agree with pitrou that it seems
pointless).

Well, that's my own aesthetical feeling. Others may disagree :)
But I think the public API should stay reasonably compact, so that unit
testing doesn't incur a lot of cognitive overhead.

[gvanrossum]

The naming pattern (assertListEqual, assertDictEqual, etc.) is pretty
easy to pick up so I don't think the addition of several new methods is
a big deal. Making assertEqual() do the right thing all the time is
hard if there are subclasses that might override __eq__ or do other
tricks, so I am glad that Greg only put the simplest test possible in
assertEqual. And I would still want to have a way to explicit *disable*
that in case I am comparing containers with millions of items. New
assertions like assertSameElements, assertDictContainsSubset or
assertMultilineEqual are not things that can be guessed, and they *are*
useful.

The names are also trivial to understand when reading tests that someone
else wrote, so I am not so worried about the cognitive effort. I would
rather start deprecating the fail* variants and use assertTrue in favor
of assert_, and settle the question of whether to prefer assertEqual vs.
assertEquals (the former, please).

Finally, a simple refactoring of the code might be helpful where the
preferred name for each function is actually the name used in the 'def'
for it, so that tracebacks involving such assertions show the preferred
names, and so that their definitions can be found by searching the code
for e.g. "def assertEqual".

[smontanaro]

Personally, I would prefer it if unittest got rid of all the various
ways to spell "assert" and just let test cases use the assert statement.
I use nose for most of my stuff which supports/allows use of the assert
statement. I think my test cases are better for it.

(crap, crap, crap. I was scrolling through the messages and my finger slid
off the button on my laptop. I pressed it again without really looking.
I didn't realize the mouse was no longer over the scrollbar's thumb and,
too late, saw that I had clicked the Remove button of this message "msg84360".
I hope I clicked the cancel button soon enough. If not, can it be
"unremoved" programmatically? I have the text of the message if need be.)

[smontanaro]

Just in case it can't be retrieved, here is Greg's text from msg84360:

--------------------------------
Oh for reference, i left these out but they may interest people for
completeness sake.

assert_ 15%
assertTrue 9%
assertFalse 5%

We don't currently have the auto type checking in assertEqual in our
internal codebase, direct use of the type specific methods has been
encouraged in the past but that doesn't mean it is the right thing for
Python.
-------------------------------------------

Again, my apologies.

Skip

[gpolo]

Skip: bugs.python.org/issueXXXX?@action=edit&@add@messages=MSGNUM

This "Remove" button... :)

[gvanrossum]

I would love to use the time machine to kill the 'fail' variants as well
as 'assert_'. However they are in use and we don't want to break
people's tests. Breaking tests is painful. And there are a lots of
tests. So we'll have to support them for a long, long time. Maybe we
can stop documenting them in 3.1 (this would also require changing
things so that the 'def' lines all use the 'assert' variants). Then
maybe we can start deprecating them in 3.2 and 3.3, and perhaps remove
them in 3.4. I recommend we let 2.x alone, but 2to3 should fix all these.

(And don't worry, Skip, somebody already put that message back. You can
click on the link to deleted items in the History section at the bottom
of the issue.)

[benjaminp]

2009/3/29 Guido van Rossum <report@bugs.python.org>:

Guido van Rossum <guido@python.org> added the comment:

I would love to use the time machine to kill the 'fail' variants as well
as 'assert_'.  However they are in use and we don't want to break
people's tests.  Breaking tests is painful.  And there are a lots of
tests.  So we'll have to support them for a long, long time.  Maybe we
can stop documenting them in 3.1 (this would also require changing
things so that the 'def' lines all use the 'assert' variants).  Then
maybe we can start deprecating them in 3.2 and 3.3, and perhaps remove
them in 3.4.  I recommend we let 2.x alone, but 2to3 should fix all these.

Yes, actually this would be a great secondary use of 2to3's
infrastructure, and I think it could be done quite correctly because
people usually don't have many custom methods named failUnlessEqual()
etc.

Besides, they're tests, so people would know if they're broken. :)

[gpshead]

Updated patch after sprinting on this with Michael Foord.

TODO:

• update assertRaisesWithRegexpMatch to be a context manager.
• documentation.

i'm working on the docs now.

[gvanrossum]

I absolutely do not want it to warn by default in 3.1.
PendingDeprecationError is fine.

[gpshead]

sounds good. This is done for 2.7.

I will make sure it gets merged into 3.1 properly before closing.

[benjaminp]

Do you mind if I rename all the double underscore methods that Greg's
patch added to _ underscore matching the style of the rest of the module?

[gvanrossum]

Do I mind? I *insist*! I thought put a note about that in the code
review but I guess Greg missed it. Thanks for noticing.

[benjaminp]

Done in r71004.

[ncoghlan]

Bikeshedding a bit - the 'assertGreaterEqual' and 'assertLessEqual'
method names grate on me a little. My brain has to do a double take in
order to correctly insert the implied 'Or'. This reaction is made worse
when these two methods are juxtaposed with all the other "assert*Equal"
methods like DictEqual, ListEqual, SetEqual etc.

Would it be possible to lengthen these specific names to the more
explicit "assertGreaterOrEqual" and "assertLessOrEqual"?

Some really nice additions to the test module here! :)

[gvanrossum]

Nick Coghlan <ncoghlan@gmail.com> added the comment:
Bikeshedding a bit - the 'assertGreaterEqual' and 'assertLessEqual'
method names grate on me a little. My brain has to do a double take in
order to correctly insert the implied 'Or'. This reaction is made worse
when these two methods are juxtaposed with all the other "assert*Equal"
methods like DictEqual, ListEqual, SetEqual etc.

Would it be possible to lengthen these specific names to the more
explicit "assertGreaterOrEqual" and "assertLessOrEqual"?

No, please. Get used to it. Maybe it helps if you realize the parallel
with '__ge__'?

[smontanaro]

Am I the only person who wishes all the assert* and fail* methods would
simply go away in favor of simply using the assert statement?

Skip

[voidspace]

Why do you need the assert methods to go away in order to use assert
statements?

[smontanaro]

Michael> Why do you need the assert methods to go away in order to use
Michael> assert statements?

You don't, but use of assert statements seems a hell of a lot more Pythonic
to me than all the assert* or fail* method names which I can never remember.

S

[pitrou]

You don't, but use of assert statements seems a hell of a lot more Pythonic
to me than all the assert* or fail* method names which I can never remember.

Two problems:

1. they are optimized away in "-Oxxx" mode
2. they don't provide good reporting in case of failure

[ncoghlan]

Thanks Guido - the __ge__/le parallel does actually help me remember
the current names.

[smontanaro]

> You don't, but use of assert statements seems a hell of a lot more
>> Pythonic to me than all the assert* or fail* method names which I can
>> never remember.

Antoine> 1. they are optimized away in "-Oxxx" mode

As far as I can tell for Python itself we never use -O while running unit
tests.

Antoine> 2. they don't provide good reporting in case of failure

I've never had a problem with that. It yields a perfectly useful traceback,
and in the case of the nose package at least, it does the right thing with
it.

In any case, your test machinery will have to catch AssertionError
exceptions anyway.

[voidspace]

No - you catch self.failureException rather than AssertionError directly.

I use unittest precisely because of the rich failure information from
the assert methods - particularly the new ones in this patch!

[voidspace]

Why is assertMultiLineEqual not the default assert method for basestring?

Even for small strings the output is useful.

[gpshead]

assertMultiLineEqual might be reasonable as a default assertEqual for
unicode. I wouldn't want to do it for bytes (py 2.x string).

[voidspace]

Fair point. :-)

[voidspace]

I'd like to add assertIs and assertNotIs.

We have these at work and I would find them useful whilst writing tests
for another set of changes to unittest I'm currently working on!

[pitrou]

Le samedi 04 avril 2009 à 15:06 +0000, Michael Foord a écrit :

Michael Foord <michael@voidspace.org.uk> added the comment:

I'd like to add assertIs and assertNotIs.

Let's call it AssertIsNot (we type "x is not None", not "x not is
None").

[voidspace]

I would *mmuch* prefer assertIsNot but it is not symmetrical with the
other asserts.

[pitrou]

I would *mmuch* prefer assertIsNot but it is not symmetrical with the
other asserts.

Well, apparently assertIsNotNone() has been added, so it would be good
to be symmetrical with that :)

[benjaminp]

I found while merging this to Py3k that dicts can't be used in
assertSameElements because they are unorderable in py3k.

[voidspace]

Patch with assertIs and assertIsNot. Docs but nothing in NEWS as already
covered.

Ok to apply?

[voidspace]

Patch for Py3k with fallback for comparing unsortable sequences in
assertSameElements.

Removed the expected failure and added another test case to confirm that
this patch works for unsortable sequences that are the same (no fail)
and different (fail).

[pitrou]

Seeing how a change in assertNotEqual unveiled a bug in OrderedDict,
would it be desireable for each of assertEqual and assertNotEqual to
test both the '!=' and '==' operators?

[ncoghlan]

No, because the rich comparison docs explicitly state that the
interpreter makes no assumptions about the relationship between != and
== (or, more precisely, __eq__ and __ne__).

"""There are no implied relationships among the comparison operators.
The truth of x==y does not imply that x!=y is false. """

(from http://docs.python.org/reference/datamodel.html#basic-customization)

If someone is writing unit tests for comparison methods they should
explicitly test all 6 operations.

[gpshead]

fyi - Your docs in unittest-assertis.diff mention the wrong method name
(assertIsNotNone instead of assertIsNot). fix that and commit.

The unittest-sameelements.diff patch looks good.

[voidspace]

Committed in revision 71263. Closing as there is nothing outstanding on
this issue.