Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LocalBearTestHelper: Add ``assertObjectsEqual`` #4310

Merged
merged 1 commit into from Feb 21, 2018

Conversation

@yash-nisar
Copy link
Member

@yash-nisar yash-nisar commented Jun 2, 2017

Add assertObjectsEqual(..) method that compares individual
fields of the Result object and yields better, easy to understand
messages in case of an attribute mismatch.

Closes #4302

For short term contributors: we understand that getting your commits well
defined like we require is a hard task and takes some learning. If you
look to help without wanting to contribute long term there's no need
for you to learn this. Just drop us a message and we'll take care of brushing
up your stuff for merge!

Checklist

  • I read the commit guidelines and I've followed
    them.
  • I ran coala over my code locally. (All commits have to pass
    individually.
    It is not sufficient to have "fixup commits" on your PR,
    our bot will still report the issues for the previous commit.) You will
    likely receive a lot of bot comments and build failures if coala does not
    pass on every single commit!

After you submit your pull request, DO NOT click the 'Update Branch' button.
When asked for a rebase, consult coala.io/rebase
instead.

Please consider helping us by reviewing other peoples pull requests as well:

The more you review, the more your score will grow at coala.io and we will
review your PRs faster!

@yash-nisar
Copy link
Member Author

@yash-nisar yash-nisar commented Jun 2, 2017

status/blocked on coala_utils release which will include https://gitlab.com/coala/coala-utils/merge_requests/42

@yash-nisar
Copy link
Member Author

@yash-nisar yash-nisar commented Jun 2, 2017

@yash-nisar
Copy link
Member Author

@yash-nisar yash-nisar commented Jun 2, 2017

Note that the tests will fail because we still don't have the __compare_fields attribute which will only be added after https://gitlab.com/coala/coala-utils/merge_requests/42 is merged and a new version of coala_utils is released(maybe).

coalib/testing/LocalBearTestHelper.py Outdated
@@ -185,10 +201,11 @@ def check_results(self,
create_tempfile=create_tempfile,
tempfile_kwargs=tempfile_kwargs,
settings=settings)
self.addTypeEqualityFunc(list, self.assertObjectsEqual)

This comment has been minimized.

@Makman2

Makman2 Jun 3, 2017
Member

  1. This should be run somewhere in the beginning of a base test class or so, otherwise it takes effect only in check_validity
  2. list doesn't use generate_eq or generate_repr, so this is not really working

This comment has been minimized.

@yash-nisar

yash-nisar Jun 3, 2017
Author Member

But we're basically comparing list of Result objects, so whenever we pass lists to the assertEqual(...) for comparison, it will call the assertObjectsEqual(...) method IIUC. What I had initially done was put Result object instead of list since we wanna call that method whenever we're comparing Result objects, see below :
Snippet: self.addTypeEqualityFunc(list, self.assertObjectsEqual)
Output:

E   AssertionError: 'mixed spaces and tabs' != 'mixed spaces and tab'
E   - mixed spaces and tabs
E   ?                     -
E   + mixed spaces and tab
E    : message_base mismatch

Snippet: self.addTypeEqualityFunc(Result, self.assertObjectsEqual)
Output:

E   AssertionError: Lists differ: [<Result object(id=0xf33dc5cf898c48b4b7114cffd86e8172, origin='[490 chars]438>] != [<Result object(id=0xf211aef4abb24ecb8cc36beebbd848a8, origin='[489 chars]d30>]
E   
E   First differing element 0:
E   <Result object(id=0xf33dc5cf898c48b4b7114cffd86e8172, origin='[489 chars]a438>
E   <Result object(id=0xf211aef4abb24ecb8cc36beebbd848a8, origin='[488 chars]2d30>
E   
E   Diff is 2240 characters long. Set self.maxDiff to None to see it.

CC @Makman2

This comment has been minimized.

@yash-nisar

yash-nisar Jun 3, 2017
Author Member

Should I create an __init__ method inside our LocalBearTestHelper class and put addTypeEqualityFunc(..) there ?

This comment has been minimized.

@yash-nisar

yash-nisar Jun 3, 2017
Author Member

Another alternative that I've found is we pass the Result object to assertEqual which is something like(Error handling of empty lists, and so on will be done..) :

        if not check_order:
            self.assertEqual(sorted(bear_output)[0], sorted(results)[0])
        else:
            self.assertEqual(bear_output[0], results[0])

Then self.addTypeEqualityFunc(Result, self.assertObjectsEqual) snippet is justified IIUC.

This comment has been minimized.

@Makman2

Makman2 Jun 4, 2017
Member

Ouch haven't thought about that we pass results with lists... here we could just iterate over all passed results and compare them one by one with assertEqual. And we still register addTypeEqualityFunc with Result globally (or an ABC-type of classes using generate_eq and generate_ordering which would be the ideal solution).

Hm this is better than nothing, but ideally we would be able to have nicer outputs for collections... don't know something yet, maybe you also get some ideas how to make self.assertObjectsEqual working automatically everywhere in our tests, even those ones not testing only bears.

This comment has been minimized.

@yash-nisar

yash-nisar Jun 4, 2017
Author Member

Ouch haven't thought about that we pass results with lists... here we could just iterate over all passed results and compare them one by one with assertEqual. And we still register addTypeEqualityFunc with Result globally (or an ABC-type of classes using generate_eq and generate_ordering which would be the ideal solution).

Aren't we doing this already in #4310 (comment) ?

Hm this is better than nothing, but ideally we would be able to have nicer outputs for collections... don't know something yet, maybe you also get some ideas how to make self.assertObjectsEqual working automatically everywhere in our tests, even those ones not testing only bears

Our main concern ATM is the not so useful output of bears so IMO we should get this basic functionality merged and those not testing only bears can be looked after later on ? What say ?

@gitmate-bot
Copy link
Collaborator

@gitmate-bot gitmate-bot commented Jun 11, 2017

Hey! This pull request hasn't been updated for a while :/ It would be nice if we could get this going again!

coalib/testing/LocalBearTestHelper.py Outdated
@@ -185,10 +201,11 @@ def check_results(self,
create_tempfile=create_tempfile,
tempfile_kwargs=tempfile_kwargs,
settings=settings)
self.addTypeEqualityFunc(list, self.assertObjectsEqual)

This comment has been minimized.

@Makman2

Makman2 Jun 13, 2017
Member

We could actually use Iterable instead of only list, so we support any kind of iterable 👍

This comment has been minimized.

@Makman2

Makman2 Jun 13, 2017
Member

Or maybe for this case we should explicitly use assertObjectsEqual instead of registering it for tests. Makes it easier to understand^^

coalib/testing/LocalBearTestHelper.py Outdated
else:
self.assertEqual(bear_output, results, msg=msg)
self.assertEqual(bear_output, results)

This comment has been minimized.

@Makman2

Makman2 Jun 13, 2017
Member

can we delete the msg variable above as we don't seem to use it any more?

@yash-nisar
Copy link
Member Author

@yash-nisar yash-nisar commented Jun 13, 2017

Will fix things suggested by the review comments in the next iteration and iterate over the list containing more than multiple results.

@yash-nisar yash-nisar force-pushed the yash-nisar:improve-testing-api branch Jun 19, 2017
@yash-nisar yash-nisar changed the title LocalBearTestHelper: Add ``assertObjectsEqual`` WIP LocalBearTestHelper: Add ``assertObjectsEqual`` Jun 19, 2017
@yash-nisar yash-nisar force-pushed the yash-nisar:improve-testing-api branch Jun 19, 2017
@yash-nisar
Copy link
Member Author

@yash-nisar yash-nisar commented Jun 19, 2017

Updated @Makman2

coalib/results/Result.py Outdated
@@ -25,7 +26,7 @@
'aspect',
'additional_info',
'debug_msg')
class Result:
class Result(Comparables):

This comment has been minimized.

@Makman2

Makman2 Jun 19, 2017
Member

We want to register Result to be a deriving type of Comparables inside the generate_eq/generate_ordering decorators, not here 👍

This comment has been minimized.

@yash-nisar

yash-nisar Jun 20, 2017
Author Member

Yeah, I just got to know the difference between the 2 ways of subclassing, the real one and the virtual one. I had actually referred to https://pymotw.com/3/abc/ which doesn't explicitly state the difference. My bad . :)

@yash-nisar yash-nisar force-pushed the yash-nisar:improve-testing-api branch Jun 21, 2017
@gitmate-bot gitmate-bot added size/S and removed size/M labels Jun 21, 2017
@yash-nisar yash-nisar force-pushed the yash-nisar:improve-testing-api branch 2 times, most recently Jun 21, 2017
@gitmate-bot gitmate-bot added size/M and removed size/S labels Jun 22, 2017
@yash-nisar yash-nisar force-pushed the yash-nisar:improve-testing-api branch 2 times, most recently Jun 22, 2017
@yash-nisar
Copy link
Member Author

@yash-nisar yash-nisar commented Jun 22, 2017

How does this look ? @Makman2

@yash-nisar
Copy link
Member Author

@yash-nisar yash-nisar commented Feb 2, 2018

@Makman2 Ready for merge ?

@gitmate-bot
Copy link
Collaborator

@gitmate-bot gitmate-bot commented Feb 2, 2018

Sorry @yash-nisar, you do not have the necessary permission levels to perform the action.

@Makman2
Copy link
Member

@Makman2 Makman2 commented Feb 14, 2018

ack 14642d8

@Makman2
Copy link
Member

@Makman2 Makman2 commented Feb 21, 2018

@gitmate-bot rebase

@gitmate-bot
Copy link
Collaborator

@gitmate-bot gitmate-bot commented Feb 21, 2018

Hey! I'm GitMate.io! This pull request is being rebased automatically. Please DO NOT push while rebase is in progress or your changes would be lost permanently ⚠️

Add ``assertObjectsEqual(..)`` method that compares individual
fields of the Result object and yields better, easy to understand
messages in case of an attribute mismatch.

Closes #4302
@gitmate-bot
Copy link
Collaborator

@gitmate-bot gitmate-bot commented Feb 21, 2018

Automated rebase with GitMate.io was successful! 🎉

@Makman2
Copy link
Member

@Makman2 Makman2 commented Feb 21, 2018

@gitmate-bot
Copy link
Collaborator

@gitmate-bot gitmate-bot commented Feb 21, 2018

Hey! I'm GitMate.io! This pull request is being fastforwarded automatically. Please DO NOT push while fastforward is in progress or your changes would be lost permanently ⚠️

@gitmate-bot
Copy link
Collaborator

@gitmate-bot gitmate-bot commented Feb 21, 2018

Automated fastforward with GitMate.io was successful! 🎉

@gitmate-bot gitmate-bot merged commit 31ad864 into coala:master Feb 21, 2018
8 of 10 checks passed
8 of 10 checks passed
continuous-integration/appveyor/branch Waiting for AppVeyor build to complete
Details
continuous-integration/travis-ci/push The Travis CI build is in progress
Details
ci/circleci Your tests passed on CircleCI!
Details
codecov/project 100% (target 100%)
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
review/gitmate/commit This commit has no issues. :)
Details
review/gitmate/manual This commit was acknowledged. :)
Details
review/gitmate/manual/pr This PR is reviewed. :)
Details
review/gitmate/pr This PR has no issues. :)
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Linked issues

Successfully merging this pull request may close these issues.

6 participants
You can’t perform that action at this time.