Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

unexpected successes are not output #67004

Closed
rbtcollins opened this issue Nov 7, 2014 · 11 comments
Closed

unexpected successes are not output #67004

rbtcollins opened this issue Nov 7, 2014 · 11 comments
Assignees
Labels
3.11 bug and security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@rbtcollins
Copy link
Member

BPO 22815
Nosy @rbtcollins, @ezio-melotti, @voidspace, @serhiy-storchaka, @iritkatriel
PRs
  • bpo-22815: Print unexpected successes in summary in TextTestResult #30138
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = 'https://github.com/serhiy-storchaka'
    closed_at = <Date 2021-12-26.11:23:43.418>
    created_at = <Date 2014-11-07.10:21:48.117>
    labels = ['type-feature', 'library', '3.11']
    title = 'unexpected successes are not output'
    updated_at = <Date 2021-12-26.11:23:43.418>
    user = 'https://github.com/rbtcollins'

    bugs.python.org fields:

    activity = <Date 2021-12-26.11:23:43.418>
    actor = 'serhiy.storchaka'
    assignee = 'serhiy.storchaka'
    closed = True
    closed_date = <Date 2021-12-26.11:23:43.418>
    closer = 'serhiy.storchaka'
    components = ['Library (Lib)']
    creation = <Date 2014-11-07.10:21:48.117>
    creator = 'rbcollins'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 22815
    keywords = ['patch']
    message_count = 11.0
    messages = ['230783', '407833', '407834', '408679', '408681', '408683', '408685', '408689', '408690', '408693', '409190']
    nosy_count = 5.0
    nosy_names = ['rbcollins', 'ezio.melotti', 'michael.foord', 'serhiy.storchaka', 'iritkatriel']
    pr_nums = ['30138']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue22815'
    versions = ['Python 3.11']

    @rbtcollins
    Copy link
    Member Author

    Unexpected successes cause failures, but we don't output details about them at the end of the run.

    From https://code.google.com/p/unittest-ext/issues/detail?id=22

    A complicating factor is that we don't have a backtrace to show - but we may have captured stdout/stderr or in future various test attachments to show.

    @rbtcollins rbtcollins added type-bug An unexpected behavior, bug, or error stdlib Python modules in the Lib dir labels Nov 7, 2014
    @iritkatriel
    Copy link
    Member

    I think this has been fixed:

    % cat mm.py
    import unittest

    class TestStringMethods(unittest.TestCase):
    
        @unittest.expectedFailure
        def test_upper(self):
            self.assertEqual(2, 2)
    
    if __name__ == '__main__':
        unittest.main()

    % ./python.exe mm.py
    x
    ----------------------------------------------------------------------
    Ran 1 test in 0.001s

    OK (expected failures=1)
    iritkatriel@Irits-MBP cpython % vi mm.py
    iritkatriel@Irits-MBP cpython % ./python.exe mm.py
    u
    ----------------------------------------------------------------------
    Ran 1 test in 0.000s

    FAILED (unexpected successes=1)

    @iritkatriel
    Copy link
    Member

    Or do you mean that there should be more output?

    @iritkatriel
    Copy link
    Member

    In verbose mode we do get output like the one in the code.google issue:

    % cat tmp.py

    import unittest
    
    class TestStringMethods(unittest.TestCase):
    
        @unittest.expectedFailure
        def test_upper(self):
            self.assertEqual(2, 2)
    
        def test_lower(self):
            return
    
    if __name__ == '__main__':
        unittest.main()

    % ./python.exe -m tmp -v
    test_lower (main.TestStringMethods) ... ok
    test_upper (main.TestStringMethods) ... unexpected success

    @iritkatriel
    Copy link
    Member

    I don't think adding more information from the test would be helpful - when a test fails you have (and need) information about what happened. But when a test succeeds the code of the test basically tells you what happened - all the assertions passed.

    I think we can close this.

    @serhiy-storchaka
    Copy link
    Member

    I think the OP means that test details (test description, traceback, captured output) are printed for ERROR and FAIL in TextTestResult.printErrors(). Other possible results besides error and failure are success, skipped, expected failure and unexpected success. Success, skipped and expected failure are expected, they do not cause total failure, but unexpected success is different. Result.wasSuccessful() returns False if there are any errors, failures or unexpected successes. But unlike errors and failures we do not have any details about unexpected successes. It looks like oversight to me.

    Yes, there are no traceback for unexpected success, but at least we can output the test description and captured output.

    @serhiy-storchaka serhiy-storchaka added the 3.11 bug and security fixes label Dec 16, 2021
    @serhiy-storchaka serhiy-storchaka self-assigned this Dec 16, 2021
    @serhiy-storchaka
    Copy link
    Member

    There are no other details for unexpected successes, only test descriptions.

    @serhiy-storchaka serhiy-storchaka added type-feature A feature request or enhancement and removed type-bug An unexpected behavior, bug, or error labels Dec 16, 2021
    @iritkatriel
    Copy link
    Member

    Yes, this is why I don’t understand the request here. When a test fails you know which assertion was violated or which line raised an exception. When a multi line test succeeds, you don’t know which line was supposed to fail.

    @iritkatriel
    Copy link
    Member

    I know that in unittest2 an unexpected success did not cause the test to fail (when we moved from unittest2 to unittest at work we got test failures due to unexpected successes that previously did not show up). I don't know the whole history of how unittest2 came to be, I think it was a clone of an older copy of unittest? Maybe that it the background for this issue?

    @serhiy-storchaka
    Copy link
    Member

    Unless you tun tests in verbose mode you do not know which of tests was unexpectedly successful. And even in the verbose mode it is not easy to find that test in long output.

    Yes, unexpected successes considered successes in earlier versions. See bpo-20165.

    You can find many interesting when read git logs.

    @serhiy-storchaka
    Copy link
    Member

    New changeset 1944434 by Serhiy Storchaka in branch 'main':
    bpo-22815: Print unexpected successes in summary in TextTestResult (GH-30138)
    1944434

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.11 bug and security fixes stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    3 participants