Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix test failure if pytest's flaky plugin is installed #104

Merged
merged 2 commits into from Sep 6, 2021

Conversation

mgorny
Copy link
Contributor

@mgorny mgorny commented Sep 3, 2021

Disable pytest's flaky plugin in test_simple_pytest_import_error_cli
in order to fix a test failure due to it mangling pytest's return code
on import error.

Disable pytest's flaky plugin in test_simple_pytest_import_error_cli
in order to fix a test failure due to it mangling pytest's return code
on import error.
@Erotemic Erotemic changed the base branch from master to dev/0.15.9 September 3, 2021 16:03
@Erotemic
Copy link
Owner

Erotemic commented Sep 3, 2021

What environment was this error caused in? None of the dashboards seem to have caught it. I'm happy to merge this, but I'd also like to reproduce the error so I can understand it and prevent similar errors in the future.

(I'm also wondering why the CircleCI dashboards aren't running, but maybe they will if I merge this into the development branch for 0.15.9, probably a permission issue.).

@mgorny
Copy link
Contributor Author

mgorny commented Sep 3, 2021

It happens if flaky is installed on the system. I suppose you could catch it via CI if you install that package in addition to the dependencies.

@Erotemic
Copy link
Owner

Erotemic commented Sep 4, 2021

I was able to reproduce this. It looks like with flaky installed the error code returned is a 3 instead of 1:

===================================================================================== FAILURES =====================================================================================
_______________________________________________________________________ test_simple_pytest_import_error_cli ________________________________________________________________________

    def test_simple_pytest_import_error_cli():
        """
        This test case triggers an excessively long callback in xdoctest <
        dev/0.15.7
    
        xdoctest ~/code/xdoctest/testing/test_pytest_cli.py test_simple_pytest_import_error_cli
    
        import sys, ubelt
        sys.path.append(ubelt.expandpath('~/code/xdoctest/testing'))
        from test_pytest_cli import *  # NOQA
        """
        module_text = utils.codeblock(
            '''
            # There are lines before the bad line
            import os
            import sys
            import does_not_exist
    
            def module_func1():
                """
                This module has a doctest
    
                Example:
                    >>> print('hello world')
                """
            ''')
        temp_module = util_misc.TempModule(module_text, modname='imperr_test_mod')
        command = sys.executable + ' -m pytest -v -s --xdoctest-verbose=3 --xdoctest ' + temp_module.dpath
        print(command)
        info = cmd(command)
        print(info['out'])
    
        # info = cmd('pytest --xdoctest ' + temp_module.modpath)
        # print(info['out'])
    
>       assert info['ret'] == 1
E       assert 3 == 1

testing/test_pytest_cli.py:78: AssertionError
============================================================================= short test summary info ==============================================================================
FAILED testing/test_pytest_cli.py::test_simple_pytest_import_error_cli - assert 3 == 1
=========================================================================== 1 failed, 4 passed in 3.26s ============================================================================

Perhaps we could just change the assert to check that assert info['ret'] != 0 and not have to explicitly call out a specific plugin, which I would like to avoid, xdoctest should try to avoid conflicting with other established plugins.

@mgorny
Copy link
Contributor Author

mgorny commented Sep 4, 2021

I'm happy with any solution that makes the tests pass.

@Erotemic
Copy link
Owner

Erotemic commented Sep 4, 2021

If you make modify the PR to make that change I'll merge it and push out a new patch.

@Erotemic
Copy link
Owner

Erotemic commented Sep 6, 2021

Thanks for this bug report! I pushed up the appropriate commits to this branch. Merging.

Do you need a new version of xdoctest with this fix released on pypi anytime soon? If so I can push one up and bump the patch version, if not I might wait a bit before making the new release.

@Erotemic Erotemic merged commit 54d83e3 into Erotemic:dev/0.15.9 Sep 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants