Add new options to report fixture setup and teardown #1647

sallner · 2016-06-22T15:56:41Z

This PR implements the cli flags to show potential fixtures without actually executing it (--setup-plan) and actually testing all the setup and teardown of fixtures without executing the test functions (--setup-only).

Here's a quick checklist that should be present in PRs:

Target: for bug or doc fixes, target master; for new features, target features
Make sure to include one or more tests for your change
Add yourself to AUTHORS
Add a new entry to the CHANGELOG (choose any open position to avoid merge conflicts with other PRs)

Factor setuponly code out of runtestprotocol().

Add printing of fixture dependencies

* Fix the tests * Fix .format string failures on python 2.6

…ixtures

coveralls · 2016-06-22T16:07:23Z

Coverage increased (+0.04%) to 92.434% when pulling ecc97aa on sallner:features into 7d60fcc on pytest-dev:features.

coveralls · 2016-06-22T16:07:24Z

Coverage increased (+0.04%) to 92.434% when pulling ecc97aa on sallner:features into 7d60fcc on pytest-dev:features.

coveralls · 2016-06-22T16:07:24Z

Coverage increased (+0.04%) to 92.434% when pulling ecc97aa on sallner:features into 7d60fcc on pytest-dev:features.

coveralls · 2016-06-22T16:09:21Z

Coverage increased (+0.04%) to 92.434% when pulling ecc97aa on sallner:features into 7d60fcc on pytest-dev:features.

The-Compiler · 2016-06-22T17:54:15Z

_pytest/python.py

+        config = self._fixturemanager.config
+        capman = config.pluginmanager.getplugin('capturemanager')
+        if capman:
+            capman.suspendcapture()


Some earlier PR of @nicoddemus did the same, but did out, err = capman.suspendcapture() and then printed out/err to stdout/stderr after resuming capturing, as to not lose any captured content. Wouldn't the same apply here as well?

You're right, we didn't realise that .suspendcapture() + .resumecapture() loses the output unless explicitly passed through. Fixed.

The-Compiler · 2016-06-22T18:08:37Z

I haven't investigated yet, but I get this with --setup-plan for some tests when I tried this with qutebrowser's repo:

tests/unit/commands/test_runners.py 
      SETUP    F fail_tests_on_warnings
      TEARDOWN F cmdline_test
      TEARDOWN F fail_tests_on_warningsE

_________________________________________________________________________________ ERROR at setup of TestCommandRunner.test_parse_all[leave-mode] __________________________________________________________________________________

self = <CallInfo when='setup' exception: 'function' object is not subscriptable>, func = <function call_runtest_hook.<locals>.<lambda> at 0x7f6df9cc3950>, when = 'setup'

    def __init__(self, func, when):
        #: context of invocation: one of "setup", "call",
        #: "teardown", "memocollect"
        self.when = when
        self.start = time()
        try:
>           self.result = func()

.tox/py35/lib/python3.5/site-packages/_pytest/runner.py:163: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.tox/py35/lib/python3.5/site-packages/_pytest/runner.py:151: in <lambda>
    return CallInfo(lambda: ihook(item=item, **kwds), when=when)
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:724: in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:338: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:333: in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:595: in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:253: in _wrapped_call
    return call_outcome.get_result()
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:278: in get_result
    raise ex[1].with_traceback(ex[2])
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:264: in __init__
    self.result = func()
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:595: in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:253: in _wrapped_call
    return call_outcome.get_result()
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:278: in get_result
    raise ex[1].with_traceback(ex[2])
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:264: in __init__
    self.result = func()
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:595: in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:253: in _wrapped_call
    return call_outcome.get_result()
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:278: in get_result
    raise ex[1].with_traceback(ex[2])
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:264: in __init__
    self.result = func()
.tox/py35/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py:596: in execute
    res = hook_impl.function(*args)
.tox/py35/lib/python3.5/site-packages/_pytest/runner.py:100: in pytest_runtest_setup
    item.session._setupstate.prepare(item)
.tox/py35/lib/python3.5/site-packages/_pytest/runner.py:421: in prepare
    col.setup()
.tox/py35/lib/python3.5/site-packages/_pytest/python.py:1799: in setup
    fillfixtures(self)
.tox/py35/lib/python3.5/site-packages/_pytest/python.py:851: in fillfixtures
    request._fillfixtures()
.tox/py35/lib/python3.5/site-packages/_pytest/python.py:1947: in _fillfixtures
    item.funcargs[argname] = self.getfuncargvalue(argname)
.tox/py35/lib/python3.5/site-packages/_pytest/python.py:1990: in getfuncargvalue
    return self._get_active_fixturedef(argname).cached_result[0]
.tox/py35/lib/python3.5/site-packages/_pytest/python.py:2007: in _get_active_fixturedef
    result = self._getfuncargvalue(fixturedef)
.tox/py35/lib/python3.5/site-packages/_pytest/python.py:2072: in _getfuncargvalue
    val = fixturedef.execute(request=subrequest)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <FixtureDef name='cmdline_test' scope='function' baseid='tests' >, request = <SubRequest 'cmdline_test' for <Function 'test_parse_all[leave-mode]'>>

    def execute(self, request):
        # get required arguments and register our own finish()
        # with their finalization
        kwargs = {}
        for argname in self.argnames:
            fixturedef = request._get_active_fixturedef(argname)
            result, arg_cache_key, exc = fixturedef.cached_result
            request._check_scope(argname, request.scope, fixturedef.scope)
            kwargs[argname] = result
            if argname != "request":
                fixturedef.addfinalizer(self.finish)

        my_cache_key = request.param_index
        cached_result = getattr(self, "cached_result", None)
        if cached_result is not None:
            result, cache_key, err = cached_result
            if my_cache_key == cache_key:
                if err is not None:
                    py.builtin._reraise(*err)
                else:
                    return result
            # we have a previous but differently parametrized fixture instance
            # so we need to tear it down before creating a new one
            self.finish()
            assert not hasattr(self, "cached_result")

        fixturefunc = self.func

        if self.unittest:
            if request.instance is not None:
                # bind the unbound method to the TestCase instance
                fixturefunc = self.func.__get__(request.instance)
        else:
            # the fixture function needs to be bound to the actual
            # request.instance so that code working with "self" behaves
            # as expected.
            if request.instance is not None:
                fixturefunc = getimfunc(self.func)
                if fixturefunc != self.func:
                    fixturefunc = fixturefunc.__get__(request.instance)

        try:
            config = request.config
            if config.option.setupplan:
                result = None
            else:
                result = call_fixture_func(fixturefunc, request, kwargs)
            if config.option.setuponly or config.option.setupplan:
                # We want to access the params of ids if they exist also in during
                # the finish() method.
                if hasattr(request, 'param'):
                    if self.ids:
                        ind = self.params.index(request.param)
>                       self.cached_param = self.ids[ind]
E                       TypeError: 'function' object is not subscriptable

.tox/py35/lib/python3.5/site-packages/_pytest/python.py:2541: TypeError

tests/unit/commands/test_runners.py 
      SETUP    F fail_tests_on_warnings
      TEARDOWN F cmdline_test
      TEARDOWN F fail_tests_on_warningsE

kvas-it · 2016-06-23T08:27:22Z

Your qutebrowser test error is caused by ids being a function instead of a list. We missed that case, I will fix it.

coveralls · 2016-06-23T08:30:51Z

Coverage increased (+0.03%) to 92.427% when pulling 1a5e530 on sallner:features into 7d60fcc on pytest-dev:features.

kvas-it · 2016-06-23T08:57:28Z

@The-Compiler the qutebrowser issue should be fixed by the last commit.

coveralls · 2016-06-23T09:01:16Z

Coverage increased (+0.03%) to 92.428% when pulling c6af737 on sallner:features into 7d60fcc on pytest-dev:features.

The-Compiler · 2016-06-23T20:28:56Z

Interestingly enough this now triggers a segfault in Qt/PyQt 😆 Pretty sure that's not your fault though.

kvas-it · 2016-06-24T09:19:17Z

The segfault happens in this line:

        tw.write('[{0}]'.format(self.cached_param))

Where self.cached_param is the parameter of the fixture. It's not completely clear from the report which test was being set up, but I guess it's test_signal_name, which would mean that the parameter is SignalObject().signal1 (which is a pyqtSignal()). Could it be that it segfaults when format tries to repr it?

The-Compiler · 2016-06-24T19:54:53Z

That seems like a PyQt issue indeed, reported to their mailing list.

Also, changed their group from "general" to "debugconfig".

coveralls · 2016-06-25T10:50:39Z

Coverage increased (+0.06%) to 92.461% when pulling 32ca5cd on sallner:features into 7d60fcc on pytest-dev:features.

coveralls · 2016-06-25T11:03:48Z

Coverage increased (+0.06%) to 92.461% when pulling 32ca5cd on sallner:features into 7d60fcc on pytest-dev:features.

hpk42 · 2016-06-25T14:38:43Z

thanks!

sallner and others added 17 commits June 22, 2016 09:42

Implement working version of --setuponly

499c955

Factor setuponly code out of runtestprotocol().

6874c3a

Add test for setuponly option

7d19f83

Refactor logging of fixtures

92bcc36

Merge pull request #1 from kvas-it/features

de9ed5e

Factor setuponly code out of runtestprotocol().

Disable capturing for setuponly output

2c6cfa4

Add printing of fixture dependencies

2c5c4f3

Merge pull request #2 from kvas-it/features

ee311e1

Add printing of fixture dependencies

Fix the tests

1a75139

Fix .format string failures on python 2.6

5e0d78f

Fix the tests (#3)

f59d8f7

* Fix the tests * Fix .format string failures on python 2.6

Implement --setup-plan option

61992b4

Merge with upstream

c8c32fd

Add a test for handling of dynamic requests for fixtures from other f…

f7d50df

…ixtures

Add to changelog and authors.

0dbf77e

Merge from upstream

dd97a2e

Use correct links in changelog.

ecc97aa

The-Compiler reviewed Jun 22, 2016
View reviewed changes

Fix capturing with --setup-only/--setup-plan

1a5e530

Fix fixture parameter display when ids is a function

c6af737

splunk-djenkins added 5 commits June 25, 2016 12:19

Switch setuponly and setupplan options to a hook-based implementation.

032ce8b

Move setupplan and setuponly options to their respective modules.

da5c579

Also, changed their group from "general" to "debugconfig".

Add docs for setuponly and setupplan options.

7a3daac

Improve commenting for setupplan unittest.

9877bf4

Update changelog for new fixture hooks.

32ca5cd

hpk42 merged commit 13a188f into pytest-dev:features Jun 25, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new options to report fixture setup and teardown #1647

Add new options to report fixture setup and teardown #1647

sallner commented Jun 22, 2016

coveralls commented Jun 22, 2016 •

edited

Loading

coveralls commented Jun 22, 2016

coveralls commented Jun 22, 2016

coveralls commented Jun 22, 2016 •

edited

Loading

The-Compiler Jun 22, 2016 •

edited

Loading

kvas-it Jun 23, 2016

The-Compiler commented Jun 22, 2016 •

edited

Loading

kvas-it commented Jun 23, 2016

coveralls commented Jun 23, 2016 •

edited

Loading

kvas-it commented Jun 23, 2016

coveralls commented Jun 23, 2016 •

edited

Loading

The-Compiler commented Jun 23, 2016 •

edited

Loading

kvas-it commented Jun 24, 2016

The-Compiler commented Jun 24, 2016

coveralls commented Jun 25, 2016 •

edited

Loading

coveralls commented Jun 25, 2016 •

edited

Loading

hpk42 commented Jun 25, 2016

Add new options to report fixture setup and teardown #1647

Add new options to report fixture setup and teardown #1647

Conversation

sallner commented Jun 22, 2016

coveralls commented Jun 22, 2016 • edited Loading

coveralls commented Jun 22, 2016

coveralls commented Jun 22, 2016

coveralls commented Jun 22, 2016 • edited Loading

The-Compiler Jun 22, 2016 • edited Loading

Choose a reason for hiding this comment

kvas-it Jun 23, 2016

Choose a reason for hiding this comment

The-Compiler commented Jun 22, 2016 • edited Loading

kvas-it commented Jun 23, 2016

coveralls commented Jun 23, 2016 • edited Loading

kvas-it commented Jun 23, 2016

coveralls commented Jun 23, 2016 • edited Loading

The-Compiler commented Jun 23, 2016 • edited Loading

kvas-it commented Jun 24, 2016

The-Compiler commented Jun 24, 2016

coveralls commented Jun 25, 2016 • edited Loading

coveralls commented Jun 25, 2016 • edited Loading

hpk42 commented Jun 25, 2016

coveralls commented Jun 22, 2016 •

edited

Loading

coveralls commented Jun 22, 2016 •

edited

Loading

The-Compiler Jun 22, 2016 •

edited

Loading

The-Compiler commented Jun 22, 2016 •

edited

Loading

coveralls commented Jun 23, 2016 •

edited

Loading

coveralls commented Jun 23, 2016 •

edited

Loading

The-Compiler commented Jun 23, 2016 •

edited

Loading

coveralls commented Jun 25, 2016 •

edited

Loading

coveralls commented Jun 25, 2016 •

edited

Loading