Reduce pytest's overhead for running tests

[pytestbot]

Originally reported by: Alex Gaynor (BitBucket: alex_gaynor, GitHub: alex_gaynor)

---

Consider this example:

import time

import pytest


@pytest.mark.parametrize("x", xrange(5000))
def test_foo(x):
    time.sleep(.001)

5000 tests * .001 seconds each = expected about 5 seconds of runtime. Add a second or two for overhead.

On my system this actually takes about 15 seconds. As you add more and more tests, this overhead appears to be linear (10,000 tests takes ~30 seconds).

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/401

[pytestbot]

Original comment by Anatoly Bubenkov (BitBucket: bubenkoff, GitHub: bubenkoff):

---

This is something recently introduced or it was like this with earlier releases?

[pytestbot]

Original comment by holger krekel (BitBucket: hpk42, GitHub: hpk42):

---

It's a know fact (to me) that pytest's overhead in running tests is larger than with nose or unittest. If we consider just the running of an empty test suite (similar to what Alex posted) there are several contributing factors:

• fd-level capturing takes >10 percent. It's performed for each setup, call and teardown phase of a test and it's known to not be very efficient because it tries too hard to remain compatible to pytest-capturelog. I believe we should aim at integrating the pytest-capturelog plugin and simplify and speed up capturing.
• hook invocation: at test run-time there is too much dynamism involved in calling plugin hooks. Instead, the plugin system should work more like the fixture system: for each plugin we detect all hooks and memorize a complete list according to location. We can also generate a "trampolin function" for each hook, adapting the signature (pytest hook impls can accept fewer parameters than what is available). IOW, we invest a bit more constant overhead in order to decrease the overhead for each test.
• yet unknown/unanalyzed factors

I think it takes about a week of concerted focused effort to realize significant gains.

[RonnyPfannschmidt]

@hpk42 as far as i can tell you solved the hook details, i'd like to discuss a way to solve capture soon

[RonnyPfannschmidt]

hooks where addressed, with the 2 followups i opened we can close this one