Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Functional and unit tests can not be run at the same time due to side effects #52

Closed
tswicegood opened this issue Jul 7, 2010 · 5 comments
Closed

Functional and unit tests can not be run at the same time due to side effects #52

tswicegood opened this issue Jul 7, 2010 · 5 comments

Comments

@tswicegood
Copy link
Contributor

I am unable to run all of the tests by calling nosetests tests/ due to side effects in the tests/functional/test_runner.py test cases.

I've narrowed the failure down to the @with_setup() calls to prepare_stderr and prepare_stdout. It appears there's some side effect to calling CALLBACK_REGISTRY.clear() and STEP_REGISTRY.clear().

Here's the test suite runs:

tests/functional/ all pass:

(lettuce)master!lettuce> nosetests tests/functional/
......................................................
----------------------------------------------------------------------
Ran 54 tests in 0.598s

OK

Followed by:

(lettuce)master!lettuce> nosetests tests/unit/
..............................................................................................
----------------------------------------------------------------------
Ran 94 tests in 0.098s

OK

But, when run together:

(lettuce)master!lettuce *> nosetests tests/functional/ tests/unit/
......................................................................................................................EEEEEEEEEEEE..................
======================================================================
ERROR: The scenario result has the steps passed, failed and skipped steps. And total steps as well.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/work/github/lettuce/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/Users/travis/work/github/lettuce/tests/unit/test_step_runner.py", line 104, in test_can_count_steps_and_its_states
    feature_result = f.run()
  File "/Users/travis/work/github/lettuce/lettuce/core.py", line 699, in run
    call_hook('before_each', 'feature', self)
  File "/Users/travis/work/github/lettuce/lettuce/registry.py", line 87, in call_hook
    sys.exit(2)
SystemExit: 2
-------------------- >> begin captured stdout << ---------------------


--------------------- >> end captured stdout << ----------------------

======================================================================
ERROR: The scenario result has also the undefined steps.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/work/github/lettuce/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/Users/travis/work/github/lettuce/tests/unit/test_step_runner.py", line 117, in test_can_point_undefined_steps
    feature_result = f.run()
  File "/Users/travis/work/github/lettuce/lettuce/core.py", line 699, in run
    call_hook('before_each', 'feature', self)
  File "/Users/travis/work/github/lettuce/lettuce/registry.py", line 87, in call_hook
    sys.exit(2)
SystemExit: 2
-------------------- >> begin captured stdout << ---------------------


--------------------- >> end captured stdout << ----------------------

======================================================================
ERROR: It can figure out why the test has failed
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/work/github/lettuce/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/Users/travis/work/github/lettuce/tests/unit/test_step_runner.py", line 133, in test_can_figure_out_why_has_failed
    feature_result = f.run()
  File "/Users/travis/work/github/lettuce/lettuce/core.py", line 699, in run
    call_hook('before_each', 'feature', self)
  File "/Users/travis/work/github/lettuce/lettuce/registry.py", line 87, in call_hook
    sys.exit(2)
SystemExit: 2
-------------------- >> begin captured stdout << ---------------------


--------------------- >> end captured stdout << ----------------------

======================================================================
ERROR: Skipped steps can be retrieved as steps
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/work/github/lettuce/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/Users/travis/work/github/lettuce/tests/unit/test_step_runner.py", line 147, in test_skipped_steps_can_be_retrieved_as_steps
    feature_result = f.run()
  File "/Users/travis/work/github/lettuce/lettuce/core.py", line 699, in run
    call_hook('before_each', 'feature', self)
  File "/Users/travis/work/github/lettuce/lettuce/registry.py", line 87, in call_hook
    sys.exit(2)
SystemExit: 2
-------------------- >> begin captured stdout << ---------------------


--------------------- >> end captured stdout << ----------------------

======================================================================
ERROR: By default lettuce ignore case on step definitions
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/work/github/lettuce/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/Users/travis/work/github/lettuce/tests/unit/test_step_runner.py", line 156, in test_ignore_case_on_step_definitions
    feature_result = f.run()
  File "/Users/travis/work/github/lettuce/lettuce/core.py", line 699, in run
    call_hook('before_each', 'feature', self)
  File "/Users/travis/work/github/lettuce/lettuce/registry.py", line 87, in call_hook
    sys.exit(2)
SystemExit: 2
-------------------- >> begin captured stdout << ---------------------


--------------------- >> end captured stdout << ----------------------

======================================================================
ERROR: Lettuce can, optionally consider case on step definitions
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/work/github/lettuce/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/Users/travis/work/github/lettuce/tests/unit/test_step_runner.py", line 166, in test_doesnt_ignore_case
    feature_result = f.run(ignore_case=False)
  File "/Users/travis/work/github/lettuce/lettuce/core.py", line 699, in run
    call_hook('before_each', 'feature', self)
  File "/Users/travis/work/github/lettuce/lettuce/registry.py", line 87, in call_hook
    sys.exit(2)
SystemExit: 2
-------------------- >> begin captured stdout << ---------------------


--------------------- >> end captured stdout << ----------------------

======================================================================
ERROR: Steps are aware of its definitions line numbers and file names
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/work/github/lettuce/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/Users/travis/work/github/lettuce/tests/unit/test_step_runner.py", line 177, in test_steps_are_aware_of_its_definitions
    feature_result = f.run()
  File "/Users/travis/work/github/lettuce/lettuce/core.py", line 699, in run
    call_hook('before_each', 'feature', self)
  File "/Users/travis/work/github/lettuce/lettuce/registry.py", line 87, in call_hook
    sys.exit(2)
SystemExit: 2
-------------------- >> begin captured stdout << ---------------------


--------------------- >> end captured stdout << ----------------------

======================================================================
ERROR: Steps that match groups takes them as parameters
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/work/github/lettuce/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/Users/travis/work/github/lettuce/tests/unit/test_step_runner.py", line 196, in test_steps_that_match_groups_takes_them_as_parameters
    feature_result = f.run()
  File "/Users/travis/work/github/lettuce/lettuce/core.py", line 699, in run
    call_hook('before_each', 'feature', self)
  File "/Users/travis/work/github/lettuce/lettuce/registry.py", line 87, in call_hook
    sys.exit(2)
SystemExit: 2
-------------------- >> begin captured stdout << ---------------------


--------------------- >> end captured stdout << ----------------------

======================================================================
ERROR: Steps that match named groups takes them as parameters
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/work/github/lettuce/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/Users/travis/work/github/lettuce/tests/unit/test_step_runner.py", line 209, in test_steps_that_match_named_groups_takes_them_as_parameters
    feature_result = f.run()
  File "/Users/travis/work/github/lettuce/lettuce/core.py", line 699, in run
    call_hook('before_each', 'feature', self)
  File "/Users/travis/work/github/lettuce/lettuce/registry.py", line 87, in call_hook
    sys.exit(2)
SystemExit: 2
-------------------- >> begin captured stdout << ---------------------


--------------------- >> end captured stdout << ----------------------

======================================================================
ERROR: Steps that match groups and named groups takes just the named as parameters
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/work/github/lettuce/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/Users/travis/work/github/lettuce/tests/unit/test_step_runner.py", line 221, in test_steps_that_match_groups_and_named_groups_takes_just_named_as_params
    feature_result = f.run()
  File "/Users/travis/work/github/lettuce/lettuce/core.py", line 699, in run
    call_hook('before_each', 'feature', self)
  File "/Users/travis/work/github/lettuce/lettuce/registry.py", line 87, in call_hook
    sys.exit(2)
SystemExit: 2
-------------------- >> begin captured stdout << ---------------------


--------------------- >> end captured stdout << ----------------------

======================================================================
ERROR: Step definitions takes step object as first argument
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/work/github/lettuce/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/Users/travis/work/github/lettuce/tests/unit/test_step_runner.py", line 240, in test_step_definitions_takes_the_step_object_as_first_argument
    feature_result = f.run()
  File "/Users/travis/work/github/lettuce/lettuce/core.py", line 699, in run
    call_hook('before_each', 'feature', self)
  File "/Users/travis/work/github/lettuce/lettuce/registry.py", line 87, in call_hook
    sys.exit(2)
SystemExit: 2
-------------------- >> begin captured stdout << ---------------------


--------------------- >> end captured stdout << ----------------------

======================================================================
ERROR: Features can run only specified scenarios, by index + 1
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/travis/work/github/lettuce/lib/python2.6/site-packages/nose/case.py", line 186, in runTest
    self.test(*self.arg)
  File "/Users/travis/work/github/lettuce/tests/unit/test_step_runner.py", line 256, in test_feature_can_run_only_specified_scenarios
    feature.run(scenarios=(2, 5))
  File "/Users/travis/work/github/lettuce/lettuce/core.py", line 699, in run
    call_hook('before_each', 'feature', self)
  File "/Users/travis/work/github/lettuce/lettuce/registry.py", line 87, in call_hook
    sys.exit(2)
SystemExit: 2
-------------------- >> begin captured stdout << ---------------------


--------------------- >> end captured stdout << ----------------------

----------------------------------------------------------------------
Ran 148 tests in 0.675s

FAILED (errors=12)
(lettuce)master!lett
@gabrielfalcao
Copy link
Owner

The right way to run tests is through:

unit tests

make unit

functional tests

make functional

integration tests (with django)

make integration

@tswicegood
Copy link
Contributor Author

In an ideal world, all tests would be runnable at once. Is there a specific reason that the functional tests should have a side effect that cause issues with the unit tests?

@gabrielfalcao
Copy link
Owner

unit tests are unwired and test stuff in a much steady, they also are many times faster than the functional ones.

functional tests depends on filesystem, operational system (e.g. stdout validation) and so on, which make them a bit slower.

integration tests, do test lettuce with different django projects, with different structure, and are the slower ones.

there is also the "acceptance" tests, which validates code within documentation (sphinx), but I don't really care for them so far.

Actually, in a ideal world, my tests are ran separately, so that I cat get feedback faster, as the test cases and suites grows and take even more time to run.

@tswicegood
Copy link
Contributor Author

I took this afternoon to dig through this code and figure it out. It was a globals issue where the registry was being setup in anticipation of a test inside tests.unit.test_step_runner, then getting blown away by the various prepare_std* functions inside the functional tests.

I've patched it by explicitly setting up the environment that's expected inside the tests.unit.test_step_runner code.

This doesn't effect the other way of running code and definitely takes longer (~7.5 seconds on my machine) to run the whole thing, but it corrects the issue of a false-failure when the entire test suite is run at once.

You can see the changes in commit 7f6e852. I'd love to get this back upstream.

@gabrielfalcao
Copy link
Owner

So, love it is! :) 0bf1f87

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tswicegood @gabrielfalcao