New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[POC] #24721 -- Test extensions #4926

Closed
wants to merge 10 commits into
base: master
from

Conversation

Projects
None yet
3 participants
@mjtamlyn
Member

mjtamlyn commented Jun 26, 2015

Proof of concept code to demonstrate test extensions, with the two main examples implemented.

Other possible use cases:

  • Alternative data stores - a RedisExtension which selects an empty redis db and flushes it after each test
  • Queue management - record celery tasks which have been queued and/or called by means of a custom Broker
  • Setting up an external authentication system

Thoughts:

  • Recursively iterating through the test suite classes to add _extensions is kinda ugly, but I deemed it preferable to the alternative code which would involve a global set of extensions. If nothing else, this will be easier to test.
  • As yet, I have not added a setting. There is some merit in TEXT_EXTENSIONS rather than subclassing the test runner, but given we already have TEST_RUNNER, and this functionality is dependent on the test runner being a subclass of our DiscoverRunner, I'm not sure it's worth worrying.
  • Current code in TestCase is not robust to the absence of _extensions.
  • Might be nice to allow the customisation of extensions on individual TestCase classes.
  • Instead of attaching to mail.outbox and messages.outbox, how about attaching TestCase._env where such objects could live?
  • Adding assertion helper methods could be nice too, but would be very magical
[POC] #24721 -- Test extensions
Proof of concept code to demonstrate test extensions, with the two main
examples implemented.
Show outdated Hide outdated django/contrib/messages/test.py Outdated
@carljm

This comment has been minimized.

Show comment
Hide comment
@carljm

carljm Jun 26, 2015

Member

I'd vote for departing from the TestCase method naming for the public test extensions API. I think names like session_setup, session_teardown, test_setup, and test_teardown would be more intuitive.

I also think it would be really useful if you could apply extensions directly to a TestCase rather than just to your test runner. This would be good for extensions where the per-test setup or teardown is a bit more expensive, such that you'd like to only do it for the tests that actually need it rather than for all tests. It's also just generally easier to use that way, since you're always defining your own TestCase subclasses, and you might not otherwise need a runner subclass. I think if we had per-TestCase extensions we might not even need the ability to globally apply them via test runner (even if we had it, most people probably wouldn't use it, as it'd still be easier to define your own TestCase subclass with some extensions applied to it and then inherit from that).

The main trick here would be that each extension would need to keep track of whether its session_setup has happened yet, and run it the first time it is encountered. (Or it could be done eagerly; the runner could examine all collected test cases to find out which extensions need their session_setup called at setup_test_environment time).

Member

carljm commented Jun 26, 2015

I'd vote for departing from the TestCase method naming for the public test extensions API. I think names like session_setup, session_teardown, test_setup, and test_teardown would be more intuitive.

I also think it would be really useful if you could apply extensions directly to a TestCase rather than just to your test runner. This would be good for extensions where the per-test setup or teardown is a bit more expensive, such that you'd like to only do it for the tests that actually need it rather than for all tests. It's also just generally easier to use that way, since you're always defining your own TestCase subclasses, and you might not otherwise need a runner subclass. I think if we had per-TestCase extensions we might not even need the ability to globally apply them via test runner (even if we had it, most people probably wouldn't use it, as it'd still be easier to define your own TestCase subclass with some extensions applied to it and then inherit from that).

The main trick here would be that each extension would need to keep track of whether its session_setup has happened yet, and run it the first time it is encountered. (Or it could be done eagerly; the runner could examine all collected test cases to find out which extensions need their session_setup called at setup_test_environment time).

@mjtamlyn

This comment has been minimized.

Show comment
Hide comment
@mjtamlyn

mjtamlyn Jun 26, 2015

Member

I'm not convinced about session_ - session has a specific meaning in the context of Django. I think environment is a better name, I'm not averse to shortening the name setup_environment though, especially if we go for the self._env.mail option over mail.outbox. test_setup also looks somewhat like a test name, so maybe just setUp?

Member

mjtamlyn commented Jun 26, 2015

I'm not convinced about session_ - session has a specific meaning in the context of Django. I think environment is a better name, I'm not averse to shortening the name setup_environment though, especially if we go for the self._env.mail option over mail.outbox. test_setup also looks somewhat like a test name, so maybe just setUp?

@carljm

This comment has been minimized.

Show comment
Hide comment
@carljm

carljm Jun 26, 2015

Member

How about setup_environment, setup_test, teardown_test, teardown_environment?

Member

carljm commented Jun 26, 2015

How about setup_environment, setup_test, teardown_test, teardown_environment?

@mjtamlyn

This comment has been minimized.

Show comment
Hide comment
@mjtamlyn

mjtamlyn Jun 26, 2015

Member

Done.

Member

mjtamlyn commented Jun 26, 2015

Done.

mjtamlyn added some commits Jun 27, 2015

Allow extensions on TestCase.
Collect them up and setup/teardown the environment only once.
@mjtamlyn

This comment has been minimized.

Show comment
Hide comment
@mjtamlyn

mjtamlyn Jun 27, 2015

Member

I've added the ability to set extensions on a TestCase as well, with eager collection during the setup phase. The classes are deduplicated, whether specified by class or by dotted path.

I think this should just need some documentation now. There are other interested ideas which could be added but they could happen later.

Member

mjtamlyn commented Jun 27, 2015

I've added the ability to set extensions on a TestCase as well, with eager collection during the setup phase. The classes are deduplicated, whether specified by class or by dotted path.

I think this should just need some documentation now. There are other interested ideas which could be added but they could happen later.

mjtamlyn added some commits Jun 27, 2015

@timgraham

This comment has been minimized.

Show comment
Hide comment
@timgraham

timgraham Dec 24, 2015

Member

Closing due to inactivity. Feel free to reopen if you return to it, Marc.

Member

timgraham commented Dec 24, 2015

Closing due to inactivity. Feel free to reopen if you return to it, Marc.

@timgraham timgraham closed this Dec 24, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment