Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Fixed #20483 -- Made TransactionTestCase faster. #1240

Closed
wants to merge 4 commits into from

6 participants

@aaugustin
Owner

No description provided.

@alex
Collaborator

How much faster?

@alex
Collaborator

I have to say this whole thing scares me quite a bit. Why isn't @overide_settings(INSTALLED_APPS=['app1', 'app2']) sufficient? So much global state :(

@aaugustin
Owner

The entire test suite passes in about 150 seconds with this patch. See the ticket for more information.

https://code.djangoproject.com/ticket/20483

@charettes
Collaborator

@alex the @overide_settings(INSTALLED_APPS=['app1', 'app2']) approach should be used once the app cache refactor is merged.

@akaariai
Collaborator

I don't believe using @override_settings would actually change anything. You still have to change global state (the used app-cache, or what the cache contains), so there is zero gain in that respect. And as mentioned in the ticket, you still have the problem that imports happen only once.

Using override_settings so that we don't have two things that do almost the same thing could be a good idea.

@akaariai
Collaborator

I think you can get rid of the nastiest parts of this changeset by requiring that all "cascades" are included in installed_apps. That is, all models which will need to be truncated or cascaded into by delete are included, and thus you will not need to do any changes there.

It is true that this is trading code complexity for testing speed. On my machine PostgreSQL tests are 4500s vs 300s between master and the original isolated_apps branch. IMO important enough to have some complexity.

@mjtamlyn mjtamlyn commented on the diff
django/contrib/auth/management/__init__.py
@@ -60,6 +60,11 @@ def _check_permission_clashing(custom, builtin, ctype):
pool.add(codename)
def create_permissions(app, created_models, verbosity, db=DEFAULT_DB_ALIAS, **kwargs):
+ try:
+ get_model('auth', 'Permission')
+ except UnavailableApp:
+ return
@mjtamlyn Collaborator
mjtamlyn added a note

Presumably we don't actually want this to fail silently? If for some reason some end user code was managing to make this throw UnavailableApp I don't think it should hide it.

@akaariai Collaborator
akaariai added a note

The reason this is done is that flush sends post_syncdb, and permission & contenttype creation are listening for the signal.

This seems to be one point more for override_settings() based approach. If done that way, permissions & contenttypes could register a listener for settings_changed and add/remove the signal listener as needed. No need to know about "app mask" at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@mjtamlyn
Collaborator

Is it possible we may be able to speed things up even more by setting available_apps = [] on tests which don't actually hit the DB therefore don't need to reset anything? This might be dangerous...

The speed improvements are very impressive though!

django/db/models/loading.py
@@ -15,6 +15,8 @@
__all__ = ('get_apps', 'get_app', 'get_models', 'get_model', 'register_models',
'load_app', 'app_cache_ready')
+class UnavailableApp(ValueError):
@ptone Collaborator
ptone added a note

minor - but any reason to subclass ValueError instead of Exception?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
django/db/models/loading.py
@@ -287,3 +321,4 @@ def register_models(self, app_label, *models):
register_models = cache.register_models
load_app = cache.load_app
app_cache_ready = cache.app_cache_ready
+set_app_mask = cache.set_app_mask
@ptone Collaborator
ptone added a note

these module level functions are only here for legacy reasons - as this is a new feature, it doesn't really need to be included in this list like this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
django/db/models/manager.py
@@ -124,6 +125,9 @@ def get_queryset(self):
"""Returns a new QuerySet object. Subclasses can override this method
to easily customize the behavior of the Manager.
"""
+ # Check that the model is available in the app cache. This is required
+ # when TransactionTestCase.available_apps is set.
+ get_model(self.model._meta.app_label, self.model._meta.model_name)
@ptone Collaborator
ptone added a note

can this be checked more directly by looking at cache.available_apps for the app_label, and raise UnavailableApp directly instead of just using get_model as a throw away here? or adding a a cache.model_available('applabel.modelname') type of method? But either way I'm assuming you are wanting to throw the exception here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@ptone ptone commented on the diff
tests/file_storage/tests.py
@@ -7,6 +7,10 @@
import sys
import tempfile
import time
+try:
@ptone Collaborator
ptone added a note

are changes to this test related to this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@akaariai akaariai commented on the diff
django/db/models/loading.py
((9 lines not shown))
# Ensure the returned list is always in the same order (with new apps
# added at the end). This avoids unstable ordering on the admin app
# list page, for example.
- apps = [(v, k) for k, v in self.app_store.items()]
- apps.sort()
- return [elt[1] for elt in apps]
+ apps = sorted(apps, key=lambda elt: elt[1])
@akaariai Collaborator
akaariai added a note

This is likely irrelevant but the sort order isn't exactly the same as before - lambda elt: elt[1], elt[0] is exactly same...

(But now that I look deeper I don't get the whole code. app_store is SortedDict, elt[1] seems to be insertion order. So, .keys() should be in the wanted order directly. But maybe better to just ignore this comment and wait for app-loading refactor to deal with this...)

@aaugustin Owner

There are no duplicates in elt[1]. This should be all right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
aaugustin added some commits
@aaugustin aaugustin Refactored file_storage tests to avoid depending on servers tests. 7a65c95
@aaugustin aaugustin Added a stealth option to flush to allow cascades.
This allows using flush on a subset of the tables without having to
manually cascade to all tables with foreign keys to the tables being
truncated, when they're known to be empty.

On databases where truncate is implemented with DELETE FROM, this
doesn't make a difference. The cascade is allowed, not mandatory.
13b7f29
@aaugustin aaugustin Added TransactionTestCase.available_apps.
This makes Django's test suite significantly faster by reducing the
number of models for which content types and permissions must be created
and tables must be flushed in each non-transactional test.

Most of the credit goes to Anssi. He got the idea and did the research.

It's documented for Django contributors and committers but it's branded
as a private API to preserve our freedom to change it in the future.

Refs #20483.
4e7dcc9
@aaugustin aaugustin Defined available_apps in relevant tests.
Fixed #20483.
815be6c
@aaugustin
Owner

Merged

@aaugustin aaugustin closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jun 10, 2013
  1. @aaugustin
  2. @aaugustin

    Added a stealth option to flush to allow cascades.

    aaugustin authored
    This allows using flush on a subset of the tables without having to
    manually cascade to all tables with foreign keys to the tables being
    truncated, when they're known to be empty.
    
    On databases where truncate is implemented with DELETE FROM, this
    doesn't make a difference. The cascade is allowed, not mandatory.
  3. @aaugustin

    Added TransactionTestCase.available_apps.

    aaugustin authored
    This makes Django's test suite significantly faster by reducing the
    number of models for which content types and permissions must be created
    and tables must be flushed in each non-transactional test.
    
    Most of the credit goes to Anssi. He got the idea and did the research.
    
    It's documented for Django contributors and committers but it's branded
    as a private API to preserve our freedom to change it in the future.
    
    Refs #20483.
  4. @aaugustin

    Defined available_apps in relevant tests.

    aaugustin authored
    Fixed #20483.
This page is out of date. Refresh to see the latest.
Showing with 328 additions and 65 deletions.
  1. +9 −0 django/contrib/admin/tests.py
  2. +12 −3 django/contrib/auth/management/__init__.py
  3. +8 −0 django/contrib/auth/tests/test_handlers.py
  4. +6 −1 django/contrib/contenttypes/management.py
  5. +5 −2 django/core/management/commands/flush.py
  6. +2 −2 django/core/management/sql.py
  7. +6 −2 django/db/backends/__init__.py
  8. +5 −2 django/db/backends/mysql/base.py
  9. +6 −6 django/db/backends/oracle/base.py
  10. +14 −5 django/db/backends/postgresql_psycopg2/operations.py
  11. +6 −6 django/db/backends/sqlite3/base.py
  12. +1 −1  django/db/models/__init__.py
  13. +50 −12 django/db/models/loading.py
  14. +30 −14 django/test/testcases.py
  15. +28 −0 docs/topics/testing/overview.txt
  16. +7 −0 tests/admin_scripts/tests.py
  17. +7 −0 tests/backends/tests.py
  18. +3 −0  tests/basic/tests.py
  19. +2 −0  tests/cache/tests.py
  20. +6 −0 tests/delete_regress/tests.py
  21. +13 −7 tests/file_storage/tests.py
  22. +7 −0 tests/file_storage/urls.py
  23. +7 −0 tests/fixtures/tests.py
  24. +3 −0  tests/fixtures_model_package/tests.py
  25. +6 −0 tests/fixtures_regress/tests.py
  26. +2 −0  tests/get_or_create/tests.py
  27. +2 −0  tests/handlers/tests.py
  28. +1 −0  tests/m2m_through_regress/tests.py
  29. +3 −0  tests/middleware/tests.py
  30. +7 −1 tests/proxy_model_inheritance/tests.py
  31. +2 −0  tests/requests/tests.py
  32. +6 −0 tests/runtests.py
  33. +2 −0  tests/select_for_update/tests.py
  34. +3 −0  tests/serializers/tests.py
  35. +8 −1 tests/servers/tests.py
  36. +2 −0  tests/settings_tests/tests.py
  37. +2 −0  tests/test_runner/tests.py
  38. +16 −0 tests/transactions/tests.py
  39. +21 −0 tests/transactions_regress/tests.py
  40. +2 −0  tests/view_tests/tests/test_i18n.py
View
9 django/contrib/admin/tests.py
@@ -5,7 +5,16 @@
from django.utils.unittest import SkipTest
from django.utils.translation import ugettext as _
+
class AdminSeleniumWebDriverTestCase(LiveServerTestCase):
+
+ available_apps = [
+ 'django.contrib.admin',
+ 'django.contrib.auth',
+ 'django.contrib.contenttypes',
+ 'django.contrib.sessions',
+ 'django.contrib.sites',
+ ]
webdriver_class = 'selenium.webdriver.firefox.webdriver.WebDriver'
@classmethod
View
15 django/contrib/auth/management/__init__.py
@@ -11,7 +11,7 @@
from django.core import exceptions
from django.core.management.base import CommandError
from django.db import DEFAULT_DB_ALIAS, router
-from django.db.models import get_models, signals
+from django.db.models import get_model, get_models, signals, UnavailableApp
from django.utils.encoding import DEFAULT_LOCALE_ENCODING
from django.utils import six
from django.utils.six.moves import input
@@ -60,6 +60,11 @@ def _check_permission_clashing(custom, builtin, ctype):
pool.add(codename)
def create_permissions(app, created_models, verbosity, db=DEFAULT_DB_ALIAS, **kwargs):
+ try:
+ get_model('auth', 'Permission')
+ except UnavailableApp:
+ return
@mjtamlyn Collaborator
mjtamlyn added a note

Presumably we don't actually want this to fail silently? If for some reason some end user code was managing to make this throw UnavailableApp I don't think it should hide it.

@akaariai Collaborator
akaariai added a note

The reason this is done is that flush sends post_syncdb, and permission & contenttype creation are listening for the signal.

This seems to be one point more for override_settings() based approach. If done that way, permissions & contenttypes could register a listener for settings_changed and add/remove the signal listener as needed. No need to know about "app mask" at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+
if not router.allow_syncdb(db, auth_app.Permission):
return
@@ -101,9 +106,13 @@ def create_permissions(app, created_models, verbosity, db=DEFAULT_DB_ALIAS, **kw
def create_superuser(app, created_models, verbosity, db, **kwargs):
- from django.core.management import call_command
+ try:
+ get_model('auth', 'Permission')
+ UserModel = get_user_model()
+ except UnavailableApp:
+ return
- UserModel = get_user_model()
+ from django.core.management import call_command
if UserModel in created_models and kwargs.get('interactive', True):
msg = ("\nYou just installed Django's auth system, which means you "
View
8 django/contrib/auth/tests/test_handlers.py
@@ -8,10 +8,18 @@
from django.test.utils import override_settings
+# This must be a TransactionTestCase because the WSGI auth handler performs
+# its own transaction management.
class ModWsgiHandlerTestCase(TransactionTestCase):
"""
Tests for the mod_wsgi authentication handler
"""
+
+ available_apps = [
+ 'django.contrib.auth',
+ 'django.contrib.contenttypes',
+ ]
+
@skipIfCustomUser
def test_check_password(self):
"""
View
7 django/contrib/contenttypes/management.py
@@ -1,6 +1,6 @@
from django.contrib.contenttypes.models import ContentType
from django.db import DEFAULT_DB_ALIAS, router
-from django.db.models import get_apps, get_models, signals
+from django.db.models import get_apps, get_model, get_models, signals, UnavailableApp
from django.utils.encoding import smart_text
from django.utils import six
from django.utils.six.moves import input
@@ -11,6 +11,11 @@ def update_contenttypes(app, created_models, verbosity=2, db=DEFAULT_DB_ALIAS, *
Creates content types for models in the given app, removing any model
entries that no longer have a matching model class.
"""
+ try:
+ get_model('contenttypes', 'ContentType')
+ except UnavailableApp:
+ return
+
if not router.allow_syncdb(db, ContentType):
return
View
7 django/core/management/commands/flush.py
@@ -32,8 +32,9 @@ def handle_noargs(self, **options):
connection = connections[db]
verbosity = int(options.get('verbosity'))
interactive = options.get('interactive')
- # 'reset_sequences' is a stealth option
+ # 'reset_sequences' and 'allow_cascade' are stealth options
reset_sequences = options.get('reset_sequences', True)
+ allow_cascade = options.get('allow_cascade', False)
self.style = no_style()
@@ -45,7 +46,9 @@ def handle_noargs(self, **options):
except ImportError:
pass
- sql_list = sql_flush(self.style, connection, only_django=True, reset_sequences=reset_sequences)
+ sql_list = sql_flush(self.style, connection, only_django=True,
+ reset_sequences=reset_sequences,
+ allow_cascade=allow_cascade)
if interactive:
confirm = input("""You have requested a flush of the database.
View
4 django/core/management/sql.py
@@ -102,7 +102,7 @@ def sql_delete(app, style, connection):
return output[::-1] # Reverse it, to deal with table dependencies.
-def sql_flush(style, connection, only_django=False, reset_sequences=True):
+def sql_flush(style, connection, only_django=False, reset_sequences=True, allow_cascade=False):
"""
Returns a list of the SQL statements used to flush the database.
@@ -114,7 +114,7 @@ def sql_flush(style, connection, only_django=False, reset_sequences=True):
else:
tables = connection.introspection.table_names()
seqs = connection.introspection.sequence_list() if reset_sequences else ()
- statements = connection.ops.sql_flush(style, tables, seqs)
+ statements = connection.ops.sql_flush(style, tables, seqs, allow_cascade)
return statements
View
8 django/db/backends/__init__.py
@@ -390,7 +390,7 @@ def constraint_checks_disabled(self):
def disable_constraint_checking(self):
"""
Backends can implement as needed to temporarily disable foreign key
- constraint checking. Should return True if the constraints were
+ constraint checking. Should return True if the constraints were
disabled and will need to be reenabled.
"""
return False
@@ -947,7 +947,7 @@ def set_time_zone_sql(self):
"""
return ''
- def sql_flush(self, style, tables, sequences):
+ def sql_flush(self, style, tables, sequences, allow_cascade=False):
"""
Returns a list of SQL statements required to remove all data from
the given database tables (without actually removing the tables
@@ -958,6 +958,10 @@ def sql_flush(self, style, tables, sequences):
The `style` argument is a Style object as returned by either
color_style() or no_style() in django.core.management.color.
+
+ The `allow_cascade` argument determines whether truncation may cascade
+ to tables with foreign keys pointing the tables being truncated.
+ PostgreSQL requires a cascade even if these tables are empty.
"""
raise NotImplementedError()
View
7 django/db/backends/mysql/base.py
@@ -298,14 +298,17 @@ def quote_name(self, name):
def random_function_sql(self):
return 'RAND()'
- def sql_flush(self, style, tables, sequences):
+ def sql_flush(self, style, tables, sequences, allow_cascade=False):
# NB: The generated SQL below is specific to MySQL
# 'TRUNCATE x;', 'TRUNCATE y;', 'TRUNCATE z;'... style SQL statements
# to clear all tables of all data
if tables:
sql = ['SET FOREIGN_KEY_CHECKS = 0;']
for table in tables:
- sql.append('%s %s;' % (style.SQL_KEYWORD('TRUNCATE'), style.SQL_FIELD(self.quote_name(table))))
+ sql.append('%s %s;' % (
+ style.SQL_KEYWORD('TRUNCATE'),
+ style.SQL_FIELD(self.quote_name(table)),
+ ))
sql.append('SET FOREIGN_KEY_CHECKS = 1;')
sql.extend(self.sequence_reset_by_name_sql(style, sequences))
return sql
View
12 django/db/backends/oracle/base.py
@@ -339,17 +339,17 @@ def savepoint_create_sql(self, sid):
def savepoint_rollback_sql(self, sid):
return convert_unicode("ROLLBACK TO SAVEPOINT " + self.quote_name(sid))
- def sql_flush(self, style, tables, sequences):
+ def sql_flush(self, style, tables, sequences, allow_cascade=False):
# Return a list of 'TRUNCATE x;', 'TRUNCATE y;',
# 'TRUNCATE z;'... style SQL statements
if tables:
# Oracle does support TRUNCATE, but it seems to get us into
# FK referential trouble, whereas DELETE FROM table works.
- sql = ['%s %s %s;' % \
- (style.SQL_KEYWORD('DELETE'),
- style.SQL_KEYWORD('FROM'),
- style.SQL_FIELD(self.quote_name(table)))
- for table in tables]
+ sql = ['%s %s %s;' % (
+ style.SQL_KEYWORD('DELETE'),
+ style.SQL_KEYWORD('FROM'),
+ style.SQL_FIELD(self.quote_name(table))
+ ) for table in tables]
# Since we've just deleted all the rows, running our sequence
# ALTER code will reset the sequence to 0.
sql.extend(self.sequence_reset_by_name_sql(style, sequences))
View
19 django/db/backends/postgresql_psycopg2/operations.py
@@ -101,15 +101,24 @@ def quote_name(self, name):
def set_time_zone_sql(self):
return "SET TIME ZONE %s"
- def sql_flush(self, style, tables, sequences):
+ def sql_flush(self, style, tables, sequences, allow_cascade=False):
if tables:
# Perform a single SQL 'TRUNCATE x, y, z...;' statement. It allows
# us to truncate tables referenced by a foreign key in any other
# table.
- sql = ['%s %s;' % \
- (style.SQL_KEYWORD('TRUNCATE'),
- style.SQL_FIELD(', '.join([self.quote_name(table) for table in tables]))
- )]
+ tables_sql = ', '.join(
+ style.SQL_FIELD(self.quote_name(table)) for table in tables)
+ if allow_cascade:
+ sql = ['%s %s %s;' % (
+ style.SQL_KEYWORD('TRUNCATE'),
+ tables_sql,
+ style.SQL_KEYWORD('CASCADE'),
+ )]
+ else:
+ sql = ['%s %s;' % (
+ style.SQL_KEYWORD('TRUNCATE'),
+ tables_sql,
+ )]
sql.extend(self.sequence_reset_by_name_sql(style, sequences))
return sql
else:
View
12 django/db/backends/sqlite3/base.py
@@ -209,15 +209,15 @@ def quote_name(self, name):
def no_limit_value(self):
return -1
- def sql_flush(self, style, tables, sequences):
+ def sql_flush(self, style, tables, sequences, allow_cascade=False):
# NB: The generated SQL below is specific to SQLite
# Note: The DELETE FROM... SQL generated below works for SQLite databases
# because constraints don't exist
- sql = ['%s %s %s;' % \
- (style.SQL_KEYWORD('DELETE'),
- style.SQL_KEYWORD('FROM'),
- style.SQL_FIELD(self.quote_name(table))
- ) for table in tables]
+ sql = ['%s %s %s;' % (
+ style.SQL_KEYWORD('DELETE'),
+ style.SQL_KEYWORD('FROM'),
+ style.SQL_FIELD(self.quote_name(table))
+ ) for table in tables]
# Note: No requirement for reset of auto-incremented indices (cf. other
# sql_flush() implementations). Just return SQL at this point
return sql
View
2  django/db/models/__init__.py
@@ -1,7 +1,7 @@
from functools import wraps
from django.core.exceptions import ObjectDoesNotExist, ImproperlyConfigured
-from django.db.models.loading import get_apps, get_app_paths, get_app, get_models, get_model, register_models
+from django.db.models.loading import get_apps, get_app_paths, get_app, get_models, get_model, register_models, UnavailableApp
from django.db.models.query import Q
from django.db.models.expressions import F
from django.db.models.manager import Manager
View
62 django/db/models/loading.py
@@ -15,6 +15,8 @@
__all__ = ('get_apps', 'get_app', 'get_models', 'get_model', 'register_models',
'load_app', 'app_cache_ready')
+class UnavailableApp(Exception):
+ pass
class AppCache(object):
"""
@@ -43,6 +45,7 @@ class AppCache(object):
postponed=[],
nesting_level=0,
_get_models_cache={},
+ available_apps=None,
)
def __init__(self):
@@ -135,12 +138,17 @@ def get_apps(self):
"""
self._populate()
+ apps = self.app_store.items()
+ if self.available_apps is not None:
+ apps = [elt for elt in apps
+ if self._label_for(elt[0]) in self.available_apps]
+
# Ensure the returned list is always in the same order (with new apps
# added at the end). This avoids unstable ordering on the admin app
# list page, for example.
- apps = [(v, k) for k, v in self.app_store.items()]
- apps.sort()
- return [elt[1] for elt in apps]
+ apps = sorted(apps, key=lambda elt: elt[1])
@akaariai Collaborator
akaariai added a note

This is likely irrelevant but the sort order isn't exactly the same as before - lambda elt: elt[1], elt[0] is exactly same...

(But now that I look deeper I don't get the whole code. app_store is SortedDict, elt[1] seems to be insertion order. So, .keys() should be in the wanted order directly. But maybe better to just ignore this comment and wait for app-loading refactor to deal with this...)

@aaugustin Owner

There are no duplicates in elt[1]. This should be all right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+
+ return [elt[0] for elt in apps]
def get_app_paths(self):
"""
@@ -161,8 +169,12 @@ def get_app_paths(self):
def get_app(self, app_label, emptyOK=False):
"""
- Returns the module containing the models for the given app_label. If
- the app has no models in it and 'emptyOK' is True, returns None.
+ Returns the module containing the models for the given app_label.
+
+ Returns None if the app has no models in it and emptyOK is True.
+
+ Raises UnavailableApp when set_available_apps() in in effect and
+ doesn't include app_label.
"""
self._populate()
imp.acquire_lock()
@@ -170,12 +182,11 @@ def get_app(self, app_label, emptyOK=False):
for app_name in settings.INSTALLED_APPS:
if app_label == app_name.split('.')[-1]:
mod = self.load_app(app_name, False)
- if mod is None:
- if emptyOK:
- return None
+ if mod is None and not emptyOK:
raise ImproperlyConfigured("App with label %s is missing a models.py module." % app_label)
- else:
- return mod
+ if self.available_apps is not None and app_label not in self.available_apps:
+ raise UnavailableApp("App with label %s isn't available." % app_label)
+ return mod
raise ImproperlyConfigured("App with label %s could not be found" % app_label)
finally:
imp.release_lock()
@@ -209,8 +220,13 @@ def get_models(self, app_mod=None,
include_swapped, they will be.
"""
cache_key = (app_mod, include_auto_created, include_deferred, only_installed, include_swapped)
+ model_list = None
try:
- return self._get_models_cache[cache_key]
+ model_list = self._get_models_cache[cache_key]
+ if self.available_apps is not None and only_installed:
+ model_list = [m for m in model_list
+ if m._meta.app_label in self.available_apps]
+ return model_list
except KeyError:
pass
self._populate()
@@ -235,6 +251,9 @@ def get_models(self, app_mod=None,
(not model._meta.swapped or include_swapped))
)
self._get_models_cache[cache_key] = model_list
+ if self.available_apps is not None and only_installed:
+ model_list = [m for m in model_list
+ if m._meta.app_label in self.available_apps]
return model_list
def get_model(self, app_label, model_name,
@@ -244,12 +263,21 @@ def get_model(self, app_label, model_name,
model_name.
Returns None if no model is found.
+
+ Raises UnavailableApp when set_available_apps() in in effect and
+ doesn't include app_label.
"""
if seed_cache:
self._populate()
if only_installed and app_label not in self.app_labels:
return None
- return self.app_models.get(app_label, SortedDict()).get(model_name.lower())
+ if (self.available_apps is not None and only_installed
+ and app_label not in self.available_apps):
+ raise UnavailableApp("App with label %s isn't available." % app_label)
+ try:
+ return self.app_models[app_label][model_name.lower()]
+ except KeyError:
+ return None
def register_models(self, app_label, *models):
"""
@@ -274,6 +302,16 @@ def register_models(self, app_label, *models):
model_dict[model_name] = model
self._get_models_cache.clear()
+ def set_available_apps(self, available):
+ if not set(available).issubset(set(settings.INSTALLED_APPS)):
+ extra = set(available) - set(settings.INSTALLED_APPS)
+ raise ValueError("Available apps isn't a subset of installed "
+ "apps, extra apps: " + ", ".join(extra))
+ self.available_apps = set(app.rsplit('.', 1)[-1] for app in available)
+
+ def unset_available_apps(self):
+ self.available_apps = None
+
cache = AppCache()
# These methods were always module level, so are kept that way for backwards
View
44 django/test/testcases.py
@@ -28,6 +28,7 @@
WSGIServerException)
from django.core.urlresolvers import clear_url_caches, set_urlconf
from django.db import connection, connections, DEFAULT_DB_ALIAS, transaction
+from django.db.models.loading import cache
from django.forms.fields import CharField
from django.http import QueryDict
from django.test.client import Client
@@ -725,6 +726,9 @@ class TransactionTestCase(SimpleTestCase):
# test case
reset_sequences = False
+ # Subclasses can enable only a subset of apps for faster tests
+ available_apps = None
+
def _pre_setup(self):
"""Performs any pre-test setup. This includes:
@@ -733,7 +737,14 @@ def _pre_setup(self):
named fixtures.
"""
super(TransactionTestCase, self)._pre_setup()
- self._fixture_setup()
+ if self.available_apps is not None:
+ cache.set_available_apps(self.available_apps)
+ try:
+ self._fixture_setup()
+ except Exception:
+ if self.available_apps is not None:
+ cache.unset_available_apps()
+ raise
def _databases_names(self, include_mirrors=True):
# If the test case has a multi_db=True flag, act on all databases,
@@ -775,22 +786,27 @@ def _post_teardown(self):
* Force closing the connection, so that the next test gets
a clean cursor.
"""
- self._fixture_teardown()
- super(TransactionTestCase, self)._post_teardown()
- # Some DB cursors include SQL statements as part of cursor
- # creation. If you have a test that does rollback, the effect
- # of these statements is lost, which can effect the operation
- # of tests (e.g., losing a timezone setting causing objects to
- # be created with the wrong time).
- # To make sure this doesn't happen, get a clean connection at the
- # start of every test.
- for conn in connections.all():
- conn.close()
+ try:
+ self._fixture_teardown()
+ super(TransactionTestCase, self)._post_teardown()
+ # Some DB cursors include SQL statements as part of cursor
+ # creation. If you have a test that does rollback, the effect of
+ # these statements is lost, which can effect the operation of
+ # tests (e.g., losing a timezone setting causing objects to be
+ # created with the wrong time). To make sure this doesn't happen,
+ # get a clean connection at the start of every test.
+ for conn in connections.all():
+ conn.close()
+ finally:
+ cache.unset_available_apps()
def _fixture_teardown(self):
+ # Allow TRUNCATE ... CASCADE when flushing only a subset of the apps
+ allow_cascade = self.available_apps is not None
for db_name in self._databases_names(include_mirrors=False):
- call_command('flush', verbosity=0, interactive=False, database=db_name,
- skip_validation=True, reset_sequences=False)
+ call_command('flush', verbosity=0, interactive=False,
+ database=db_name, skip_validation=True,
+ reset_sequences=False, allow_cascade=allow_cascade)
def assertQuerysetEqual(self, qs, values, transform=repr, ordered=True):
items = six.moves.map(transform, qs)
View
28 docs/topics/testing/overview.txt
@@ -985,6 +985,34 @@ to test the effects of commit and rollback:
Using ``reset_sequences = True`` will slow down the test, since the primary
key reset is an relatively expensive database operation.
+.. attribute:: TransactionTestCase.available_apps
+
+ .. warning::
+
+ This attribute is a private API. It may be changed or removed without
+ a deprecation period in the future, for instance to accomodate changes
+ in application loading.
+
+ It's used to optimize Django's own test suite, which contains hundreds
+ of models but no relations between models in different applications.
+
+ .. versionadded:: 1.6
+
+ By default, ``available_apps`` is set to ``None`` and has no effect.
+ Setting it to a list of applications tells Django to behave as if only the
+ models from these applications were available:
+
+ - Before each test, Django creates content types and permissions only for
+ these models.
+ - After each test, Django flushes only the corresponding tables. However,
+ at the database level, truncation may cascade to other related models,
+ even if they aren't in ``available_apps``.
+
+ Since the database isn't fully flushed, if a test creates instances of
+ models not included in ``available_apps``, they will leak and they may
+ cause unrelated tests to fail. Be careful with tests that use sessions;
+ the default session engine stores them in the database.
+
TestCase
~~~~~~~~
View
7 tests/admin_scripts/tests.py
@@ -1474,6 +1474,13 @@ def test_option_then_setting_then_option(self):
class StartProject(LiveServerTestCase, AdminScriptTestCase):
+ available_apps = [
+ 'admin_scripts',
+ 'django.contrib.auth',
+ 'django.contrib.contenttypes',
+ 'django.contrib.sessions',
+ ]
+
def test_wrong_args(self):
"Make sure passing the wrong kinds of arguments raises a CommandError"
out, err = self.run_django_admin(['startproject'])
View
7 tests/backends/tests.py
@@ -345,6 +345,8 @@ def test_connect_and_rollback(self):
# connection would implicitly rollback and cause problems during teardown.
class ConnectionCreatedSignalTest(TransactionTestCase):
+ available_apps = []
+
# Unfortunately with sqlite3 the in-memory test database cannot be closed,
# and so it cannot be re-opened during testing.
@skipUnlessDBFeature('test_db_allows_multiple_connections')
@@ -514,6 +516,8 @@ def test_duplicate_table_error(self):
# verify if its type is django.database.db.IntegrityError.
class FkConstraintsTests(TransactionTestCase):
+ available_apps = ['backends']
+
def setUp(self):
# Create a Reporter.
self.r = models.Reporter.objects.create(first_name='John', last_name='Smith')
@@ -777,6 +781,9 @@ def test_zero_as_autoval(self):
class DBConstraintTestCase(TransactionTestCase):
+
+ available_apps = ['backends']
+
def test_can_reference_existant(self):
obj = models.Object.objects.create()
ref = models.ObjectReference.objects.create(obj=obj)
View
3  tests/basic/tests.py
@@ -686,6 +686,9 @@ def test_ticket_20278(self):
class ConcurrentSaveTests(TransactionTestCase):
+
+ available_apps = ['basic']
+
@skipUnlessDBFeature('test_db_allows_multiple_connections')
def test_concurrent_delete_with_save(self):
"""
View
2  tests/cache/tests.py
@@ -827,6 +827,8 @@ def custom_key_func(key, key_prefix, version):
class DBCacheTests(BaseCacheTests, TransactionTestCase):
+
+ available_apps = []
backend_name = 'django.core.cache.backends.db.DatabaseCache'
def setUp(self):
View
6 tests/delete_regress/tests.py
@@ -16,6 +16,9 @@
# Can't run this test under SQLite, because you can't
# get two connections to an in-memory database.
class DeleteLockingTest(TransactionTestCase):
+
+ available_apps = ['delete_regress']
+
def setUp(self):
# Create a second connection to the default database
new_connections = ConnectionHandler(settings.DATABASES)
@@ -107,6 +110,9 @@ def test_15776(self):
class DeleteCascadeTransactionTests(TransactionTestCase):
+
+ available_apps = ['delete_regress']
+
def test_inheritance(self):
"""
Auto-created many-to-many through tables referencing a parent model are
View
20 tests/file_storage/tests.py
@@ -7,6 +7,10 @@
import sys
import tempfile
import time
+try:
@ptone Collaborator
ptone added a note

are changes to this test related to this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ from urllib.request import urlopen
+except ImportError: # Python 2
+ from urllib2 import urlopen
import zlib
from datetime import datetime, timedelta
from io import BytesIO
@@ -22,12 +26,11 @@
from django.core.files.images import get_image_dimensions
from django.core.files.storage import FileSystemStorage, get_storage_class
from django.core.files.uploadedfile import UploadedFile
-from django.test import SimpleTestCase
+from django.test import LiveServerTestCase, SimpleTestCase
from django.utils import six
from django.utils import unittest
from django.utils._os import upath
from django.test.utils import override_settings
-from servers.tests import LiveServerBase
try:
from django.utils.image import Image
@@ -613,10 +616,15 @@ def test_noname_file_default_name(self):
def test_noname_file_get_size(self):
self.assertEqual(File(BytesIO(b'A file with no name')).size, 19)
-class FileLikeObjectTestCase(LiveServerBase):
+
+class FileLikeObjectTestCase(LiveServerTestCase):
"""
Test file-like objects (#15644).
"""
+
+ available_apps = []
+ urls = 'file_storage.urls'
+
def setUp(self):
self.temp_dir = tempfile.mkdtemp()
self.storage = FileSystemStorage(location=self.temp_dir)
@@ -628,12 +636,10 @@ def test_urllib2_urlopen(self):
"""
Test the File storage API with a file like object coming from urllib2.urlopen()
"""
-
- file_like_object = self.urlopen('/example_view/')
+ file_like_object = urlopen(self.live_server_url + '/')
f = File(file_like_object)
stored_filename = self.storage.save("remote_file.html", f)
- remote_file = self.urlopen('/example_view/')
-
+ remote_file = urlopen(self.live_server_url + '/')
with self.storage.open(stored_filename) as stored_file:
self.assertEqual(stored_file.read(), remote_file.read())
View
7 tests/file_storage/urls.py
@@ -0,0 +1,7 @@
+from django.conf.urls import patterns, url
+from django.http import HttpResponse
+
+
+urlpatterns = patterns('',
+ url(r'^$', lambda req: HttpResponse('example view')),
+)
View
7 tests/fixtures/tests.py
@@ -354,6 +354,13 @@ def test_output_formats(self):
class FixtureTransactionTests(DumpDataAssertMixin, TransactionTestCase):
+ available_apps = [
+ 'fixtures',
+ 'django.contrib.contenttypes',
+ 'django.contrib.auth',
+ 'django.contrib.sites',
+ ]
+
@skipUnlessDBFeature('supports_forward_references')
def test_format_discovery(self):
# Load fixture 1 again, using format discovery
View
3  tests/fixtures_model_package/tests.py
@@ -26,6 +26,9 @@ def testClassFixtures(self):
class TestNoInitialDataLoading(TransactionTestCase):
+
+ available_apps = ['fixtures_model_package']
+
def test_syncdb(self):
transaction.set_autocommit(False)
try:
View
6 tests/fixtures_regress/tests.py
@@ -685,6 +685,12 @@ def test_normal_pk(self):
class TestTicket11101(TransactionTestCase):
+ available_apps = [
+ 'fixtures_regress',
+ 'django.contrib.auth',
+ 'django.contrib.contenttypes',
+ ]
+
def ticket_11101(self):
management.call_command(
'loaddata',
View
2  tests/get_or_create/tests.py
@@ -94,6 +94,8 @@ def test_get_or_create_empty(self):
class GetOrCreateTransactionTests(TransactionTestCase):
+ available_apps = ['get_or_create']
+
def test_get_or_create_integrityerror(self):
# Regression test for #15117. Requires a TransactionTestCase on
# databases that delay integrity checks until the end of transactions,
View
2  tests/handlers/tests.py
@@ -37,6 +37,8 @@ def test_bad_path_info(self):
class TransactionsPerRequestTests(TransactionTestCase):
+
+ available_apps = []
urls = 'handlers.urls'
def test_no_transaction(self):
View
1  tests/m2m_through_regress/tests.py
@@ -10,6 +10,7 @@
class M2MThroughTestCase(TestCase):
+
def test_everything(self):
bob = Person.objects.create(name="Bob")
jim = Person.objects.create(name="Jim")
View
3  tests/middleware/tests.py
@@ -707,6 +707,9 @@ class TransactionMiddlewareTest(IgnorePendingDeprecationWarningsMixin, Transacti
"""
Test the transaction middleware.
"""
+
+ available_apps = ['middleware']
+
def setUp(self):
super(TransactionMiddlewareTest, self).setUp()
self.request = HttpRequest()
View
8 tests/proxy_model_inheritance/tests.py
@@ -22,6 +22,8 @@ class ProxyModelInheritanceTests(TransactionTestCase):
apps and calls syncdb, then verifies that the table has been created.
"""
+ available_apps = []
+
def setUp(self):
self.old_sys_path = sys.path[:]
sys.path.append(os.path.dirname(os.path.abspath(upath(__file__))))
@@ -38,7 +40,11 @@ def tearDown(self):
del cache.app_models['app2']
def test_table_exists(self):
- call_command('syncdb', verbosity=0)
+ try:
+ cache.set_available_apps(settings.INSTALLED_APPS)
+ call_command('syncdb', verbosity=0)
+ finally:
+ cache.unset_available_apps()
from .app1.models import ProxyModel
from .app2.models import NiceModel
self.assertEqual(NiceModel.objects.all().count(), 0)
View
2  tests/requests/tests.py
@@ -683,6 +683,8 @@ def read(self, len=0):
"Cannot establish two connections to an in-memory SQLite database.")
class DatabaseConnectionHandlingTests(TransactionTestCase):
+ available_apps = []
+
def setUp(self):
# Use a temporary connection to avoid messing with the main one.
self._old_default_connection = connections['default']
View
6 tests/runtests.py
@@ -72,6 +72,12 @@ def get_installed():
def setup(verbosity, test_labels):
from django.conf import settings
from django.db.models.loading import get_apps, load_app
+ from django.test.testcases import TransactionTestCase, TestCase
+
+ # Force declaring available_apps in TransactionTestCase for faster tests.
+ TransactionTestCase.available_apps = []
+ TestCase.available_apps = None
+
state = {
'INSTALLED_APPS': settings.INSTALLED_APPS,
'ROOT_URLCONF': getattr(settings, "ROOT_URLCONF", ""),
View
2  tests/select_for_update/tests.py
@@ -23,6 +23,8 @@
class SelectForUpdateTests(TransactionTestCase):
+ available_apps = ['select_for_update']
+
def setUp(self):
transaction.enter_transaction_management()
self.person = Person.objects.create(name='Reinhardt')
View
3  tests/serializers/tests.py
@@ -259,6 +259,9 @@ def test_pkless_serialized_strings(self):
class SerializersTransactionTestBase(object):
+
+ available_apps = ['serializers']
+
def test_forward_refs(self):
"""
Tests that objects ids can be referenced before they are
View
9 tests/servers/tests.py
@@ -30,8 +30,15 @@
class LiveServerBase(LiveServerTestCase):
- urls = 'servers.urls'
+
+ available_apps = [
+ 'servers',
+ 'django.contrib.auth',
+ 'django.contrib.contenttypes',
+ 'django.contrib.sessions',
+ ]
fixtures = ['testdata.json']
+ urls = 'servers.urls'
@classmethod
def setUpClass(cls):
View
2  tests/settings_tests/tests.py
@@ -12,6 +12,8 @@
@override_settings(TEST='override')
class FullyDecoratedTranTestCase(TransactionTestCase):
+ available_apps = []
+
def test_override(self):
self.assertEqual(settings.TEST, 'override')
View
2  tests/test_runner/tests.py
@@ -318,6 +318,8 @@ class AutoIncrementResetTest(TransactionTestCase):
that AutoField values start from 1 for each transactional test case.
"""
+ available_apps = ['test_runner']
+
reset_sequences = True
@skipUnlessDBFeature('supports_sequence_reset')
View
16 tests/transactions/tests.py
@@ -26,6 +26,8 @@ class AtomicTests(TransactionTestCase):
syntax and the bulk of the tests use the context manager syntax.
"""
+ available_apps = ['transactions']
+
def test_decorator_syntax_commit(self):
@transaction.atomic
def make_reporter():
@@ -232,6 +234,8 @@ def tearDown(self):
class AtomicMergeTests(TransactionTestCase):
"""Test merging transactions with savepoint=False."""
+ available_apps = ['transactions']
+
def test_merged_outer_rollback(self):
with transaction.atomic():
Reporter.objects.create(first_name="Tintin")
@@ -286,6 +290,8 @@ def test_merged_outer_rollback_after_inner_failure_and_inner_success(self):
"'atomic' requires transactions and savepoints.")
class AtomicErrorsTests(TransactionTestCase):
+ available_apps = []
+
def test_atomic_prevents_setting_autocommit(self):
autocommit = transaction.get_autocommit()
with transaction.atomic():
@@ -311,6 +317,8 @@ def test_atomic_prevents_calling_transaction_management_methods(self):
class AtomicMiscTests(TransactionTestCase):
+ available_apps = []
+
def test_wrap_callable_instance(self):
# Regression test for #20028
class Callable(object):
@@ -322,6 +330,8 @@ def __call__(self):
class TransactionTests(IgnorePendingDeprecationWarningsMixin, TransactionTestCase):
+ available_apps = ['transactions']
+
def create_a_reporter_then_fail(self, first, last):
a = Reporter(first_name=first, last_name=last)
a.save()
@@ -477,6 +487,9 @@ def test_manually_managed_with_using(self):
class TransactionRollbackTests(IgnorePendingDeprecationWarningsMixin, TransactionTestCase):
+
+ available_apps = ['transactions']
+
def execute_bad_sql(self):
cursor = connection.cursor()
cursor.execute("INSERT INTO transactions_reporter (first_name, last_name) VALUES ('Douglas', 'Adams');")
@@ -494,6 +507,9 @@ def test_bad_sql(self):
transaction.rollback()
class TransactionContextManagerTests(IgnorePendingDeprecationWarningsMixin, TransactionTestCase):
+
+ available_apps = ['transactions']
+
def create_reporter_and_fail(self):
Reporter.objects.create(first_name="Bob", last_name="Holtzman")
raise Exception
View
21 tests/transactions_regress/tests.py
@@ -10,6 +10,9 @@
from .models import Mod, M2mA, M2mB, SubMod
class ModelInheritanceTests(TransactionTestCase):
+
+ available_apps = ['transactions_regress']
+
def test_save(self):
# First, create a SubMod, then try to save another with conflicting
# cnt field. The problem was that transactions were committed after
@@ -33,6 +36,13 @@ class TestTransactionClosing(IgnorePendingDeprecationWarningsMixin, TransactionT
when they should be, and aren't left pending after operations
have been performed in them. Refs #9964.
"""
+
+ available_apps = [
+ 'transactions_regress',
+ 'django.contrib.auth',
+ 'django.contrib.contenttypes',
+ ]
+
def test_raw_committed_on_success(self):
"""
Make sure a transaction consisting of raw SQL execution gets
@@ -188,6 +198,9 @@ class TestNewConnection(IgnorePendingDeprecationWarningsMixin, TransactionTestCa
"""
Check that new connections don't have special behaviour.
"""
+
+ available_apps = ['transactions_regress']
+
def setUp(self):
self._old_backend = connections[DEFAULT_DB_ALIAS]
settings = self._old_backend.settings_dict.copy()
@@ -235,6 +248,9 @@ class TestPostgresAutocommitAndIsolation(IgnorePendingDeprecationWarningsMixin,
is restored after entering and leaving transaction management.
Refs #16047, #18130.
"""
+
+ available_apps = ['transactions_regress']
+
def setUp(self):
from psycopg2.extensions import (ISOLATION_LEVEL_AUTOCOMMIT,
ISOLATION_LEVEL_SERIALIZABLE,
@@ -311,6 +327,9 @@ def test_enter_autocommit(self):
class TestManyToManyAddTransaction(IgnorePendingDeprecationWarningsMixin, TransactionTestCase):
+
+ available_apps = ['transactions_regress']
+
def test_manyrelated_add_commit(self):
"Test for https://code.djangoproject.com/ticket/16818"
a = M2mA.objects.create()
@@ -327,6 +346,8 @@ def test_manyrelated_add_commit(self):
class SavepointTest(IgnorePendingDeprecationWarningsMixin, TransactionTestCase):
+ available_apps = ['transactions_regress']
+
@skipIf(connection.vendor == 'sqlite',
"SQLite doesn't support savepoints in managed mode")
@skipUnlessDBFeature('uses_savepoints')
View
2  tests/view_tests/tests/test_i18n.py
@@ -183,6 +183,8 @@ def testI18NWithLocalePaths(self):
@unittest.skipIf(skip_selenium, 'Selenium tests not requested')
@unittest.skipUnless(firefox, 'Selenium not installed')
class JavascriptI18nTests(LiveServerTestCase):
+
+ available_apps = []
urls = 'view_tests.urls'
@classmethod
Something went wrong with that request. Please try again.