Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Prevent Oracle from changing field.null to True #19

Closed
wants to merge 1 commit into from

2 participants

@akaariai
Collaborator

Fixed #17957 -- when using Oracle and character fields, the fields
were set null = True to ease the handling of empty strings. This
caused problems when using multiple databases from different vendors,
or when the character field happened to be also a primary key.

The handling was changed so that NOT NULL is not emitted on Oracle
even if field.null = False, and field.null is not touched otherwise.

Thanks to bhuztez for the report, ramiro for triaging & comments and
ikelly for the patch.

@akaariai
Collaborator

I am still unsure how I should commit code to github - so here is another pull request fixing Oracle specific stuff.

@alex
Collaborator

Do all tests still pass on Oracle?

@akaariai
Collaborator

All tests haven't passed in a while on Oracle. I'm working on fixes...

At least one more test passes with the patch applied (test_runner tests, related to character field primary keys having null=True). I haven't done a comparison between full test suite with and without this patch as it takes a lot of time to do that. I can leave the full test suite running for the night if needed.

@alex
Collaborator

Ok, FWIW my concern was whether this was needed so the correct INNER vs. OUTER JOINs were generated.

@akaariai
Collaborator

Worth a test at least (if not already tested somewhere). It seems this could cause some weird behavior there. Should I close this pull request while I check this? I am not sure how long it will take to check and possibly correct errors caused by this...

@alex
Collaborator

I think it's fine to leave the pull request open.

@akaariai
Collaborator

I have investigated the queryset usage of field.null issue a little, and it seems clear that the change could result in some queries breaking when using Oracle. It is a bit scary to use Oracle's view of if the field is nullable when constructing the query because a later .using() clause could then lead to problems.

Luckily it seems that both of the field.null uses in sql/query.py will work correctly if field.null is "guessed" to true even if it is not true for the backend. So, I am going to update the patch so that sql/query.py will treat fields where (empty strings are allowed and connection is Oracle) as nullable. To me it seems in the worst case this generates slightly less performant queries when a query is generated using Oracle, but executed using some other backend - in other words, we really don't need to care about this issue too much...

I ran the full test suite during the night, before there were 3 failures, with patch only two.

@akaariai akaariai Prevent Oracle from changing field.null to True
Fixed #17957 -- when using Oracle and character fields, the fields
were set null = True to ease the handling of empty strings. This
caused problems when using multiple databases from different vendors,
or when the character field happened to be also a primary key.

The handling was changed so that NOT NULL is not emitted on Oracle
even if field.null = False, and field.null is not touched otherwise.

Thanks to bhuztez for the report, ramiro for triaging & comments,
ikelly for the patch and alex for reviewing.
0d45eef
@akaariai
Collaborator

I added some tests to check that character fields are handled correctly in query.py. The field.null handling is done using is_nullable(field). It has to use the DEFAULT_DB_ALIAS when deciding if the field should be treated as nullable as the sql.QuerySet doesn't know which connection is used. The field.null handling is thus identical to pre-patch situation. That is, slightly broken in weird corner cases.

@alex
Collaborator

Ok now I remember this, unfortunately your analysis exactly matches what mine was. That is: this decision should be made later, but that's a ton of work. Patch LGTM feel free to commit!

@akaariai
Collaborator

I pushed this manually -- closing.

@akaariai akaariai closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Apr 29, 2012
  1. @akaariai

    Prevent Oracle from changing field.null to True

    akaariai authored
    Fixed #17957 -- when using Oracle and character fields, the fields
    were set null = True to ease the handling of empty strings. This
    caused problems when using multiple databases from different vendors,
    or when the character field happened to be also a primary key.
    
    The handling was changed so that NOT NULL is not emitted on Oracle
    even if field.null = False, and field.null is not touched otherwise.
    
    Thanks to bhuztez for the report, ramiro for triaging & comments,
    ikelly for the patch and alex for reviewing.
This page is out of date. Refresh to see the latest.
View
8 django/db/backends/creation.py
@@ -50,7 +50,13 @@ def sql_create_model(self, model, style, known_models=set()):
# Make the definition (e.g. 'foo VARCHAR(30)') for this field.
field_output = [style.SQL_FIELD(qn(f.column)),
style.SQL_COLTYPE(col_type)]
- if not f.null:
+ # Oracle treats the empty string ('') as null, so coerce the null
+ # option whenever '' is a possible value.
+ null = f.null
+ if (f.empty_strings_allowed and not f.primary_key and
+ self.connection.features.interprets_empty_strings_as_nulls):
+ null = True
+ if not null:
field_output.append(style.SQL_KEYWORD('NOT NULL'))
if f.primary_key:
field_output.append(style.SQL_KEYWORD('PRIMARY KEY'))
View
5 django/db/models/fields/__init__.py
@@ -85,11 +85,6 @@ def __init__(self, verbose_name=None, name=None, primary_key=False,
self.primary_key = primary_key
self.max_length, self._unique = max_length, unique
self.blank, self.null = blank, null
- # Oracle treats the empty string ('') as null, so coerce the null
- # option whenever '' is a possible value.
- if (self.empty_strings_allowed and
- connection.features.interprets_empty_strings_as_nulls):
- self.null = True
self.rel = rel
self.default = default
self.editable = editable
View
23 django/db/models/sql/query.py
@@ -1193,7 +1193,7 @@ def add_filter(self, filter_expr, connector=AND, negate=False, trim=False,
entry.negate()
self.where.add(entry, AND)
break
- if field.null:
+ if self.is_nullable(field):
# In SQL NULL = anyvalue returns unknown, and NOT unknown
# is still unknown. However, in Python None = anyvalue is False
# (and not False is True...), and we want to return this Python's
@@ -1396,7 +1396,8 @@ def setup_joins(self, names, opts, alias, dupe_multis, allow_many=True,
opts, target)
alias = self.join((alias, table, from_col, to_col),
- exclusions=exclusions, nullable=field.null)
+ exclusions=exclusions,
+ nullable=self.is_nullable(field))
joins.append(alias)
else:
# Non-relation fields.
@@ -1946,6 +1947,24 @@ def set_start(self, start):
self.select = [(select_alias, select_col)]
self.remove_inherited_models()
+ def is_nullable(self, field):
+ """
+ A helper to check if the given field should be treated as nullable.
+
+ Some backends treat '' as null and Django treats such fields as
+ nullable for those backends. In such situations field.null can be
+ False even if we should treat the field as nullable.
+ """
+ # We need to use DEFAULT_DB_ALIAS here, as QuerySet does not have
+ # (nor should it have) knowledge of which connection is going to be
+ # used. The proper fix would be to defer all decisions where
+ # is_nullable() is needed to the compiler stage, but that is not easy
+ # to do currently.
+ if ((connections[DEFAULT_DB_ALIAS].features.interprets_empty_strings_as_nulls)
+ and field.empty_strings_allowed):
+ return True
+ else:
+ return field.null
def get_order_dir(field, default='ASC'):
"""
View
10 docs/ref/databases.txt
@@ -685,11 +685,11 @@ NULL and empty strings
Django generally prefers to use the empty string ('') rather than
NULL, but Oracle treats both identically. To get around this, the
-Oracle backend coerces the ``null=True`` option on fields that have
-the empty string as a possible value. When fetching from the database,
-it is assumed that a NULL value in one of these fields really means
-the empty string, and the data is silently converted to reflect this
-assumption.
+Oracle backend ignores an explicit ``null`` option on fields that
+have the empty string as a possible value and generates DDL as if
+``null=True``. When fetching from the database, it is assumed that
+a ``NULL`` value in one of these fields really means the empty
+string, and the data is silently converted to reflect this assumption.
``TextField`` limitations
-------------------------
View
5 docs/ref/models/fields.txt
@@ -55,9 +55,8 @@ string, not ``NULL``.
.. note::
- When using the Oracle database backend, the ``null=True`` option will be
- coerced for string-based fields that have the empty string as a possible
- value, and the value ``NULL`` will be stored to denote the empty string.
+ When using the Oracle database backend, the value ``NULL`` will be stored to
+ denote the empty string regardless of this attribute.
If you want to accept :attr:`~Field.null` values with :class:`BooleanField`,
use :class:`NullBooleanField` instead.
View
22 tests/regressiontests/queries/tests.py
@@ -1930,3 +1930,25 @@ def test_col_not_in_list_containing_null(self):
self.assertQuerysetEqual(
NullableName.objects.exclude(name__in=[None]),
['i1'], attrgetter('name'))
+
+class EmptyStringsAsNullTest(TestCase):
+ """
+ Test that filtering on non-null character fields works as expected.
+ The reason for these tests is that Oracle treats '' as NULL, and this
+ can cause problems in query construction. Refs #17957.
+ """
+
+ def setUp(self):
+ self.nc = NamedCategory.objects.create(name='')
+
+ def test_direct_exclude(self):
+ self.assertQuerysetEqual(
+ NamedCategory.objects.exclude(name__in=['nonexisting']),
+ [self.nc.pk], attrgetter('pk')
+ )
+
+ def test_joined_exclude(self):
+ self.assertQuerysetEqual(
+ DumbCategory.objects.exclude(namedcategory__name__in=['nonexisting']),
+ [self.nc.pk], attrgetter('pk')
+ )
Something went wrong with that request. Please try again.