Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Prevent Oracle from changing field.null to True #19

Closed
wants to merge 1 commit into from

2 participants

Anssi Kääriäinen Alex Gaynor
Anssi Kääriäinen
Collaborator

Fixed #17957 -- when using Oracle and character fields, the fields
were set null = True to ease the handling of empty strings. This
caused problems when using multiple databases from different vendors,
or when the character field happened to be also a primary key.

The handling was changed so that NOT NULL is not emitted on Oracle
even if field.null = False, and field.null is not touched otherwise.

Thanks to bhuztez for the report, ramiro for triaging & comments and
ikelly for the patch.

Anssi Kääriäinen
Collaborator

I am still unsure how I should commit code to github - so here is another pull request fixing Oracle specific stuff.

Alex Gaynor
Collaborator

Do all tests still pass on Oracle?

Anssi Kääriäinen
Collaborator

All tests haven't passed in a while on Oracle. I'm working on fixes...

At least one more test passes with the patch applied (test_runner tests, related to character field primary keys having null=True). I haven't done a comparison between full test suite with and without this patch as it takes a lot of time to do that. I can leave the full test suite running for the night if needed.

Alex Gaynor
Collaborator

Ok, FWIW my concern was whether this was needed so the correct INNER vs. OUTER JOINs were generated.

Anssi Kääriäinen
Collaborator

Worth a test at least (if not already tested somewhere). It seems this could cause some weird behavior there. Should I close this pull request while I check this? I am not sure how long it will take to check and possibly correct errors caused by this...

Alex Gaynor
Collaborator

I think it's fine to leave the pull request open.

Anssi Kääriäinen
Collaborator

I have investigated the queryset usage of field.null issue a little, and it seems clear that the change could result in some queries breaking when using Oracle. It is a bit scary to use Oracle's view of if the field is nullable when constructing the query because a later .using() clause could then lead to problems.

Luckily it seems that both of the field.null uses in sql/query.py will work correctly if field.null is "guessed" to true even if it is not true for the backend. So, I am going to update the patch so that sql/query.py will treat fields where (empty strings are allowed and connection is Oracle) as nullable. To me it seems in the worst case this generates slightly less performant queries when a query is generated using Oracle, but executed using some other backend - in other words, we really don't need to care about this issue too much...

I ran the full test suite during the night, before there were 3 failures, with patch only two.

Anssi Kääriäinen akaariai Prevent Oracle from changing field.null to True
Fixed #17957 -- when using Oracle and character fields, the fields
were set null = True to ease the handling of empty strings. This
caused problems when using multiple databases from different vendors,
or when the character field happened to be also a primary key.

The handling was changed so that NOT NULL is not emitted on Oracle
even if field.null = False, and field.null is not touched otherwise.

Thanks to bhuztez for the report, ramiro for triaging & comments,
ikelly for the patch and alex for reviewing.
0d45eef
Anssi Kääriäinen
Collaborator

I added some tests to check that character fields are handled correctly in query.py. The field.null handling is done using is_nullable(field). It has to use the DEFAULT_DB_ALIAS when deciding if the field should be treated as nullable as the sql.QuerySet doesn't know which connection is used. The field.null handling is thus identical to pre-patch situation. That is, slightly broken in weird corner cases.

Alex Gaynor
Collaborator

Ok now I remember this, unfortunately your analysis exactly matches what mine was. That is: this decision should be made later, but that's a ton of work. Patch LGTM feel free to commit!

Anssi Kääriäinen
Collaborator

I pushed this manually -- closing.

Anssi Kääriäinen akaariai closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Apr 29, 2012
  1. Anssi Kääriäinen

    Prevent Oracle from changing field.null to True

    akaariai authored
    Fixed #17957 -- when using Oracle and character fields, the fields
    were set null = True to ease the handling of empty strings. This
    caused problems when using multiple databases from different vendors,
    or when the character field happened to be also a primary key.
    
    The handling was changed so that NOT NULL is not emitted on Oracle
    even if field.null = False, and field.null is not touched otherwise.
    
    Thanks to bhuztez for the report, ramiro for triaging & comments,
    ikelly for the patch and alex for reviewing.
This page is out of date. Refresh to see the latest.
8 django/db/backends/creation.py
View
@@ -50,7 +50,13 @@ def sql_create_model(self, model, style, known_models=set()):
# Make the definition (e.g. 'foo VARCHAR(30)') for this field.
field_output = [style.SQL_FIELD(qn(f.column)),
style.SQL_COLTYPE(col_type)]
- if not f.null:
+ # Oracle treats the empty string ('') as null, so coerce the null
+ # option whenever '' is a possible value.
+ null = f.null
+ if (f.empty_strings_allowed and not f.primary_key and
+ self.connection.features.interprets_empty_strings_as_nulls):
+ null = True
+ if not null:
field_output.append(style.SQL_KEYWORD('NOT NULL'))
if f.primary_key:
field_output.append(style.SQL_KEYWORD('PRIMARY KEY'))
5 django/db/models/fields/__init__.py
View
@@ -85,11 +85,6 @@ def __init__(self, verbose_name=None, name=None, primary_key=False,
self.primary_key = primary_key
self.max_length, self._unique = max_length, unique
self.blank, self.null = blank, null
- # Oracle treats the empty string ('') as null, so coerce the null
- # option whenever '' is a possible value.
- if (self.empty_strings_allowed and
- connection.features.interprets_empty_strings_as_nulls):
- self.null = True
self.rel = rel
self.default = default
self.editable = editable
23 django/db/models/sql/query.py
View
@@ -1193,7 +1193,7 @@ def add_filter(self, filter_expr, connector=AND, negate=False, trim=False,
entry.negate()
self.where.add(entry, AND)
break
- if field.null:
+ if self.is_nullable(field):
# In SQL NULL = anyvalue returns unknown, and NOT unknown
# is still unknown. However, in Python None = anyvalue is False
# (and not False is True...), and we want to return this Python's
@@ -1396,7 +1396,8 @@ def setup_joins(self, names, opts, alias, dupe_multis, allow_many=True,
opts, target)
alias = self.join((alias, table, from_col, to_col),
- exclusions=exclusions, nullable=field.null)
+ exclusions=exclusions,
+ nullable=self.is_nullable(field))
joins.append(alias)
else:
# Non-relation fields.
@@ -1946,6 +1947,24 @@ def set_start(self, start):
self.select = [(select_alias, select_col)]
self.remove_inherited_models()
+ def is_nullable(self, field):
+ """
+ A helper to check if the given field should be treated as nullable.
+
+ Some backends treat '' as null and Django treats such fields as
+ nullable for those backends. In such situations field.null can be
+ False even if we should treat the field as nullable.
+ """
+ # We need to use DEFAULT_DB_ALIAS here, as QuerySet does not have
+ # (nor should it have) knowledge of which connection is going to be
+ # used. The proper fix would be to defer all decisions where
+ # is_nullable() is needed to the compiler stage, but that is not easy
+ # to do currently.
+ if ((connections[DEFAULT_DB_ALIAS].features.interprets_empty_strings_as_nulls)
+ and field.empty_strings_allowed):
+ return True
+ else:
+ return field.null
def get_order_dir(field, default='ASC'):
"""
10 docs/ref/databases.txt
View
@@ -685,11 +685,11 @@ NULL and empty strings
Django generally prefers to use the empty string ('') rather than
NULL, but Oracle treats both identically. To get around this, the
-Oracle backend coerces the ``null=True`` option on fields that have
-the empty string as a possible value. When fetching from the database,
-it is assumed that a NULL value in one of these fields really means
-the empty string, and the data is silently converted to reflect this
-assumption.
+Oracle backend ignores an explicit ``null`` option on fields that
+have the empty string as a possible value and generates DDL as if
+``null=True``. When fetching from the database, it is assumed that
+a ``NULL`` value in one of these fields really means the empty
+string, and the data is silently converted to reflect this assumption.
``TextField`` limitations
-------------------------
5 docs/ref/models/fields.txt
View
@@ -55,9 +55,8 @@ string, not ``NULL``.
.. note::
- When using the Oracle database backend, the ``null=True`` option will be
- coerced for string-based fields that have the empty string as a possible
- value, and the value ``NULL`` will be stored to denote the empty string.
+ When using the Oracle database backend, the value ``NULL`` will be stored to
+ denote the empty string regardless of this attribute.
If you want to accept :attr:`~Field.null` values with :class:`BooleanField`,
use :class:`NullBooleanField` instead.
22 tests/regressiontests/queries/tests.py
View
@@ -1930,3 +1930,25 @@ def test_col_not_in_list_containing_null(self):
self.assertQuerysetEqual(
NullableName.objects.exclude(name__in=[None]),
['i1'], attrgetter('name'))
+
+class EmptyStringsAsNullTest(TestCase):
+ """
+ Test that filtering on non-null character fields works as expected.
+ The reason for these tests is that Oracle treats '' as NULL, and this
+ can cause problems in query construction. Refs #17957.
+ """
+
+ def setUp(self):
+ self.nc = NamedCategory.objects.create(name='')
+
+ def test_direct_exclude(self):
+ self.assertQuerysetEqual(
+ NamedCategory.objects.exclude(name__in=['nonexisting']),
+ [self.nc.pk], attrgetter('pk')
+ )
+
+ def test_joined_exclude(self):
+ self.assertQuerysetEqual(
+ DumbCategory.objects.exclude(namedcategory__name__in=['nonexisting']),
+ [self.nc.pk], attrgetter('pk')
+ )
Something went wrong with that request. Please try again.