Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Prevent Oracle from changing field.null to True #19

Closed
wants to merge 1 commit into from

2 participants

Anssi Kääriäinen Alex Gaynor
Anssi Kääriäinen
Owner

Fixed #17957 -- when using Oracle and character fields, the fields
were set null = True to ease the handling of empty strings. This
caused problems when using multiple databases from different vendors,
or when the character field happened to be also a primary key.

The handling was changed so that NOT NULL is not emitted on Oracle
even if field.null = False, and field.null is not touched otherwise.

Thanks to bhuztez for the report, ramiro for triaging & comments and
ikelly for the patch.

Anssi Kääriäinen
Owner

I am still unsure how I should commit code to github - so here is another pull request fixing Oracle specific stuff.

Alex Gaynor
Owner
alex commented April 28, 2012

Do all tests still pass on Oracle?

Anssi Kääriäinen
Owner

All tests haven't passed in a while on Oracle. I'm working on fixes...

At least one more test passes with the patch applied (test_runner tests, related to character field primary keys having null=True). I haven't done a comparison between full test suite with and without this patch as it takes a lot of time to do that. I can leave the full test suite running for the night if needed.

Alex Gaynor
Owner
alex commented April 28, 2012

Ok, FWIW my concern was whether this was needed so the correct INNER vs. OUTER JOINs were generated.

Anssi Kääriäinen
Owner

Worth a test at least (if not already tested somewhere). It seems this could cause some weird behavior there. Should I close this pull request while I check this? I am not sure how long it will take to check and possibly correct errors caused by this...

Alex Gaynor
Owner
alex commented April 28, 2012

I think it's fine to leave the pull request open.

Anssi Kääriäinen
Owner

I have investigated the queryset usage of field.null issue a little, and it seems clear that the change could result in some queries breaking when using Oracle. It is a bit scary to use Oracle's view of if the field is nullable when constructing the query because a later .using() clause could then lead to problems.

Luckily it seems that both of the field.null uses in sql/query.py will work correctly if field.null is "guessed" to true even if it is not true for the backend. So, I am going to update the patch so that sql/query.py will treat fields where (empty strings are allowed and connection is Oracle) as nullable. To me it seems in the worst case this generates slightly less performant queries when a query is generated using Oracle, but executed using some other backend - in other words, we really don't need to care about this issue too much...

I ran the full test suite during the night, before there were 3 failures, with patch only two.

Anssi Kääriäinen Prevent Oracle from changing field.null to True
Fixed #17957 -- when using Oracle and character fields, the fields
were set null = True to ease the handling of empty strings. This
caused problems when using multiple databases from different vendors,
or when the character field happened to be also a primary key.

The handling was changed so that NOT NULL is not emitted on Oracle
even if field.null = False, and field.null is not touched otherwise.

Thanks to bhuztez for the report, ramiro for triaging & comments,
ikelly for the patch and alex for reviewing.
0d45eef
Anssi Kääriäinen
Owner

I added some tests to check that character fields are handled correctly in query.py. The field.null handling is done using is_nullable(field). It has to use the DEFAULT_DB_ALIAS when deciding if the field should be treated as nullable as the sql.QuerySet doesn't know which connection is used. The field.null handling is thus identical to pre-patch situation. That is, slightly broken in weird corner cases.

Alex Gaynor
Owner
alex commented April 29, 2012

Ok now I remember this, unfortunately your analysis exactly matches what mine was. That is: this decision should be made later, but that's a ton of work. Patch LGTM feel free to commit!

Anssi Kääriäinen
Owner

I pushed this manually -- closing.

Anssi Kääriäinen akaariai closed this April 29, 2012
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Showing 1 unique commit by 1 author.

Apr 29, 2012
Anssi Kääriäinen Prevent Oracle from changing field.null to True
Fixed #17957 -- when using Oracle and character fields, the fields
were set null = True to ease the handling of empty strings. This
caused problems when using multiple databases from different vendors,
or when the character field happened to be also a primary key.

The handling was changed so that NOT NULL is not emitted on Oracle
even if field.null = False, and field.null is not touched otherwise.

Thanks to bhuztez for the report, ramiro for triaging & comments,
ikelly for the patch and alex for reviewing.
0d45eef
This page is out of date. Refresh to see the latest.
8  django/db/backends/creation.py
@@ -50,7 +50,13 @@ def sql_create_model(self, model, style, known_models=set()):
50 50
             # Make the definition (e.g. 'foo VARCHAR(30)') for this field.
51 51
             field_output = [style.SQL_FIELD(qn(f.column)),
52 52
                 style.SQL_COLTYPE(col_type)]
53  
-            if not f.null:
  53
+            # Oracle treats the empty string ('') as null, so coerce the null
  54
+            # option whenever '' is a possible value.
  55
+            null = f.null
  56
+            if (f.empty_strings_allowed and not f.primary_key and
  57
+                    self.connection.features.interprets_empty_strings_as_nulls):
  58
+                null = True
  59
+            if not null:
54 60
                 field_output.append(style.SQL_KEYWORD('NOT NULL'))
55 61
             if f.primary_key:
56 62
                 field_output.append(style.SQL_KEYWORD('PRIMARY KEY'))
5  django/db/models/fields/__init__.py
@@ -85,11 +85,6 @@ def __init__(self, verbose_name=None, name=None, primary_key=False,
85 85
         self.primary_key = primary_key
86 86
         self.max_length, self._unique = max_length, unique
87 87
         self.blank, self.null = blank, null
88  
-        # Oracle treats the empty string ('') as null, so coerce the null
89  
-        # option whenever '' is a possible value.
90  
-        if (self.empty_strings_allowed and
91  
-            connection.features.interprets_empty_strings_as_nulls):
92  
-            self.null = True
93 88
         self.rel = rel
94 89
         self.default = default
95 90
         self.editable = editable
23  django/db/models/sql/query.py
@@ -1193,7 +1193,7 @@ def add_filter(self, filter_expr, connector=AND, negate=False, trim=False,
1193 1193
                             entry.negate()
1194 1194
                             self.where.add(entry, AND)
1195 1195
                             break
1196  
-                if field.null:
  1196
+                if self.is_nullable(field):
1197 1197
                     # In SQL NULL = anyvalue returns unknown, and NOT unknown
1198 1198
                     # is still unknown. However, in Python None = anyvalue is False
1199 1199
                     # (and not False is True...), and we want to return this Python's
@@ -1396,7 +1396,8 @@ def setup_joins(self, names, opts, alias, dupe_multis, allow_many=True,
1396 1396
                                 opts, target)
1397 1397
 
1398 1398
                     alias = self.join((alias, table, from_col, to_col),
1399  
-                            exclusions=exclusions, nullable=field.null)
  1399
+                                      exclusions=exclusions,
  1400
+                                      nullable=self.is_nullable(field))
1400 1401
                     joins.append(alias)
1401 1402
                 else:
1402 1403
                     # Non-relation fields.
@@ -1946,6 +1947,24 @@ def set_start(self, start):
1946 1947
         self.select = [(select_alias, select_col)]
1947 1948
         self.remove_inherited_models()
1948 1949
 
  1950
+    def is_nullable(self, field):
  1951
+        """
  1952
+        A helper to check if the given field should be treated as nullable.
  1953
+
  1954
+        Some backends treat '' as null and Django treats such fields as
  1955
+        nullable for those backends. In such situations field.null can be
  1956
+        False even if we should treat the field as nullable.
  1957
+        """
  1958
+        # We need to use DEFAULT_DB_ALIAS here, as QuerySet does not have
  1959
+        # (nor should it have) knowledge of which connection is going to be
  1960
+        # used. The proper fix would be to defer all decisions where
  1961
+        # is_nullable() is needed to the compiler stage, but that is not easy
  1962
+        # to do currently.
  1963
+        if ((connections[DEFAULT_DB_ALIAS].features.interprets_empty_strings_as_nulls)
  1964
+            and field.empty_strings_allowed):
  1965
+            return True
  1966
+        else:
  1967
+            return field.null
1949 1968
 
1950 1969
 def get_order_dir(field, default='ASC'):
1951 1970
     """
10  docs/ref/databases.txt
@@ -685,11 +685,11 @@ NULL and empty strings
685 685
 
686 686
 Django generally prefers to use the empty string ('') rather than
687 687
 NULL, but Oracle treats both identically. To get around this, the
688  
-Oracle backend coerces the ``null=True`` option on fields that have
689  
-the empty string as a possible value. When fetching from the database,
690  
-it is assumed that a NULL value in one of these fields really means
691  
-the empty string, and the data is silently converted to reflect this
692  
-assumption.
  688
+Oracle backend ignores an explicit ``null`` option on fields that
  689
+have the empty string as a possible value and generates DDL as if
  690
+``null=True``. When fetching from the database, it is assumed that
  691
+a ``NULL`` value in one of these fields really means the empty
  692
+string, and the data is silently converted to reflect this assumption.
693 693
 
694 694
 ``TextField`` limitations
695 695
 -------------------------
5  docs/ref/models/fields.txt
@@ -55,9 +55,8 @@ string, not ``NULL``.
55 55
 
56 56
 .. note::
57 57
 
58  
-    When using the Oracle database backend, the ``null=True`` option will be
59  
-    coerced for string-based fields that have the empty string as a possible
60  
-    value, and the value ``NULL`` will be stored to denote the empty string.
  58
+    When using the Oracle database backend, the value ``NULL`` will be stored to
  59
+    denote the empty string regardless of this attribute.
61 60
 
62 61
 If you want to accept :attr:`~Field.null` values with :class:`BooleanField`,
63 62
 use :class:`NullBooleanField` instead.
22  tests/regressiontests/queries/tests.py
@@ -1930,3 +1930,25 @@ def test_col_not_in_list_containing_null(self):
1930 1930
         self.assertQuerysetEqual(
1931 1931
             NullableName.objects.exclude(name__in=[None]),
1932 1932
             ['i1'], attrgetter('name'))
  1933
+
  1934
+class EmptyStringsAsNullTest(TestCase):
  1935
+    """
  1936
+    Test that filtering on non-null character fields works as expected.
  1937
+    The reason for these tests is that Oracle treats '' as NULL, and this
  1938
+    can cause problems in query construction. Refs #17957.
  1939
+    """
  1940
+
  1941
+    def setUp(self):
  1942
+        self.nc = NamedCategory.objects.create(name='')
  1943
+
  1944
+    def test_direct_exclude(self):
  1945
+        self.assertQuerysetEqual(
  1946
+            NamedCategory.objects.exclude(name__in=['nonexisting']),
  1947
+            [self.nc.pk], attrgetter('pk')
  1948
+        )
  1949
+
  1950
+    def test_joined_exclude(self):
  1951
+        self.assertQuerysetEqual(
  1952
+            DumbCategory.objects.exclude(namedcategory__name__in=['nonexisting']),
  1953
+            [self.nc.pk], attrgetter('pk')
  1954
+        )
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.