Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Fix fetching models from complex RawQuerySets in SQLite #2069

Closed
wants to merge 6 commits into from

3 participants

@AlexHill

SQLite will sometimes (returning columns from subqueries or views for example) return column names in the form table_alias."column_name" or simply "column_name" instead of just column_name.

This meant that RawQuerySet can't match up the columns when trying to create model instances, and raises InvalidQuery: Raw query must include the primary key.

This change introduces a column_name_converter() to the backend introspection code to mirror table_name_converter() and implements the fix on the SQLite backend.

This is the same behaviour accounted for in SQLAlchemy here: https://github.com/zzzeek/sqlalchemy/blob/4663ec98b226a7d495846f0d89c646110705bb30/lib/sqlalchemy/dialects/sqlite/base.py#L591

@apollo13
Owner

Where is the accepted ticket for this issue?

@AlexHill

Hi Florian,

The ticket isn't accepted, but it's here, with some discussion: https://code.djangoproject.com/ticket/21603

django/db/backends/__init__.py
@@ -1242,6 +1242,13 @@ def table_name_converter(self, name):
"""
return name
+ def column_name_converter(self, name):
+ """Apply a conversion to the column name for the purposes of comparison.
+
+ The default column name converter is for case sensitive comparison.
+ """
+ return name
@timgraham Owner

maybe it would be better to call table_name_converter() as a "reasonable default" (since most backends seem to require the same behavior)?

@AlexHill
AlexHill added a note

Sure, I'm fine with that – it would ensure compatibility with existing third-party backends that override table_name_converter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
django/db/backends/sqlite3/introspection.py
@@ -68,6 +68,18 @@ def get_table_description(self, cursor, table_name):
return [FieldInfo(info['name'], info['type'], None, info['size'], None, None,
info['null_ok']) for info in self._table_info(cursor, table_name)]
+ def column_name_converter(self, name):
+ """
+ SQLite will in some cases, e.g. when returning columns from views and
+ subselects, return column names in 'alias."column"' format instead of
+ simply 'column'.
+
+ Affects SQLite < 3.7.15, fixed by http://www.sqlite.org/src/info/5526e0aa3c
+ """
+ # TODO: remove when SQLite < 3.7.15 is sufficiently old.
+ # 3.7.13 ships in Debian stable as of 2014-03-21.
+ return name.split('.')[-1].strip('"')
@timgraham Owner

could we add a Database.sqlite_version_info < (3, 7, 15) conditional to skip the logic on newer versions?

@AlexHill
AlexHill added a note

Sure :thumbsup:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@timgraham
Owner

buildbot, test this please.

@timgraham
Owner

merged in 938da36, thanks.

@timgraham timgraham closed this
@AlexHill AlexHill deleted the AlexHill:column_name_converter branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
View
7 django/db/backends/__init__.py
@@ -1242,6 +1242,13 @@ def table_name_converter(self, name):
"""
return name
+ def column_name_converter(self, name):
+ """Apply a conversion to the column name for the purposes of comparison.
+
+ The default column name converter is for case sensitive comparison.
+ """
+ return self.table_name_converter(name)
+
def table_names(self, cursor=None):
"""
Returns a list of names of all tables that exist in the database.
View
15 django/db/backends/sqlite3/introspection.py
@@ -68,6 +68,21 @@ def get_table_description(self, cursor, table_name):
return [FieldInfo(info['name'], info['type'], None, info['size'], None, None,
info['null_ok']) for info in self._table_info(cursor, table_name)]
+ def column_name_converter(self, name):
+ """
+ SQLite will in some cases, e.g. when returning columns from views and
+ subselects, return column names in 'alias."column"' format instead of
+ simply 'column'.
+
+ Affects SQLite < 3.7.15, fixed by http://www.sqlite.org/src/info/5526e0aa3c
+ """
+ # TODO: remove when SQLite < 3.7.15 is sufficiently old.
+ # 3.7.13 ships in Debian stable as of 2014-03-21.
+ if self.connection.Database.sqlite_version_info < (3, 7, 15):
+ return name.split('.')[-1].strip('"')
+ else:
+ return name
+
def get_relations(self, cursor, table_name):
"""
Returns a dictionary of {field_index: (field_index_other_table, other_table)}
View
2  django/db/models/sql/query.py
@@ -64,7 +64,7 @@ def convert_values(self, value, field, connection):
def get_columns(self):
if self.cursor is None:
self._execute_query()
- converter = connections[self.using].introspection.table_name_converter
+ converter = connections[self.using].introspection.column_name_converter
return [converter(column_meta[0])
for column_meta in self.cursor.description]
View
6 tests/raw_query/tests.py
@@ -239,3 +239,9 @@ def test_inheritance(self):
def test_query_count(self):
self.assertNumQueries(1, list, Author.objects.raw("SELECT * FROM raw_query_author"))
+
+ def test_subquery_in_raw_sql(self):
+ try:
+ list(Book.objects.raw('SELECT "id" FROM (SELECT * FROM raw_query_book WHERE paperback) sq'))
+ except InvalidQuery:
+ self.fail("Using a subquery in a RawQuerySet raised InvalidQuery")
Something went wrong with that request. Please try again.