Permalink
Browse files

Fixed #17260 -- Added time zone aware aggregation and lookups.

Thanks Carl Meyer for the review.

Squashed commit of the following:

commit 4f290bd
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date:   Wed Feb 13 21:21:30 2013 +0100

    Used '0:00' instead of 'UTC' which doesn't always exist in Oracle.

    Thanks Ian Kelly for the suggestion.

commit 01b6366
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date:   Wed Feb 13 13:38:43 2013 +0100

    Made tzname a parameter of datetime_extract/trunc_sql.

    This is required to work around a bug in Oracle.

commit 924a144
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date:   Wed Feb 13 14:47:44 2013 +0100

    Added support for parameters in SELECT clauses.

commit b4351d2
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date:   Mon Feb 11 22:30:22 2013 +0100

    Documented backwards incompatibilities in the two previous commits.

commit 91ef847
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date:   Mon Feb 11 09:42:31 2013 +0100

    Used QuerySet.datetimes for the admin's date_hierarchy.

commit 0d0de28
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date:   Mon Feb 11 09:29:38 2013 +0100

    Used QuerySet.datetimes in date-based generic views.

commit 9c0859f
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date:   Sun Feb 10 21:43:25 2013 +0100

    Implemented QuerySet.datetimes on Oracle.

commit 68ab511
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date:   Sun Feb 10 21:43:14 2013 +0100

    Implemented QuerySet.datetimes on MySQL.

commit 22d5268
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date:   Sun Feb 10 21:42:29 2013 +0100

    Implemented QuerySet.datetimes on SQLite.

commit f6800fd
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date:   Sun Feb 10 21:43:03 2013 +0100

    Implemented QuerySet.datetimes on PostgreSQL.

commit 0c829c2
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date:   Sun Feb 10 21:41:08 2013 +0100

    Added datetime-handling infrastructure in the ORM layers.

commit 104d82a
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date:   Mon Feb 11 10:05:55 2013 +0100

    Updated null_queries tests to avoid clashing with the __second lookup.

commit c01bbb3
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date:   Sun Feb 10 23:07:41 2013 +0100

    Updated tests of .dates().

    Replaced .dates() by .datetimes() for DateTimeFields.
    Replaced dates with datetimes in the expected output for DateFields.

commit 50fb7a5
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date:   Sun Feb 10 21:40:09 2013 +0100

    Updated and added tests for QuerySet.datetimes.

commit a8451a5
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date:   Sun Feb 10 22:34:46 2013 +0100

    Documented the new time lookups and updated the date lookups.

commit 29413ea
Author: Aymeric Augustin <aymeric.augustin@m4x.org>
Date:   Sun Feb 10 16:15:49 2013 +0100

    Documented QuerySet.datetimes and updated QuerySet.dates.
  • Loading branch information...
1 parent 91c26ea commit e74e207cce54802f897adcb42149440ee154821e @aaugustin aaugustin committed Feb 10, 2013
Showing with 1,035 additions and 294 deletions.
  1. +9 −5 django/contrib/admin/templatetags/admin_list.py
  2. +3 −0 django/contrib/gis/db/backends/mysql/compiler.py
  3. +4 −3 django/contrib/gis/db/backends/mysql/operations.py
  4. +3 −0 django/contrib/gis/db/backends/oracle/compiler.py
  5. +2 −2 django/contrib/gis/db/backends/oracle/operations.py
  6. +1 −1 django/contrib/gis/db/backends/postgis/operations.py
  7. +1 −1 django/contrib/gis/db/backends/spatialite/operations.py
  8. +1 −1 django/contrib/gis/db/backends/util.py
  9. +7 −5 django/contrib/gis/db/models/sql/aggregates.py
  10. +48 −15 django/contrib/gis/db/models/sql/compiler.py
  11. +3 −2 django/contrib/gis/db/models/sql/where.py
  12. +1 −1 django/contrib/gis/tests/geoapp/test_regress.py
  13. +41 −15 django/db/backends/__init__.py
  14. +43 −4 django/db/backends/mysql/base.py
  15. +3 −0 django/db/backends/mysql/compiler.py
  16. +54 −8 django/db/backends/oracle/base.py
  17. +3 −0 django/db/backends/oracle/compiler.py
  18. +25 −0 django/db/backends/postgresql_psycopg2/operations.py
  19. +69 −11 django/db/backends/sqlite3/base.py
  20. +13 −10 django/db/models/fields/__init__.py
  21. +3 −0 django/db/models/manager.py
  22. +48 −3 django/db/models/query.py
  23. +1 −1 django/db/models/query_utils.py
  24. +6 −5 django/db/models/sql/aggregates.py
  25. +69 −30 django/db/models/sql/compiler.py
  26. +2 −1 django/db/models/sql/constants.py
  27. +22 −1 django/db/models/sql/datastructures.py
  28. +2 −2 django/db/models/sql/expressions.py
  29. +39 −9 django/db/models/sql/subqueries.py
  30. +21 −13 django/db/models/sql/where.py
  31. +6 −3 django/views/generic/dates.py
  32. +136 −21 docs/ref/models/querysets.txt
  33. +35 −0 docs/releases/1.6.txt
  34. +4 −4 tests/modeltests/aggregation/tests.py
  35. +9 −9 tests/modeltests/basic/tests.py
  36. +28 −28 tests/modeltests/many_to_one/tests.py
  37. +2 −2 tests/modeltests/reserved_names/tests.py
  38. +100 −28 tests/modeltests/timezones/tests.py
  39. +2 −2 tests/regressiontests/aggregation_regress/tests.py
  40. +3 −3 tests/regressiontests/backends/tests.py
  41. +2 −0 tests/regressiontests/dates/models.py
  42. +19 −19 tests/regressiontests/dates/tests.py
  43. 0 tests/regressiontests/datetimes/__init__.py
  44. +28 −0 tests/regressiontests/datetimes/models.py
  45. +83 −0 tests/regressiontests/datetimes/tests.py
  46. +3 −2 tests/regressiontests/extra_regress/tests.py
  47. +9 −6 tests/regressiontests/generic_views/dates.py
  48. +2 −2 tests/regressiontests/model_inheritance_regress/tests.py
  49. +2 −1 tests/regressiontests/null_queries/models.py
  50. +3 −3 tests/regressiontests/null_queries/tests.py
  51. +12 −12 tests/regressiontests/queries/tests.py
@@ -292,6 +292,8 @@ def date_hierarchy(cl):
"""
if cl.date_hierarchy:
field_name = cl.date_hierarchy
+ field = cl.opts.get_field_by_name(field_name)[0]
+ dates_or_datetimes = 'datetimes' if isinstance(field, models.DateTimeField) else 'dates'
year_field = '%s__year' % field_name
month_field = '%s__month' % field_name
day_field = '%s__day' % field_name
@@ -323,7 +325,8 @@ def date_hierarchy(cl):
'choices': [{'title': capfirst(formats.date_format(day, 'MONTH_DAY_FORMAT'))}]
}
elif year_lookup and month_lookup:
- days = cl.query_set.filter(**{year_field: year_lookup, month_field: month_lookup}).dates(field_name, 'day')
+ days = cl.query_set.filter(**{year_field: year_lookup, month_field: month_lookup})
+ days = getattr(days, dates_or_datetimes)(field_name, 'day')
return {
'show': True,
'back': {
@@ -336,11 +339,12 @@ def date_hierarchy(cl):
} for day in days]
}
elif year_lookup:
- months = cl.query_set.filter(**{year_field: year_lookup}).dates(field_name, 'month')
+ months = cl.query_set.filter(**{year_field: year_lookup})
+ months = getattr(months, dates_or_datetimes)(field_name, 'month')
return {
- 'show' : True,
+ 'show': True,
'back': {
- 'link' : link({}),
+ 'link': link({}),
'title': _('All dates')
},
'choices': [{
@@ -349,7 +353,7 @@ def date_hierarchy(cl):
} for month in months]
}
else:
- years = cl.query_set.dates(field_name, 'year')
+ years = getattr(cl.query_set, dates_or_datetimes)(field_name, 'year')
return {
'show': True,
'choices': [{
@@ -30,3 +30,6 @@ class SQLAggregateCompiler(compiler.SQLAggregateCompiler, GeoSQLCompiler):
class SQLDateCompiler(compiler.SQLDateCompiler, GeoSQLCompiler):
pass
+
+class SQLDateTimeCompiler(compiler.SQLDateTimeCompiler, GeoSQLCompiler):
+ pass
@@ -56,12 +56,13 @@ def spatial_lookup_sql(self, lvalue, lookup_type, value, field, qn):
lookup_info = self.geometry_functions.get(lookup_type, False)
if lookup_info:
- return "%s(%s, %s)" % (lookup_info, geo_col,
- self.get_geom_placeholder(value, field.srid))
+ sql = "%s(%s, %s)" % (lookup_info, geo_col,
+ self.get_geom_placeholder(value, field.srid))
+ return sql, []
# TODO: Is this really necessary? MySQL can't handle NULL geometries
# in its spatial indexes anyways.
if lookup_type == 'isnull':
- return "%s IS %sNULL" % (geo_col, (not value and 'NOT ' or ''))
+ return "%s IS %sNULL" % (geo_col, ('' if value else 'NOT ')), []
raise TypeError("Got invalid lookup_type: %s" % repr(lookup_type))
@@ -20,3 +20,6 @@ class SQLAggregateCompiler(compiler.SQLAggregateCompiler, GeoSQLCompiler):
class SQLDateCompiler(compiler.SQLDateCompiler, GeoSQLCompiler):
pass
+
+class SQLDateTimeCompiler(compiler.SQLDateTimeCompiler, GeoSQLCompiler):
+ pass
@@ -262,7 +262,7 @@ def spatial_lookup_sql(self, lvalue, lookup_type, value, field, qn):
return lookup_info.as_sql(geo_col, self.get_geom_placeholder(field, value))
elif lookup_type == 'isnull':
# Handling 'isnull' lookup type
- return "%s IS %sNULL" % (geo_col, (not value and 'NOT ' or ''))
+ return "%s IS %sNULL" % (geo_col, ('' if value else 'NOT ')), []
raise TypeError("Got invalid lookup_type: %s" % repr(lookup_type))
@@ -288,7 +288,7 @@ def geometry_columns(self):
def spatial_ref_sys(self):
from django.contrib.gis.db.backends.oracle.models import SpatialRefSys
return SpatialRefSys
-
+
def modify_insert_params(self, placeholders, params):
"""Drop out insert parameters for NULL placeholder. Needed for Oracle Spatial
backend due to #10888
@@ -560,7 +560,7 @@ def spatial_lookup_sql(self, lvalue, lookup_type, value, field, qn):
elif lookup_type == 'isnull':
# Handling 'isnull' lookup type
- return "%s IS %sNULL" % (geo_col, (not value and 'NOT ' or ''))
+ return "%s IS %sNULL" % (geo_col, ('' if value else 'NOT ')), []
raise TypeError("Got invalid lookup_type: %s" % repr(lookup_type))
@@ -358,7 +358,7 @@ def spatial_lookup_sql(self, lvalue, lookup_type, value, field, qn):
return op.as_sql(geo_col, self.get_geom_placeholder(field, geom))
elif lookup_type == 'isnull':
# Handling 'isnull' lookup type
- return "%s IS %sNULL" % (geo_col, (not value and 'NOT ' or ''))
+ return "%s IS %sNULL" % (geo_col, ('' if value else 'NOT ')), []
raise TypeError("Got invalid lookup_type: %s" % repr(lookup_type))
@@ -16,7 +16,7 @@ def __init__(self, function='', operator='', result='', **kwargs):
self.extra = kwargs
def as_sql(self, geo_col, geometry='%s'):
- return self.sql_template % self.params(geo_col, geometry)
+ return self.sql_template % self.params(geo_col, geometry), []
def params(self, geo_col, geometry):
params = {'function' : self.function,
@@ -22,27 +22,29 @@ def __init__(self, col, source=None, is_summary=False, tolerance=0.05, **extra):
raise ValueError('Geospatial aggregates only allowed on geometry fields.')
def as_sql(self, qn, connection):
- "Return the aggregate, rendered as SQL."
+ "Return the aggregate, rendered as SQL with parameters."
if connection.ops.oracle:
self.extra['tolerance'] = self.tolerance
+ params = []
+
if hasattr(self.col, 'as_sql'):
- field_name = self.col.as_sql(qn, connection)
+ field_name, params = self.col.as_sql(qn, connection)
elif isinstance(self.col, (list, tuple)):
field_name = '.'.join([qn(c) for c in self.col])
else:
field_name = self.col
sql_template, sql_function = connection.ops.spatial_aggregate_sql(self)
- params = {
+ substitutions = {
'function': sql_function,
'field': field_name
}
- params.update(self.extra)
+ substitutions.update(self.extra)
- return sql_template % params
+ return sql_template % substitutions, params
class Collect(GeoAggregate):
pass
@@ -1,14 +1,16 @@
+import datetime
try:
from itertools import zip_longest
except ImportError:
from itertools import izip_longest as zip_longest
-from django.utils.six.moves import zip
-
-from django.db.backends.util import truncate_name, typecast_timestamp
+from django.conf import settings
+from django.db.backends.util import truncate_name, typecast_date, typecast_timestamp
from django.db.models.sql import compiler
from django.db.models.sql.constants import MULTI
from django.utils import six
+from django.utils.six.moves import zip
+from django.utils import timezone
SQLCompiler = compiler.SQLCompiler
@@ -31,6 +33,7 @@ def get_columns(self, with_aliases=False):
qn2 = self.connection.ops.quote_name
result = ['(%s) AS %s' % (self.get_extra_select_format(alias) % col[0], qn2(alias))
for alias, col in six.iteritems(self.query.extra_select)]
+ params = []
aliases = set(self.query.extra_select.keys())
if with_aliases:
col_aliases = aliases.copy()
@@ -61,7 +64,9 @@ def get_columns(self, with_aliases=False):
aliases.add(r)
col_aliases.add(col[1])
else:
- result.append(col.as_sql(qn, self.connection))
+ col_sql, col_params = col.as_sql(qn, self.connection)
+ result.append(col_sql)
+ params.extend(col_params)
if hasattr(col, 'alias'):
aliases.add(col.alias)
@@ -74,15 +79,13 @@ def get_columns(self, with_aliases=False):
aliases.update(new_aliases)
max_name_length = self.connection.ops.max_name_length()
- result.extend([
- '%s%s' % (
- self.get_extra_select_format(alias) % aggregate.as_sql(qn, self.connection),
- alias is not None
- and ' AS %s' % qn(truncate_name(alias, max_name_length))
- or ''
- )
- for alias, aggregate in self.query.aggregate_select.items()
- ])
+ for alias, aggregate in self.query.aggregate_select.items():
+ agg_sql, agg_params = aggregate.as_sql(qn, self.connection)
+ if alias is None:
+ result.append(agg_sql)
+ else:
+ result.append('%s AS %s' % (agg_sql, qn(truncate_name(alias, max_name_length))))
+ params.extend(agg_params)
# This loop customized for GeoQuery.
for (table, col), field in self.query.related_select_cols:
@@ -98,7 +101,7 @@ def get_columns(self, with_aliases=False):
col_aliases.add(col)
self._select_aliases = aliases
- return result
+ return result, params
def get_default_columns(self, with_aliases=False, col_aliases=None,
start_alias=None, opts=None, as_pairs=False, from_parent=None):
@@ -280,5 +283,35 @@ def results_iter(self):
if self.connection.ops.oracle:
date = self.resolve_columns(row, fields)[offset]
elif needs_string_cast:
- date = typecast_timestamp(str(date))
+ date = typecast_date(str(date))
+ if isinstance(date, datetime.datetime):
+ date = date.date()
yield date
+
+class SQLDateTimeCompiler(compiler.SQLDateTimeCompiler, GeoSQLCompiler):
+ """
+ This is overridden for GeoDjango to properly cast date columns, since
+ `GeoQuery.resolve_columns` is used for spatial values.
+ See #14648, #16757.
+ """
+ def results_iter(self):
+ if self.connection.ops.oracle:
+ from django.db.models.fields import DateTimeField
+ fields = [DateTimeField()]
+ else:
+ needs_string_cast = self.connection.features.needs_datetime_string_cast
+
+ offset = len(self.query.extra_select)
+ for rows in self.execute_sql(MULTI):
+ for row in rows:
+ datetime = row[offset]
+ if self.connection.ops.oracle:
+ datetime = self.resolve_columns(row, fields)[offset]
+ elif needs_string_cast:
+ datetime = typecast_timestamp(str(datetime))
+ # Datetimes are artifically returned in UTC on databases that
+ # don't support time zone. Restore the zone used in the query.
+ if settings.USE_TZ:
+ datetime = datetime.replace(tzinfo=None)
+ datetime = timezone.make_aware(datetime, self.query.tzinfo)
@frol
frol Aug 18, 2013

I have an "AttributeError: 'DateTimeQuery' object has no attribute 'tzinfo'" in this line while trying to do something like this:
Message.objects.datetimes('sent_at', 'year')
Message.objects.all().datetimes('sent_at', 'year')

I actually cannot find tzinfo neither in DateTimeQuery nor in its parent classes. Does it patched somewhere?
I use Django 1.6b2 and MySQL.

@aaugustin
aaugustin Aug 20, 2013 Member

If sent_at is a DateField (which I suspect) you must use use .dates instead of .datetimes.

If it's another issue, would you mind opening a ticket at code.djangoproject.com, so we have a record of the issue?

+ yield datetime
@@ -44,8 +44,9 @@ def make_atom(self, child, qn, connection):
lvalue, lookup_type, value_annot, params_or_value = child
if isinstance(lvalue, GeoConstraint):
data, params = lvalue.process(lookup_type, params_or_value, connection)
- spatial_sql = connection.ops.spatial_lookup_sql(data, lookup_type, params_or_value, lvalue.field, qn)
- return spatial_sql, params
+ spatial_sql, spatial_params = connection.ops.spatial_lookup_sql(
+ data, lookup_type, params_or_value, lvalue.field, qn)
+ return spatial_sql, spatial_params + params
else:
return super(GeoWhereNode, self).make_atom(child, qn, connection)
@@ -49,7 +49,7 @@ def test_unicode_date(self):
founded = datetime(1857, 5, 23)
mansfield = PennsylvaniaCity.objects.create(name='Mansfield', county='Tioga', point='POINT(-77.071445 41.823881)',
founded=founded)
- self.assertEqual(founded, PennsylvaniaCity.objects.dates('founded', 'day')[0])
+ self.assertEqual(founded, PennsylvaniaCity.objects.datetimes('founded', 'day')[0])
self.assertEqual(founded, PennsylvaniaCity.objects.aggregate(Min('founded'))['founded__min'])
def test_empty_count(self):
@@ -1,3 +1,5 @@
+import datetime
+
from django.db.utils import DatabaseError
try:
@@ -14,7 +16,7 @@
from django.utils.functional import cached_property
from django.utils.importlib import import_module
from django.utils import six
-from django.utils.timezone import is_aware
+from django.utils import timezone
class BaseDatabaseWrapper(object):
@@ -397,6 +399,9 @@ class BaseDatabaseFeatures(object):
# Can datetimes with timezones be used?
supports_timezones = True
+ # Does the database have a copy of the zoneinfo database?
+ has_zoneinfo_database = True
+
# When performing a GROUP BY, is an ORDER BY NULL required
# to remove any ordering?
requires_explicit_null_ordering_when_grouping = False
@@ -523,7 +528,7 @@ def date_interval_sql(self, sql, connector, timedelta):
def date_trunc_sql(self, lookup_type, field_name):
"""
Given a lookup_type of 'year', 'month' or 'day', returns the SQL that
- truncates the given date field field_name to a DATE object with only
+ truncates the given date field field_name to a date object with only
the given specificity.
"""
raise NotImplementedError()
@@ -537,6 +542,23 @@ def datetime_cast_sql(self):
"""
return "%s"
+ def datetime_extract_sql(self, lookup_type, field_name, tzname):
+ """
+ Given a lookup_type of 'year', 'month', 'day', 'hour', 'minute' or
+ 'second', returns the SQL that extracts a value from the given
+ datetime field field_name, and a tuple of parameters.
+ """
+ raise NotImplementedError()
+
+ def datetime_trunc_sql(self, lookup_type, field_name, tzname):
+ """
+ Given a lookup_type of 'year', 'month', 'day', 'hour', 'minute' or
+ 'second', returns the SQL that truncates the given datetime field
+ field_name to a datetime object with only the given specificity, and
+ a tuple of parameters.
+ """
+ raise NotImplementedError()
+
def deferrable_sql(self):
"""
Returns the SQL necessary to make a constraint "initially deferred"
@@ -853,7 +875,7 @@ def value_to_db_time(self, value):
"""
if value is None:
return None
- if is_aware(value):
+ if timezone.is_aware(value):
raise ValueError("Django does not support timezone-aware times.")
return six.text_type(value)
@@ -866,29 +888,33 @@ def value_to_db_decimal(self, value, max_digits, decimal_places):
return None
return util.format_number(value, max_digits, decimal_places)
- def year_lookup_bounds(self, value):
+ def year_lookup_bounds_for_date_field(self, value):
"""
Returns a two-elements list with the lower and upper bound to be used
- with a BETWEEN operator to query a field value using a year lookup
+ with a BETWEEN operator to query a DateField value using a year
+ lookup.
`value` is an int, containing the looked-up year.
"""
- first = '%s-01-01 00:00:00'
- second = '%s-12-31 23:59:59.999999'
- return [first % value, second % value]
+ first = datetime.date(value, 1, 1)
+ second = datetime.date(value, 12, 31)
+ return [first, second]
- def year_lookup_bounds_for_date_field(self, value):
+ def year_lookup_bounds_for_datetime_field(self, value):
"""
Returns a two-elements list with the lower and upper bound to be used
- with a BETWEEN operator to query a DateField value using a year lookup
+ with a BETWEEN operator to query a DateTimeField value using a year
+ lookup.
`value` is an int, containing the looked-up year.
-
- By default, it just calls `self.year_lookup_bounds`. Some backends need
- this hook because on their DB date fields can't be compared to values
- which include a time part.
"""
- return self.year_lookup_bounds(value)
+ first = datetime.datetime(value, 1, 1)
+ second = datetime.datetime(value, 12, 31, 23, 59, 59, 999999)
+ if settings.USE_TZ:
+ tz = timezone.get_current_timezone()
+ first = timezone.make_aware(first, tz)
+ second = timezone.make_aware(second, tz)
+ return [first, second]
def convert_values(self, value, field):
"""
Oops, something went wrong.

0 comments on commit e74e207

Please sign in to comment.