Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 1871 lines (1501 sloc) 73.333 kB
99a5c0c @slinkp Adding copyrights and GPL v3 everywhere
slinkp authored
1 # Copyright 2007,2008,2009,2011 Everyblock LLC, OpenPlans, and contributors
2 #
3 # This file is part of ebpub
4 #
5 # ebpub is free software: you can redistribute it and/or modify
6 # it under the terms of the GNU General Public License as published by
7 # the Free Software Foundation, either version 3 of the License, or
8 # (at your option) any later version.
9 #
10 # ebpub is distributed in the hope that it will be useful,
11 # but WITHOUT ANY WARRANTY; without even the implied warranty of
12 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
13 # GNU General Public License for more details.
14 #
15 # You should have received a copy of the GNU General Public License
16 # along with ebpub. If not, see <http://www.gnu.org/licenses/>.
17 #
18
c7c4cfb @slinkp more scraper doc work
slinkp authored
19 """
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
20
21 .. _newsitems:
22
23 .. _newsitem-schemas:
24
25 Overview: NewsItems and Schemas
26 ===============================
27
28 The ebpub system is capable of handling
29 many disparate types of news -- e.g., crime, photos and restaurant inspections.
30 Each type of news is referred to as a :py:class:`Schema`,
31 and an individual piece of news is a :py:class:`NewsItem`.
32
21b06da @slinkp Better docs on spreadsheet loading, newsitem core fields
slinkp authored
33 .. _newsitem_core_fields:
34
35 Core Fields of NewsItems
36 ------------------------
37
38 The :py:class:`NewsItem <NewsItem>` model in itself is generic -- a
39 lowest-common denominator of each piece of news on the site. It has:
40
41 * title (required)
42 * url (optional)
43 * description (optional)
44 * location_name (optional but highly desirable; can be reverse-geocoded from location if you have one)
45 * location (optional but highly desirable; can be geocoded from location_name if you have one)
46 * item_date (default: today)
47 * pub_date (default: current date and time)
48
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
49 .. _newsitem_attributes:
50
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
51 Flexible data: SchemaFields and Attributes
52 ------------------------------------------
53
21b06da @slinkp Better docs on spreadsheet loading, newsitem core fields
slinkp authored
54 If you'd like to extend your NewsItems to include
55 more information, you can use :py:class:`SchemaFields <SchemaField>`.
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
56
57 Each piece of news is described by:
58
21b06da @slinkp Better docs on spreadsheet loading, newsitem core fields
slinkp authored
59 * One :py:class:`NewsItem` instance, with just the core fields like
60 title and description.
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
61
62 * One corresponding :py:class:`Attribute` instance, which is a dictionary-like
63 object containing extra data, and is available as ``newsitem.attributes``.
64
65 * One :py:class:`Schema` that identifies the "type" of NewsItem; and
66
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
67 * A set of :py:class:`SchemaFields <SchemaField>`, each of which describes:
c7c4cfb @slinkp more scraper doc work
slinkp authored
68 a valid key for the attributes dictionary; the type of its allowed values;
69 and some configuration metadata about how to display and use that attribute.
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
70
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
71 This is intended to be flexible enough for real-world news data, while
72 still allowing for fast database queries. For more background,
73 you might be interested in the video
74 `Behind the scenes of EveryBlock.com <http://blip.tv/file/1957362>`_.
75
76 .. admonition:: Why not NoSQL?
77
78 ebpub was originally written around the time of the rise in popularity of
79 schemaless document storage systems commonly lumped together as
80 `nosql <http://en.wikipedia.org/wiki/Nosql>`_,
81 which would have made this one aspect of ebpub rather trivial.
82 However, at the time, none of them had much in the way of
83 geospatial capabilities; even today, none of them are as
84 full-featured as PostGIS.
85
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
86
c7c4cfb @slinkp more scraper doc work
slinkp authored
87 Examples might make this clearer. To assign the whole ``attributes`` dictionary::
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
88
89 ni = NewsItem.objects.get(...)
c7c4cfb @slinkp more scraper doc work
slinkp authored
90 ni.attributes = {'sale_price': 19}
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
91 # There is no need to call ni.save() or ni.attributes.save();
92 # the assignment operation does that behind the scenes.
93
94 To assign a single value::
95
c7c4cfb @slinkp more scraper doc work
slinkp authored
96 ni.attributes['sale_price'] = 19
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
97 # Again there is no need to save() anything explicilty.
98
99 To get a value::
100
c7c4cfb @slinkp more scraper doc work
slinkp authored
101 print ni.attributes['sale_price']
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
102
103 Or, from a database perspective: The "db_attribute" table stores
104 arbitrary attributes for each NewsItem, and the "db_schemafield" table
105 is the key for those attributes.
106 A SchemaField says, for example, that
107 the "int01" column in the db_attribute table for the "real estate
108 sales" Schema corresponds to the "sale price".
c7c4cfb @slinkp more scraper doc work
slinkp authored
109
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
110 We'll walk through this example in detail below.
111
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
112
113 Detailed Example
114 ------------------
115
116 Imagine you have a "real estate sales"
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
117 Schema, with an id of 5. Say, for each sale, you want to store the following
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
118 information:
119
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
120 * address
121 * sale date
122 * sale price
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
123
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
124 The first two fields should go in ``NewsItem.location_name`` and ``NewsItem.item_date``,
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
125 respectively -- there's no reason to put them in the Attribute table, because
126 the NewsItem table has a slot for them.
127
128 Sale price is a number (we'll assume it's an integer), so create a
129 :py:class:`SchemaField` defining it, with these values:
130
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
131 .. code-block:: python
132 :linenos:
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
133
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
134 field = SchemaField(schema_id = 5,
135 name = 'sale_price',
136 real_name = 'int01',
137 pretty_name = 'sale price',
138 pretty_name_plural = 'sale prices',
139 display = True,
140 display_order = 1,
141 is_searchable = False,
142 )
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
143
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
144 Line 1. ``schema_id`` is the id of our "real estate sales" schema.
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
145
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
146 Line 2. ``name`` is the alphanumeric-and-underscores-only name for this field.
147 (Used in URLs, and as the key for ``newsitem.attributes``,
148 and for the
149 :py:meth:`NewsItemQuerySet.by_attribute` method.)
150 This value must be unique with respect to the schema_id.
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
151
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
152 Line 3. ``real_name`` is the column to use in the db_attribute model. Choices are:
153 int01-07, text01, bool01-05, datetime01-04, date01-05, time01-02,
154 varchar01-05. This value must be unique with respect to the schema_id.
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
155
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
156 Lines 4-5. ``pretty_name`` and ``pretty_name_plural`` are the human-readable name
157 for this attribute.
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
158
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
159 Line 6. ``display`` controls whether to display the value on the site.
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
160
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
161 Line 7: `sort_order`` - An integer representing what order it should be displayed
162 on newsitem_detail pages.
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
163
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
164 Line 8: ``is_searchable`` - Whether you can do text searches on this field.
165 Only works with text or varchar fields.
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
166
167 Once you've created this SchemaField, the value of "int01" for any db_attribute
168 row with schema_id=5 will be the sale price.
169
170 Having done all that, using the field is as easy as::
171
172 from ebpub.db.models import NewsItem
173 ni = NewsItem(schema__id=5, title='the title', description='the description', ...)
174 ni.save()
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
175 ni.attributes['sale_price'] = 59
176
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
177
178 Searching by Attributes
179 ------------------------
180
181 There is a simple API for searching NewsItems by attribute values:
182
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
183 sale_price = SchemaField.objects.get(schema__id=5, name='sale_price')
184 NewsItem.objects.filter(schema__id=5).by_attribute(sale_price, 59)
185
186 For details see :py:meth:`NewsItemQuerySet.by_attribute`.
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
187
188
189 Attributes: Under the hood
190 ---------------------------
191
192 The dictionary-like API is provided thanks to the combination of
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
193 the ``AttributesDescriptor``, ``AttributeDict``, and
194 :py:class:`Attribute` classes; see the source code for details.
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
195
196 Images
197 ------
198
199 NewsItems have a ``newsitemimage_set`` reverse relationship
200 with the :py:class:`NewsItemImage` model, allowing any number of
201 images to be associated with one NewsItem.
202 All the images for a NewsItem can be retrieved via
203 ``item.newsitemimage_set.all()``.
204
205
206 Dates
207 -----
208
209 The distinction between ``NewsItem.pub_date`` and ``NewsItem.item_date``
210 is intended for data sets where there's
211 a lag in publishing or where the data is updated infrequently or
212 irregularly.
213
214 For example, on EveryBlock.com, Chicago crime data is published a week
215 after it is reported, so a crime's ``item_date`` is the day of the
216 crime report, whereas the ``pub_date`` is the day the data was
217 published to EveryBlock.com.
218
219 Location, location, location
220 -----------------------------
221
222 NewsItems have several distinct notions of location:
223
224 * ``NewsItem.location_name`` is a human-readable version of the location;
225 it can be anything, but typically it describes an address,
226 block, geographic area, or landmark.
227
228 * ``NewsItem.location`` is used frequently; typically a point, and
229 typically set when scraping data, by geocoding if
230 not provided in the source data. This is used in
231 many views for finding relevant NewsItems (indirectly; actually
232 see below about NewsItemLocations).
233
234 * ``NewsItem.location_set`` is a convenient way to get
235 all :py:class:`Locations <Location>` that overlap this item's ``location``.
236 It's a many-to-many relationship (via the
237 NewsItemLocation model). The NewsItemLocations are created by a sql trigger
238 whenever self.location changes; not set by any python code. Used
239 in various views for filtering.
240
241 * ``NewsItem.location_object`` is a single :py:class:`Location` reference;
242 theoretically to be explicitly assigned by a scraper script when
243 there's no known address or geographic point for this NewsItem
244 but we know the name of the general area it's within.
245
246 For example, many stories might mention a town or city name, or a
247 police report might tell you the precinct. In practice, this field
248 is usually Null; more importantly it's only used currently
249 (2011-12-06) by self.location_url(), for linking back to a location
250 view from a newsitem view. (Example of where everyblock.com uses
251 this: NYC crime aggregates,
252 eg. http://nyc.everyblock.com/crime/by-date/2010/8/23/3364632/ )
253
254 See also this ticket http://developer.openblockproject.org/ticket/93
255 about possibly making more use of self.location_object.
256
257
2a57ebf @slinkp Document all scripts in ebpub/db/bin. Closes #96
slinkp authored
258 .. _aggregates:
259
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
260 Aggregates
261 ----------
262
263 Several parts of ebpub display aggregate totals of NewsItems for a particular
264 Schema; for example, charts of how many were added each day.
265
266 Because these calculations can be expensive, there's an infrastructure
267 for caching the aggregate numbers regularly in separate tables (db_aggregate*).
268
2a57ebf @slinkp Document all scripts in ebpub/db/bin. Closes #96
slinkp authored
269 To do this, just run the :py:mod:`update_aggregates <ebpub.db.bin.update_aggregates>`
270 script on the command line.
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
271
272 You'll want to do this on a regular basis, depending on how often you update
273 your data. **Some parts of the site (such as charts) will not be visible** until
274 you populate the aggregates.
275
276 .. _future_events:
277
278 Event-like News Types
279 ----------------------
280
281 In order for OpenBlock to treat a news type as being about
282 (potentially) future events, rather than news from the (recent) past,
283 there is a simple convention that you should follow:
284
285 1. Set the schema's ``is_event=True``.
286
287 2. Add a SchemaField with ``name='start_time'``. It should be a Time
288 field, i.e. ``real_name`` should be one of ``time01``, ``time02``,
289 etc. Leave ``is_filter``, ``is_lookoup``, ``is_searchable``, and
290 ``is_charted`` set to False. The ``pretty_name`` can be whatever
291 you like of course.
292
293 3. Optionally add another SchemaField with ``name='end_time'``, if your data
294 source will include this information.
295
296 4. When adding NewsItems of this type, the NewsItem's ``item_date``
297 field should be set to the date on which the event will (or already
298 did) take place, and ``attributes['start_time']`` should be set to
299 the (local) time it will start, and ``attributes['end_time']``
300 (if needed) should be set to the (local) end time.
301
302 All-day events can be represented by leaving ``start_time`` empty.
303
304 There is no special support for repeating events or other advanced
305 calendar features.
306
307 .. _lookups:
308
309 Lookups
310 ========
311
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
312 Lookups are a way to reduce duplication and support fast searching
313 for similar NewsItems.
314
315 Consider the "real estate" schema we talked about in earlier examples.
316 We want to add a field representing "property type" for each real estate sale
317 NewsItem.
318
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
319 We could store it as a varchar field (in which case we'd set
320 real_name='varchar01') -- but that would cause a lot of duplication and
321 redundancy, because there are only a couple of property types -- the set
322 ['single-family', 'condo', 'land', 'multi-family']. To represent this set,
323 we can use a Lookup -- a way to normalize the data.
324
325 To do this, set ``SchemaField.is_lookup=True`` and make sure to use an 'int' column
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
326 for SchemaField.real_name. For example:
327
328 .. code-block:: python
329
330 field = SchemaField(schema_id = 5,
331 name = 'property_type',
332 real_name = 'int02',
333 is_lookup=True,
334 pretty_name = 'property type',
335 pretty_name_plural = 'property types',
336 display = True,
337 display_order = 2,
338 )
339
340
341 Then, for each record, get or create a :py:class:`Lookup`
342 object that represents the data, and use
343 the Lookup's id in the appropriate attribute field.
344 For example:
345
346 .. code-block:: python
347
348 condo = Lookup.objects.get_or_create_lookup(
349 schema_field=field, name='condo')
350 newsitem['property_type'] = condo
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
351
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
352 Note the convenience function :py:meth:`Lookup.objects.get_or_create_lookup() <LookupManager.get_or_create_lookup>`.
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
353
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
354 Many-to-many Lookups
355 --------------------
356
357 Sometimes a :py:class:`NewsItem` has multiple values for a single attribute.
358 For example, a restaurant inspection can have multiple violations. In
359 this case, you can use a many-to-many Lookup. To do this, just set
360 ``SchemaField.is_lookup=True`` as before, but use a varchar field for
361 the ``SchemaField.real_name``. Then, in the db_attribute column, set
362 the value to a string of comma-separated integers of the Lookup IDs:
363
364 .. code-block:: python
365
366 field = SchemaField.objects.get(schema_id=5, name='property_type')
367 field.real_name = 'varchar01'
368 field.save()
369
370 newsitem.attributes['property_type'] = '1,2,3'
371
372
373 .. _featured_lookups:
374
375 "Featured" Lookups
376 -----------------------
377
378 A :py:class:`Lookup` instance can have ``featured=True``, which you can use to
379 mark some Lookup values as "special" for eg. navigation purposes.
380 One example use case would be special tags or keywords that mark
381 any relevant NewsItems for inclusion in a special part of your homepage.
382
383 To work with Lookups that are marked with ``featured=True``, there are
384 several useful tools:
385
386 * :py:meth:`Lookup.objects.featured_lookups_for(newsitem, name) <LookupManager.featured_lookups_for>`
387 will, given a NewsItem instance and an attribute name, find all
388 featured Lookups for that attribute which are relevant to that NewsItem.
389 * The same functionality is available in templates via the
390 :py:func:`featured_lookups_for_item <ebpub.db.templatetags.eb.featured_lookups_for_item>` template tag.
391 * :py:func:`get_featured_lookups_by_schema <ebpub.db.templatetags.eb.get_featured_lookups_by_schema>`
392 is a tag that finds all currently featured Lookups, and URLs to find
393 relevant NewsItems.
394
395
396 .. _charting_and_filtering:
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
397
398 Charting and filtering lookups
399 ------------------------------
400
401 Set ``SchemaField.is_filter=True`` on a lookup SchemaField, and the detail page for
402 the NewsItem (newsitem_detail) will automatically link that field to a page
403 that lists all of the other NewsItems in that Schema with that particular
404 Lookup value.
405
406 Set ``SchemaField.is_charted=True`` on a lookup SchemaField, and the detail page
407 for the Schema (schema_detail) will include a chart of the top 10 lookup values
408 in the last 30 days' worth of data. Similar charts are on the
409 place detail overview page. (This assumes aggregates are populated; see
410 the Aggregates section below.)
411
412
413 module contents
414 ================
c7c4cfb @slinkp more scraper doc work
slinkp authored
415 """
1d1815b @slinkp Sensible defaults on DateFields and DateTimeFields. Not using 'auto_n…
slinkp authored
416
0c7d337 @slinkp getting set up to add optional images to newsitems
slinkp authored
417 from django.conf import settings
5c9826f initial import
Don Kukral authored
418 from django.contrib.gis.db import models
419 from django.contrib.gis.db.models import Count
ff0a9b6 @slinkp Change newsitem_detail URL: remove useless date arguments.
slinkp authored
420 from django.core import urlresolvers
4f9f411 @slinkp Caching for allowed_schema_ids, for performance.
slinkp authored
421 from django.core.cache import cache
a9a4d9f @slinkp ValueError -> ValidatonError to make admin form nicer
slinkp authored
422 from django.core.exceptions import ValidationError
5c9826f initial import
Don Kukral authored
423 from django.db import connection, transaction
4f9f411 @slinkp Caching for allowed_schema_ids, for performance.
slinkp authored
424 from ebpub.db import constants
220ca2c @ltucker Add some extra info to improve location and place tagging, split Miss…
ltucker authored
425 from ebpub.geocoder.parser.parsing import normalize
5309287 @slinkp Overhaul of olwidget integration.
slinkp authored
426 from ebpub.utils.geodjango import flatten_geomcollection
427 from ebpub.utils.geodjango import ensure_valid
5c9826f initial import
Don Kukral authored
428 from ebpub.utils.text import slugify
efc28d9 @slinkp * Add a foreign key between NewsItemImage and NewsItem.
slinkp authored
429 from .fields import OpenblockImageField
de7c1fe @slinkp Yet more de-hardcoded URLs.
slinkp authored
430
5c9826f initial import
Don Kukral authored
431 import datetime
5309287 @slinkp Overhaul of olwidget integration.
slinkp authored
432 import logging
f65f953 @slinkp Fix two errors triggered when NewsItems are saved for a schema with n…
slinkp authored
433 import re
5c9826f initial import
Don Kukral authored
434
5309287 @slinkp Overhaul of olwidget integration.
slinkp authored
435 logger = logging.getLogger('ebpub.db.models')
5c9826f initial import
Don Kukral authored
436
0eb2538 @slinkp Monkeypatches that need settings can't be imported before settings ar…
slinkp authored
437 # Need these monkeypatches for "natural key" support during fixture load/dump.
438 import ebpub.monkeypatches
439 ebpub.monkeypatches.patch_once()
3b0dc05 @slinkp Update stuff using Query.extra_where to use QuerySet.extra(where...).…
slinkp authored
440
9e0becb @slinkp Deleted SchemaInfo, moved all that metadata into Schema. Observed per…
slinkp authored
441 FREQUENCY_CHOICES = ('Hourly', 'Throughout the day', 'Daily', 'Twice a week', 'Weekly', 'Twice a month', 'Monthly', 'Quarterly', 'Sporadically', 'No longer updated')
442 FREQUENCY_CHOICES = [(a, a) for a in FREQUENCY_CHOICES]
443
f65f953 @slinkp Fix two errors triggered when NewsItems are saved for a schema with n…
slinkp authored
444 logger = logging.getLogger('ebpub.db.models')
0eb2538 @slinkp Monkeypatches that need settings can't be imported before settings ar…
slinkp authored
445
1fa4fec @slinkp SchemaField admin: limit choices of real_name to valid values
slinkp authored
446 def get_valid_real_names():
447 """
448 Field names of ``Attribute``, suitable for use as
449 ``SchemaField.real_name``.
450 """
451 for name in sorted(Attribute._meta.get_all_field_names()):
452 if re.search(r'\d\d$', name):
453 yield name
0eb2538 @slinkp Monkeypatches that need settings can't be imported before settings ar…
slinkp authored
454
455
5c9826f initial import
Don Kukral authored
456 def field_mapping(schema_id_list):
457 """
458 Given a list of schema IDs, returns a dictionary of dictionaries, mapping
459 schema_ids to dictionaries mapping the fields' name->real_name.
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
460 Example return value::
461
5c9826f initial import
Don Kukral authored
462 {1: {u'crime_type': 'varchar01', u'crime_date', 'date01'},
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
463 2: {u'permit_number': 'int01', 'to_date': 'date01'},
5c9826f initial import
Don Kukral authored
464 }
465 """
466 result = {}
467 for sf in SchemaField.objects.filter(schema__id__in=(schema_id_list)).values('schema', 'name', 'real_name'):
468 result.setdefault(sf['schema'], {})[sf['name']] = sf['real_name']
469 return result
470
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
471
e984c8c @slinkp Oops, the .update() method was defined in the wrong place; fixes a fa…
slinkp authored
472 class SchemaQuerySet(models.query.GeoQuerySet):
4f9f411 @slinkp Caching for allowed_schema_ids, for performance.
slinkp authored
473
474 def update(self, *args, **kwargs):
475 # Django doesn't provide pre/post_update signals, rats.
476 # See https://code.djangoproject.com/ticket/13021
477 # So we define one and send it here.
e984c8c @slinkp Oops, the .update() method was defined in the wrong place; fixes a fa…
slinkp authored
478 result = super(SchemaQuerySet, self).update(*args, **kwargs)
4f9f411 @slinkp Caching for allowed_schema_ids, for performance.
slinkp authored
479 post_update.send(sender=Schema)
480 return result
481
e984c8c @slinkp Oops, the .update() method was defined in the wrong place; fixes a fa…
slinkp authored
482
483 class SchemaManager(models.Manager):
484
485 _allowed_ids_cache_key = 'allowed_schema_ids__all'
486
487 def update(self, *args, **kwargs):
488 return self.get_query_set().update(*args, **kwargs)
489
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
490 def get_by_natural_key(self, slug):
491 return self.get(slug=slug)
492
9e0becb @slinkp Deleted SchemaInfo, moved all that metadata into Schema. Observed per…
slinkp authored
493 def get_query_set(self):
fdaebb9 @slinkp comment re #82
slinkp authored
494 """Warning: This breaks manage.py dumpdata.
495 See bug #82.
496
497 """
e984c8c @slinkp Oops, the .update() method was defined in the wrong place; fixes a fa…
slinkp authored
498 return SchemaQuerySet(model=self.model, using=self._db).defer(
9e0becb @slinkp Deleted SchemaInfo, moved all that metadata into Schema. Observed per…
slinkp authored
499 'short_description',
500 'summary',
501 'source',
502 'short_source',
503 'update_frequency',
504 )
505
01971bc @slinkp Infrastructure for restricted/premium content based on schema type: a…
slinkp authored
506 def allowed_schema_ids(self):
507 """
508 Useful for filtering out schemas (or things related to
509 schemas) based on the current Manager.
510 """
4f9f411 @slinkp Caching for allowed_schema_ids, for performance.
slinkp authored
511 ids = cache.get(self._allowed_ids_cache_key, None)
512 if ids is None:
513 ids = [s['id'] for s in self.all().values('id')]
514 cache.set(self._allowed_ids_cache_key, ids, constants.ALLOWED_IDS_CACHE_TIME)
515 return ids
9e0becb @slinkp Deleted SchemaInfo, moved all that metadata into Schema. Observed per…
slinkp authored
516
dd75abd @slinkp whitespace
slinkp authored
517
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
518 class SchemaPublicManager(SchemaManager):
519
4f9f411 @slinkp Caching for allowed_schema_ids, for performance.
slinkp authored
520 _allowed_ids_cache_key = 'allowed_schema_ids__public'
521
5c9826f initial import
Don Kukral authored
522 def get_query_set(self):
523 return super(SchemaManager, self).get_query_set().filter(is_public=True)
524
dd75abd @slinkp whitespace
slinkp authored
525
5c9826f initial import
Don Kukral authored
526 class Schema(models.Model):
188c1d2 @slinkp Much commenting on how NewsItems work
slinkp authored
527 """
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
528 Describes a type of :py:class:`NewsItem`. A NewsItem has exactly one Schema,
529 which describes its Attributes, via associated :py:class:`SchemaFields <SchemaField>`.
57a93d2 @slinkp another comment
slinkp authored
530
531 nb. to get all NewsItem instances for a Schema, you can do the usual as per
532 http://docs.djangoproject.com/en/dev/topics/db/queries/#backwards-related-objects:
533 schema.newsitem_set.all()
afb08b3 @slinkp update changelog
slinkp authored
534
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
535 nb. Some Schemas may not be visible to some users, if eg.
536 ``is_public=False``. To abstract this, use the
537 :py:func:`ebpub.utils.view_utils.get_schema_manager` function,
538 rather than directly using ``Schema.objects`` or ``Schema.public_objects``.
afb08b3 @slinkp update changelog
slinkp authored
539
94d3182 @slinkp More convenient by_request() API for restricting NewsItems to only sc…
slinkp authored
540 (To filter NewsItems appropriately, do NewsItem.objects.by_request(request)
541 which will take care of using the right Schema manager.)
188c1d2 @slinkp Much commenting on how NewsItems work
slinkp authored
542 """
5c9826f initial import
Don Kukral authored
543 name = models.CharField(max_length=32)
544 plural_name = models.CharField(max_length=32)
b4c7365 @slinkp Move Schema's comments into help_text for its fields.
slinkp authored
545 indefinite_article = models.CharField(max_length=2,
546 help_text="eg.'a' or 'an'")
e3056db @slinkp Use SlugField for slugs.
slinkp authored
547 slug = models.SlugField(max_length=32, unique=True)
b4c7365 @slinkp Move Schema's comments into help_text for its fields.
slinkp authored
548 min_date = models.DateField(
1d1815b @slinkp Sensible defaults on DateFields and DateTimeFields. Not using 'auto_n…
slinkp authored
549 help_text="The earliest available pub_date for this Schema",
550 default=lambda: datetime.date(1970, 1, 1))
830668f @slinkp comments & help_text
slinkp authored
551 last_updated = models.DateField(
552 help_text=u"Last date any NewsItems were loaded for this Schema.")
b4c7365 @slinkp Move Schema's comments into help_text for its fields.
slinkp authored
553 date_name = models.CharField(
554 max_length=32, default='Date',
555 help_text='Human-readable name for the item_date field')
dcf27b3 make schema creation slightly less detailed by adding some defaults.
Luke Tucker authored
556 date_name_plural = models.CharField(max_length=32, default='Dates')
b4c7365 @slinkp Move Schema's comments into help_text for its fields.
slinkp authored
557 importance = models.SmallIntegerField(
558 default=0,
559 help_text='Bigger number is more important; used for sorting in some places.')
560 is_public = models.BooleanField(
561 db_index=True, default=False,
7fcd1a6 @slinkp A bit of commentary about wtf 'special reports' are.
slinkp authored
562 help_text="Set False if you want only people with the staff cookie to be able to see it.")
563 is_special_report = models.BooleanField(
564 default=False,
565 help_text="Whether to use the schema_detail_special_report view for these items, eg. for displaying items that have a known general Location but not a specific point.")
5c9826f initial import
Don Kukral authored
566
d02b127 @slinkp Work in progress: Future events. Refs #246
slinkp authored
567 is_event = models.BooleanField(
568 default=False,
569 help_text="Whether these items are (potentially) future events rather than news in the past.")
570
b4c7365 @slinkp Move Schema's comments into help_text for its fields.
slinkp authored
571 can_collapse = models.BooleanField(
572 default=False,
573 help_text="Whether RSS feed should collapse many of these into one.")
5c9826f initial import
Don Kukral authored
574
b4c7365 @slinkp Move Schema's comments into help_text for its fields.
slinkp authored
575 has_newsitem_detail = models.BooleanField(
576 default=False,
577 help_text="Whether to show a detail page for NewsItems of this schema, or redirect to the NewsItem's source URL instead.")
5c9826f initial import
Don Kukral authored
578
ea38661 @ltucker rig in beginnings of commenting for news items
ltucker authored
579 allow_comments = models.BooleanField(
580 default=False,
581 help_text="Whether to allow users to add comments to NewsItems of the schema. Only applies to items with detail page."
582 )
583
f0d8d9b @slinkp Progress on moderation/flagging: admin ui mostly works, but widget ne…
slinkp authored
584 allow_flagging = models.BooleanField(
585 default=False,
33cfbfa @slinkp Reviewed docs re. commenting. Closes #252
slinkp authored
586 help_text="Whether to allow users to flag NewsItems of this schema as spam or inappropriate."
f0d8d9b @slinkp Progress on moderation/flagging: admin ui mostly works, but widget ne…
slinkp authored
587 )
588
b4c7365 @slinkp Move Schema's comments into help_text for its fields.
slinkp authored
589 allow_charting = models.BooleanField(
590 default=False,
591 help_text="Whether aggregate charts are displayed on the home page of this Schema")
5c9826f initial import
Don Kukral authored
592
b4c7365 @slinkp Move Schema's comments into help_text for its fields.
slinkp authored
593 uses_attributes_in_list = models.BooleanField(
594 default=False,
595 help_text="Whether attributes should be preloaded for NewsItems of this Schema, in the list view")
5c9826f initial import
Don Kukral authored
596
597
b4c7365 @slinkp Move Schema's comments into help_text for its fields.
slinkp authored
598 number_in_overview = models.SmallIntegerField(
599 default=5,
600 help_text="Number of records to show on place_overview")
601
830668f @slinkp comments & help_text
slinkp authored
602 # TODO: maybe this should be either a FileField or a FilePathField instead?
b4c7365 @slinkp Move Schema's comments into help_text for its fields.
slinkp authored
603 map_icon_url = models.TextField(
604 blank=True, null=True,
2a14f51 @slinkp Merge branch 'master' into autodoc
slinkp authored
605 help_text="Set this to a URL to a small image icon and it will be displayed on maps. Should be roughly 40x40 pixels. Optional.",
606 )
0b9338e @slinkp Support relative URIs in map_icon_img for both db.Schema and streets.…
slinkp authored
607
608 def get_map_icon_url(self):
609 # Could be relative.
610 url = self.map_icon_url or u''
611 if url and not (url.startswith('/') or url.startswith('http')):
612 url = '%s/%s' % (settings.STATIC_URL.rstrip('/'), url)
613 return url
614
b4c7365 @slinkp Move Schema's comments into help_text for its fields.
slinkp authored
615 map_color = models.CharField(
616 max_length=255, blank=True, null=True,
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
617 help_text="CSS color code used on maps to display this type of news. eg #FF0000. Only used if map_icon_url is not set. Optional.")
5c9826f initial import
Don Kukral authored
618
f67a602 @slinkp Add a configurable edit_window (in hours) to schemas, to limit or pro…
slinkp authored
619
620 edit_window = models.FloatField(
621 blank=True, default=0.0,
622 help_text=u"How long, in hours, the creator of an item is allowed to edit it. Set to 0 to disallow edits by non-Admin users. Set to -1 to allow editing forever."
623 )
624
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
625 objects = SchemaManager()
626 public_objects = SchemaPublicManager()
5c9826f initial import
Don Kukral authored
627
628 def __unicode__(self):
629 return self.name
630
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
631 def natural_key(self):
632 return (self.slug,)
633
528587e @slinkp Added get_absolute_url() method to models where it makes sense, so ad…
slinkp authored
634 def get_absolute_url(self):
454f867 @slinkp Add a few more indexes; also comments
slinkp authored
635 return urlresolvers.reverse('ebpub-schema-filter', args=(self.slug,))
5c9826f initial import
Don Kukral authored
636
528587e @slinkp Added get_absolute_url() method to models where it makes sense, so ad…
slinkp authored
637 # Backward compatibility.
638 url = get_absolute_url
639
e25a917 @slinkp comment
slinkp authored
640 ######################################################################
641 # Metadata fields that used to live in a separate SchemaInfo model.
a03aaad More defaults
Luke Tucker authored
642 short_description = models.TextField(blank=True, default='')
643 summary = models.TextField(blank=True, default='')
b4c7365 @slinkp Move Schema's comments into help_text for its fields.
slinkp authored
644 source = models.TextField(blank=True, default='',
a0b8eb5 @slinkp Remove the not-so-useful Schema About view. Closes #228
slinkp authored
645 help_text='Where this information came from, as one or more URLs, one per line.')
646 short_source = models.CharField(max_length=128, blank=True, default='One-line description of where this information came from.')
9e0becb @slinkp Deleted SchemaInfo, moved all that metadata into Schema. Observed per…
slinkp authored
647 update_frequency = models.CharField(max_length=64, blank=True, default='',
648 choices=FREQUENCY_CHOICES)
5c9826f initial import
Don Kukral authored
649
a11a8b0 @slinkp Lots more columns, filters, and usable default ordering in admin UI; …
slinkp authored
650 class Meta:
651 ordering = ('name',)
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
652
dd75abd @slinkp whitespace
slinkp authored
653
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
654 class SchemaFieldManager(models.Manager):
655
656 def get_by_natural_key(self, schema_slug, real_name):
657 return self.get(schema__slug=schema_slug, real_name=real_name)
658
dd75abd @slinkp whitespace
slinkp authored
659
5c9826f initial import
Don Kukral authored
660 class SchemaField(models.Model):
188c1d2 @slinkp Much commenting on how NewsItems work
slinkp authored
661 """
662 Describes the meaning of one Attribute field for one Schema type.
663 """
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
664 objects = SchemaFieldManager()
665
5c9826f initial import
Don Kukral authored
666 schema = models.ForeignKey(Schema)
c0a594d @slinkp Replace SchemaField.name with SchemaField.slug, since it was already …
slinkp authored
667
668 pretty_name = models.CharField(
669 max_length=32,
670 help_text="Human-readable name of this field, for display."
671 )
672 pretty_name_plural = models.CharField(
673 max_length=32,
674 help_text="Plural human-readable name"
675 )
2d4bb55 @slinkp Partial revert of c0a594d69e89ba31b7a57 - removing "name" was a bad i…
slinkp authored
676
677 name = models.SlugField(max_length=32)
c0a594d @slinkp Replace SchemaField.name with SchemaField.slug, since it was already …
slinkp authored
678
679 real_name = models.CharField(
680 max_length=10,
681 help_text="Column name in the Attribute model. 'varchar01', 'varchar02', etc.",
1fa4fec @slinkp SchemaField admin: limit choices of real_name to valid values
slinkp authored
682 choices=((name, name) for name in get_valid_real_names()),
c0a594d @slinkp Replace SchemaField.name with SchemaField.slug, since it was already …
slinkp authored
683 )
684 display = models.BooleanField(
685 default=True,
686 help_text='Whether to display value on the public site.'
687 )
688 is_lookup = models.BooleanField(
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
689 blank=True, default=False,
c0a594d @slinkp Replace SchemaField.name with SchemaField.slug, since it was already …
slinkp authored
690 help_text='Whether the value is a foreign key to Lookup.'
691 )
692 is_filter = models.BooleanField(
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
693 blank=True, default=False,
c0a594d @slinkp Replace SchemaField.name with SchemaField.slug, since it was already …
slinkp authored
694 help_text='Whether to link to list of items with the same value in this field. Assumes is_lookup=True.'
695 )
696 is_charted = models.BooleanField(
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
697 blank=True, default=False,
698 help_text='Whether the schema detail view displays a chart for this field; also see "trends" tabs on place overview page. Assumes is_lookup=True.'
c0a594d @slinkp Replace SchemaField.name with SchemaField.slug, since it was already …
slinkp authored
699 )
a03aaad More defaults
Luke Tucker authored
700 display_order = models.SmallIntegerField(default=10)
c0a594d @slinkp Replace SchemaField.name with SchemaField.slug, since it was already …
slinkp authored
701 is_searchable = models.BooleanField(
702 default=False,
703 help_text="Whether the value is searchable by content. Doesn't make sense if is_lookup=True."
704 )
5c9826f initial import
Don Kukral authored
705
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
706 def natural_key(self):
707 return (self.schema.slug, self.real_name)
708
90b5568 @slinkp Enforce documented uniqueness for SchemaField schema + real_name. An…
slinkp authored
709 class Meta(object):
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
710 unique_together = (('schema', 'real_name'),
711 ('schema', 'name'),
712 )
a11a8b0 @slinkp Lots more columns, filters, and usable default ordering in admin UI; …
slinkp authored
713 ordering = ('pretty_name',)
90b5568 @slinkp Enforce documented uniqueness for SchemaField schema + real_name. An…
slinkp authored
714
5c9826f initial import
Don Kukral authored
715 def __unicode__(self):
716 return u'%s - %s' % (self.schema, self.name)
717
764e1b9 @slinkp Use modern @property syntax; simplify SchemaField.is_type()
slinkp authored
718 @property
719 def datatype(self):
5c9826f initial import
Don Kukral authored
720 return self.real_name[:-2]
721
722 def is_type(self, *data_types):
723 """
724 Returns True if this SchemaField is of *any* of the given data types.
725
726 Allowed values are 'varchar', 'date', 'time', 'datetime', 'bool', 'int'.
727 """
764e1b9 @slinkp Use modern @property syntax; simplify SchemaField.is_type()
slinkp authored
728 return self.datatype in data_types
5c9826f initial import
Don Kukral authored
729
730 def is_many_to_many_lookup(self):
731 """
732 Returns True if this SchemaField is a many-to-many lookup.
733 """
734 return self.is_lookup and not self.is_type('int')
0f82c51 @slinkp Default News and GeoReport schemas are now set up by ebpub migrations;
slinkp authored
735 is_many_to_many_lookup.boolean = True
5c9826f initial import
Don Kukral authored
736
737 def all_lookups(self):
738 if not self.is_lookup:
739 raise ValueError('SchemaField.all_lookups() can only be called if is_lookup is True')
740 return Lookup.objects.filter(schema_field__id=self.id).order_by('name')
741
742 def browse_by_title(self):
743 "Returns FOO in 'Browse by FOO', for this SchemaField."
744 if self.is_type('bool'):
745 return u'whether they %s' % self.pretty_name_plural
746 return self.pretty_name
747
748 def smart_pretty_name(self):
749 """
750 Returns the pretty name for this SchemaField, taking into account
751 many-to-many fields.
752 """
753 if self.is_many_to_many_lookup():
754 return self.pretty_name_plural
755 return self.pretty_name
756
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
757
758 class LocationTypeManager(models.Manager):
759 def get_by_natural_key(self, slug):
760 return self.get(slug=slug)
761
dd75abd @slinkp whitespace
slinkp authored
762
5c9826f initial import
Don Kukral authored
763 class LocationType(models.Model):
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
764 '''
765 Used for classifying and grouping :py:class:`Location`.
766
767 You'll want to create at least one LocationType with the slug set to
768 the same value as ``settings.DEFAULT_LOCTYPE_SLUG``, because that's
769 used in various default URLs. By default this is set to
770 "neighborhoods".
771 '''
38501f6 @slinkp Turn more comments into doocstrings and help text
slinkp authored
772 name = models.CharField(max_length=255,
773 help_text='for example, "Ward" or "Congressional District"')
774 plural_name = models.CharField(max_length=64)
f5a8696 @slinkp Docs: A LOT more about setting up geographic information.
slinkp authored
775 scope = models.CharField(max_length=64,
31c39da @slinkp scope doesn't do anything
slinkp authored
776 help_text='e.g., "Chicago" or "U.S.A.". For display only; has no effect.')
e3056db @slinkp Use SlugField for slugs.
slinkp authored
777 slug = models.SlugField(max_length=32, unique=True)
c450ce2 @slinkp Minor LocationType admin UI tweaks
slinkp authored
778 is_browsable = models.BooleanField(
779 default=True, help_text="Whether this is displayed on location_type_list.") # XXX unused??
780 is_significant = models.BooleanField(
781 default=True,
38501f6 @slinkp Turn more comments into doocstrings and help text
slinkp authored
782 help_text="Whether this can be used to filter NewsItems, shows up in 'nearby locations', etc."
c450ce2 @slinkp Minor LocationType admin UI tweaks
slinkp authored
783 )
5c9826f initial import
Don Kukral authored
784
785 def __unicode__(self):
786 return u'%s, %s' % (self.name, self.scope)
787
528587e @slinkp Added get_absolute_url() method to models where it makes sense, so ad…
slinkp authored
788 def get_absolute_url(self):
2d83842 @slinkp De-hardcode more model URLs. Refs #69
slinkp authored
789 return urlresolvers.reverse('ebpub-loc-type-detail', args=(self.slug,))
5c9826f initial import
Don Kukral authored
790
528587e @slinkp Added get_absolute_url() method to models where it makes sense, so ad…
slinkp authored
791 # Backward compatibility.
792 url = get_absolute_url
793
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
794 def natural_key(self):
795 return (self.slug,)
796
a11a8b0 @slinkp Lots more columns, filters, and usable default ordering in admin UI; …
slinkp authored
797 class Meta:
798 ordering = ('name',)
799
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
800 objects = LocationTypeManager()
801
802
803 class LocationManager(models.GeoManager):
804 def get_by_natural_key(self, slug, location_type_slug):
1b4c87c @ltucker fix Location get_by_natural_key, typo in lookup
ltucker authored
805 return self.get(slug=slug, location_type__slug=location_type_slug)
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
806
dd75abd @slinkp whitespace
slinkp authored
807
5c9826f initial import
Don Kukral authored
808 class Location(models.Model):
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
809 '''
810 A polygon that represents a geographic area, such as a specific
811 neighborhood, ZIP code boundary or political boundary. Each ``Location`` has an
812 associated :py:class:`LocationType` (e.g., "neighborhood"). To add a Location to the
813 system, follow these steps:
814
815 1. Create a :py:class:`LocationType`.
816
817 2. Get the Location's geographic representation (a set of
818 longitude/latitude points that determine the border of the
819 polygon). You might want to draw this on your own using
820 desktop GIS tools or online tools, or you can try to get
821 the data from a company or government agency. (You can
822 even draw simple shapes in the OpenBlock admin UI.)
823
824 3. With the geographic representation, create a row in the
825 "db_location" table that describes the Location. See below
826 for what the various fields mean.
827
828 You can create Locations in various ways: use the admin UI;
829 use the script ``add_location`` to create one by
830 specifying its geometry in well-known text (WKT) format;
831 use the script ``import_locations`` to import them from shapefiles;
832 or use the Django model API; or do a manual SQL INSERT statement.
833 '''
834
835 name = models.CharField(max_length=255, help_text='e.g., "35th Ward"')
5c9826f initial import
Don Kukral authored
836 normalized_name = models.CharField(max_length=255, db_index=True)
e3056db @slinkp Use SlugField for slugs.
slinkp authored
837 slug = models.SlugField(max_length=32, db_index=True)
5c9826f initial import
Don Kukral authored
838 location_type = models.ForeignKey(LocationType)
839 location = models.GeometryField(null=True)
840 display_order = models.SmallIntegerField()
454f867 @slinkp Add a few more indexes; also comments
slinkp authored
841 city = models.CharField(max_length=255, db_index=True)
5c9826f initial import
Don Kukral authored
842 source = models.CharField(max_length=64)
38501f6 @slinkp Turn more comments into doocstrings and help text
slinkp authored
843 area = models.FloatField(
844 blank=True, null=True,
845 help_text="In square meters. This is populated automatically."
846 # the db trigger is created by ebpub/db/migrations/0004_st_intersects_patch.py.
847 )
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
848 population = models.IntegerField(blank=True, null=True,
849 help_text='Optional. If used, typicall found in census data.')
38501f6 @slinkp Turn more comments into doocstrings and help text
slinkp authored
850 user_id = models.IntegerField(
851 blank=True, null=True,
852 help_text="Used for 'custom' Locations created by end users.")
853 is_public = models.BooleanField(
854 help_text='Whether this is publically visible, or requires the staff cookie')
5c9826f initial import
Don Kukral authored
855 description = models.TextField(blank=True)
1d1815b @slinkp Sensible defaults on DateFields and DateTimeFields. Not using 'auto_n…
slinkp authored
856 creation_date = models.DateTimeField(blank=True, null=True,
857 default=datetime.datetime.now)
858 last_mod_date = models.DateTimeField(blank=True, null=True,
859 default=datetime.datetime.now)
e7f7764 @slinkp * Added eb_widgets templatetags with get_locations_for_item tag,
slinkp authored
860
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
861 objects = LocationManager()
5c9826f initial import
Don Kukral authored
862
1334338 @slinkp Remove Location.centroid, we can just use Location.location.centroid …
slinkp authored
863 @property
864 def centroid(self):
865 # For backward compatibility.
866 import warnings
867 warnings.warn(
868 "Location.centroid is deprecated. Use Location.location.centroid instead.",
869 DeprecationWarning)
870 return self.location.centroid
871
e3056db @slinkp Use SlugField for slugs.
slinkp authored
872 def clean(self):
e320c66 @slinkp Don't try to clean self.location if it's None
slinkp authored
873 if self.location:
874 try:
875 self.location = ensure_valid(flatten_geomcollection(self.location))
876 except ValueError, e:
877 raise ValidationError(str(e))
8466c9a @slinkp More normalization. Refs #157
slinkp authored
878 if self.normalized_name:
879 self.normalized_name = normalize(self.normalized_name)
880 else:
881 self.normalized_name = normalize(self.name)
a9a4d9f @slinkp ValueError -> ValidatonError to make admin form nicer
slinkp authored
882
5c9826f initial import
Don Kukral authored
883 class Meta:
884 unique_together = (('slug', 'location_type'),)
a11a8b0 @slinkp Lots more columns, filters, and usable default ordering in admin UI; …
slinkp authored
885 ordering = ('slug',)
5c9826f initial import
Don Kukral authored
886
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
887 def natural_key(self):
888 return (self.slug, self.location_type.slug)
889
5c9826f initial import
Don Kukral authored
890 def __unicode__(self):
891 return self.name
892
528587e @slinkp Added get_absolute_url() method to models where it makes sense, so ad…
slinkp authored
893 def get_absolute_url(self):
3939c85 @slinkp Decent-ish display for upcoming event-ish types. Refs #246.
slinkp authored
894 return urlresolvers.reverse('ebpub-location-recent',
2d83842 @slinkp De-hardcode more model URLs. Refs #69
slinkp authored
895 args=(self.location_type.slug, self.slug))
5c9826f initial import
Don Kukral authored
896
528587e @slinkp Added get_absolute_url() method to models where it makes sense, so ad…
slinkp authored
897 # Backward compatibility.
898 url = get_absolute_url
899
5c9826f initial import
Don Kukral authored
900 def rss_url(self):
2d83842 @slinkp De-hardcode more model URLs. Refs #69
slinkp authored
901 return urlresolvers.reverse('ebpub-location-rss',
902 args=(self.location_type.slug, self.slug))
5c9826f initial import
Don Kukral authored
903
904
2d83842 @slinkp De-hardcode more model URLs. Refs #69
slinkp authored
905 def alert_url(self):
906 return urlresolvers.reverse('ebpub-location-alerts',
907 args=(self.location_type.slug, self.slug))
5c9826f initial import
Don Kukral authored
908
909 # Give Location objects a "pretty_name" attribute for interoperability with
910 # Block objects. (Parts of our app accept either a Block or Location.)
764e1b9 @slinkp Use modern @property syntax; simplify SchemaField.is_type()
slinkp authored
911 @property
912 def pretty_name(self):
5c9826f initial import
Don Kukral authored
913 return self.name
914
764e1b9 @slinkp Use modern @property syntax; simplify SchemaField.is_type()
slinkp authored
915 @property
916 def is_custom(self):
5c9826f initial import
Don Kukral authored
917 return self.location_type.slug == 'custom'
764e1b9 @slinkp Use modern @property syntax; simplify SchemaField.is_type()
slinkp authored
918
5c9826f initial import
Don Kukral authored
919
220ca2c @ltucker Add some extra info to improve location and place tagging, split Miss…
ltucker authored
920 class LocationSynonymManager(models.Manager):
921 def get_canonical(self, name):
922 """
923 Returns the canonical normalized spelling of the given location name.
924 If the given location name is already correctly spelled, then it's returned as-is.
925 """
926 try:
927 normalized_name = normalize(name)
928 return self.get(normalized_name=normalized_name).location.normalized_name
929 except self.model.DoesNotExist:
930 return normalized_name
931
dd75abd @slinkp whitespace
slinkp authored
932
220ca2c @ltucker Add some extra info to improve location and place tagging, split Miss…
ltucker authored
933 class LocationSynonym(models.Model):
934 """
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
935 Represents an alternate name for a :py:class:`Location`.
220ca2c @ltucker Add some extra info to improve location and place tagging, split Miss…
ltucker authored
936 """
937 pretty_name = models.CharField(max_length=255)
938 normalized_name = models.CharField(max_length=255, db_index=True)
38501f6 @slinkp Turn more comments into doocstrings and help text
slinkp authored
939 location = models.ForeignKey(Location,
940 help_text='Location this is a synonym for.')
220ca2c @ltucker Add some extra info to improve location and place tagging, split Miss…
ltucker authored
941 objects = LocationSynonymManager()
942
035230a @slinkp Update model.save() signatures for compatibility w/ django 1.0, 1.2
slinkp authored
943 def save(self, force_insert=False, force_update=False, using=None):
9238a42 @slinkp Also normalize LocationSynonyms, and document all this stuff. Closes #80
slinkp authored
944 # Not doing this in clean() because we really don't want there to be
945 # any way to get this wrong.
946 if self.normalized_name:
947 self.normalized_name = normalize(self.normalized_name)
948 else:
220ca2c @ltucker Add some extra info to improve location and place tagging, split Miss…
ltucker authored
949 self.normalized_name = normalize(self.pretty_name)
035230a @slinkp Update model.save() signatures for compatibility w/ django 1.0, 1.2
slinkp authored
950 super(LocationSynonym, self).save(force_update=force_update, force_insert=force_insert, using=using)
220ca2c @ltucker Add some extra info to improve location and place tagging, split Miss…
ltucker authored
951
952 def __unicode__(self):
953 return self.pretty_name
954
dd75abd @slinkp whitespace
slinkp authored
955
5c9826f initial import
Don Kukral authored
956 class AttributesDescriptor(object):
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
957
958 # No docstring, not part of API.
959 #
960 # This class provides the functionality that makes the attributes available
961 # as a dictionary-like `attributes` on a model instance.
962 #
963 # You normally don't instantiate this directly.
964 # Just use newsitem.attributes like a normal dictionary.
965
5c9826f initial import
Don Kukral authored
966 def __get__(self, instance, instance_type=None):
967 if instance is None:
968 raise AttributeError("%s must be accessed via instance" % self.__class__.__name__)
969 if not hasattr(instance, '_attributes_cache'):
a726543 @ltucker be nicer when there are no schemafields
ltucker authored
970 select_dict = field_mapping([instance.schema_id]).get(instance.schema_id, {})
5c9826f initial import
Don Kukral authored
971 instance._attributes_cache = AttributeDict(instance.id, instance.schema_id, select_dict)
972 return instance._attributes_cache
973
974 def __set__(self, instance, value):
975 if instance is None:
976 raise AttributeError("%s must be accessed via instance" % self.__class__.__name__)
977 if not isinstance(value, dict):
978 raise ValueError('Only a dictionary is allowed')
78fac6b @slinkp Rename openblockapi.authorization to openblockapi.auth; also fix a Ke…
slinkp authored
979 mapping = field_mapping([instance.schema_id]).get(instance.schema_id, {}).items()
980 if not mapping:
981 if value:
982 logger.warn("Can't save non-empty attributes dict with an empty schema")
983 return
5c9826f initial import
Don Kukral authored
984 values = [value.get(k, None) for k, v in mapping]
985 cursor = connection.cursor()
986 cursor.execute("""
987 UPDATE %s
988 SET %s
989 WHERE news_item_id = %%s
990 """ % (Attribute._meta.db_table, ','.join(['%s=%%s' % v for k, v in mapping])),
991 values + [instance.id])
992 # If no records were updated, that means the DB doesn't yet have a
993 # row in the attributes table for this news item. Do an INSERT.
994 if cursor.rowcount < 1:
995 cursor.execute("""
996 INSERT INTO %s (news_item_id, schema_id, %s)
997 VALUES (%%s, %%s, %s)""" % (Attribute._meta.db_table, ','.join([v for k, v in mapping]), ','.join(['%s' for k in mapping])),
998 [instance.id, instance.schema_id] + values)
999 transaction.commit_unless_managed()
1000
dd75abd @slinkp whitespace
slinkp authored
1001
5c9826f initial import
Don Kukral authored
1002 class AttributeDict(dict):
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1003
1004 # No docstring, not part of API.
1005 #
1006 # A dictionary-like object that serves as a wrapper around attributes for a
1007 # given NewsItem.
1008 #
1009 # You normally don't instantiate this directly.
1010 # Just use news_item.attributes like a normal dictionary.
1011
5c9826f initial import
Don Kukral authored
1012 def __init__(self, news_item_id, schema_id, mapping):
1013 dict.__init__(self)
1014 self.news_item_id = news_item_id
1015 self.schema_id = schema_id
1016 self.mapping = mapping # name -> real_name dictionary
1017 self.cached = False
1018
1019 def __do_query(self):
1020 if not self.cached:
fb48fba @slinkp seeclickfix scraper now uses a new schema.
slinkp authored
1021 attr_values = Attribute.objects.filter(news_item__id=self.news_item_id).extra(select=self.mapping).values(*self.mapping.keys())
1022 # Rarely, we might have added the first SchemaField for this
1023 # Schema *after* the NewsItem was scraped. In that case
1024 # attr_values will be empty list.
1025 if attr_values:
1026 self.update(attr_values[0])
5c9826f initial import
Don Kukral authored
1027 self.cached = True
1028
ded7330 @slinkp Checking size of NewsItem.attributes is now reliable.
slinkp authored
1029 def __len__(self):
1030 # So len(self) and bool(self) work.
1031 self.__do_query()
1032 return dict.__len__(self)
1033
338de4b @ltucker output extension schema fields in items_json
ltucker authored
1034 def keys(self, *args, **kwargs):
1035 self.__do_query()
1036 return dict.keys(self, *args, **kwargs)
1037
1038 def items(self, *args, **kwargs):
1039 self.__do_query()
1040 return dict.items(self, *args, **kwargs)
1041
5c9826f initial import
Don Kukral authored
1042 def get(self, *args, **kwargs):
1043 self.__do_query()
1044 return dict.get(self, *args, **kwargs)
1045
1046 def __getitem__(self, name):
1047 self.__do_query()
1048 return dict.__getitem__(self, name)
1049
1050 def __setitem__(self, name, value):
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1051 # TODO: refactor, code overlaps largely with AttributesDescriptor.__set__
5c9826f initial import
Don Kukral authored
1052 cursor = connection.cursor()
1053 real_name = self.mapping[name]
1054 cursor.execute("""
1055 UPDATE %s
1056 SET %s = %%s
1057 WHERE news_item_id = %%s
1058 """ % (Attribute._meta.db_table, real_name), [value, self.news_item_id])
1059 # If no records were updated, that means the DB doesn't yet have a
1060 # row in the attributes table for this news item. Do an INSERT.
1061 if cursor.rowcount < 1:
1062 cursor.execute("""
1063 INSERT INTO %s (news_item_id, schema_id, %s)
1064 VALUES (%%s, %%s, %%s)""" % (Attribute._meta.db_table, real_name),
1065 [self.news_item_id, self.schema_id, value])
1066 transaction.commit_unless_managed()
1067 dict.__setitem__(self, name, value)
1068
dd75abd @slinkp whitespace
slinkp authored
1069
5c9826f initial import
Don Kukral authored
1070 class NewsItemQuerySet(models.query.GeoQuerySet):
b3fcffa @slinkp Fix #146: breakage was introduced in changeset a11a8b00 when we added…
slinkp authored
1071
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1072 """
1073 Adds special methods for searching :py:class:`NewsItem`.
1074 """
1075
5c9826f initial import
Don Kukral authored
1076 def prepare_attribute_qs(self):
1077 clone = self._clone()
1078 if 'db_attribute' not in clone.query.extra_tables:
3b0dc05 @slinkp Update stuff using Query.extra_where to use QuerySet.extra(where...).…
slinkp authored
1079 clone = clone.extra(tables=('db_attribute',))
1080 # extra_where went away in Django 1.1.
1081 # This seems to be the correct replacement as per
1082 # http://docs.djangoproject.com/en/dev/ref/models/querysets/
1083 clone = clone.extra(where=('db_newsitem.id = db_attribute.news_item_id',))
5c9826f initial import
Don Kukral authored
1084 return clone
1085
1086 def by_attribute(self, schema_field, att_value, is_lookup=False):
1087 """
1088 Returns a QuerySet of NewsItems whose attribute value for the given
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
1089 SchemaField is att_value.
1090
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1091 For example::
1092
1093 items = NewsItem.objects.filter(schema_id=1)
1094 sf = SchemaField.objects.get(name='violation', schema_id=1)
1095 items.by_attribute(sf, 'unsanitary work surface')
1096
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
1097 If att_value is a list, this will do the
5c9826f initial import
Don Kukral authored
1098 equivalent of an "OR" search, returning all NewsItems that have an
1099 attribute value in the att_value list.
1100
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
1101 Handles many-to-many lookups correctly behind the scenes.
5c9826f initial import
Don Kukral authored
1102
03d55a6 @slinkp db.Lookups: Better handling in NewsItemQuerySet.by_attribute()
slinkp authored
1103 If is_lookup is True, then each att_value must be either a
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1104 :py:class:`Lookup` instance, or the 'code' field of a Lookup instance, or an id
03d55a6 @slinkp db.Lookups: Better handling in NewsItemQuerySet.by_attribute()
slinkp authored
1105 of a Lookup instance. (If is_lookup is False, then only ids
1106 will work.)
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1107
1108 Does not support comparisons other than simple equality testing.
5c9826f initial import
Don Kukral authored
1109 """
a8bc11d @slinkp More hacking on import scripts: factoring into separate scripts, addi…
slinkp authored
1110
5c9826f initial import
Don Kukral authored
1111 clone = self.prepare_attribute_qs()
1112 real_name = str(schema_field.real_name)
03d55a6 @slinkp db.Lookups: Better handling in NewsItemQuerySet.by_attribute()
slinkp authored
1113 if isinstance(att_value, models.query.QuerySet):
1114 att_value = list(att_value)
5c9826f initial import
Don Kukral authored
1115 if not isinstance(att_value, (list, tuple)):
1116 att_value = [att_value]
1117 if is_lookup:
03d55a6 @slinkp db.Lookups: Better handling in NewsItemQuerySet.by_attribute()
slinkp authored
1118 if isinstance(att_value[0], int):
1119 # Assume all are Lookup.id values. Get just the ones
1120 # that exist.
1121 att_value = Lookup.objects.filter(schema_field__id=schema_field.id, id__in=att_value)
1122 elif not isinstance(att_value[0], Lookup):
1123 # Assume all are Lookup.code values. Get just the ones
1124 # that exist.
c0a594d @slinkp Replace SchemaField.name with SchemaField.slug, since it was already …
slinkp authored
1125 att_value = Lookup.objects.filter(schema_field__id=schema_field.id, code__in=att_value)
5c9826f initial import
Don Kukral authored
1126 if not att_value:
1127 # If the lookup values don't exist, then there aren't any
03d55a6 @slinkp db.Lookups: Better handling in NewsItemQuerySet.by_attribute()
slinkp authored
1128 # NewsItems with these attribute values. Note that we aren't
5c9826f initial import
Don Kukral authored
1129 # using QuerySet.none() here, because we want the result to
1130 # be a NewsItemQuerySet, and none() returns a normal QuerySet.
3b0dc05 @slinkp Update stuff using Query.extra_where to use QuerySet.extra(where...).…
slinkp authored
1131 clone = clone.extra(where=('1=0',))
5c9826f initial import
Don Kukral authored
1132 return clone
1133 att_value = [val.id for val in att_value]
1134 if schema_field.is_many_to_many_lookup():
1135 for value in att_value:
1136 if not str(value).isdigit():
454f867 @slinkp Add a few more indexes; also comments
slinkp authored
1137 raise ValueError('Only integer strings allowed for att_value in many-to-many SchemaFields; got %r' % value)
55d367d @slinkp Fix error in NewsItem.objects.by_attribute() with m2m lookups; looki…
slinkp authored
1138 # We have to use a regular expression search to look for
1139 # all rows with the given att_value *somewhere* in the
1140 # column. The [[:<:]] thing is a word boundary, and the
1141 # (?:) groups the possible values to distinguish them from
1142 # the word boundary part of the regex.
4867af6 @slinkp readability
slinkp authored
1143 pattern = '[[:<:]](?:%s)[[:>:]]' % '|'.join([str(val) for val in att_value])
1144 clone = clone.extra(where=("db_attribute.%s ~ '%s'" % (real_name, pattern),))
55d367d @slinkp Fix error in NewsItem.objects.by_attribute() with m2m lookups; looki…
slinkp authored
1145
5c9826f initial import
Don Kukral authored
1146 elif None in att_value:
1147 if att_value != [None]:
1148 raise ValueError('by_attribute() att_value list cannot have more than one element if it includes None')
3b0dc05 @slinkp Update stuff using Query.extra_where to use QuerySet.extra(where...).…
slinkp authored
1149 clone = clone.extra(where=("db_attribute.%s IS NULL" % real_name,))
5c9826f initial import
Don Kukral authored
1150 else:
3b0dc05 @slinkp Update stuff using Query.extra_where to use QuerySet.extra(where...).…
slinkp authored
1151 clone = clone.extra(where=("db_attribute.%s IN (%s)" % (real_name, ','.join(['%s' for val in att_value])),),
1152 params=tuple(att_value))
5c9826f initial import
Don Kukral authored
1153 return clone
1154
1155 def date_counts(self):
1156 """
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1157 Returns a dictionary mapping {item_date: count}, i.e. the number of
1158 :py:class:`NewsItem` created each day.
5c9826f initial import
Don Kukral authored
1159 """
1160 from django.db.models.query import QuerySet
1161 qs = QuerySet.values(self, 'item_date').annotate(count=models.Count('id'))
b3afb72 @slinkp Fix date span on overview page ajax date charts; also, ten days looks…
slinkp authored
1162 # Turn off ordering, as that breaks Count; see https://docs.djangoproject.com/en/dev/topics/db/aggregation/#interaction-with-default-ordering-or-order-by
1163 qs = qs.order_by()
5c9826f initial import
Don Kukral authored
1164 return dict([(v['item_date'], v['count']) for v in qs])
1165
1166 def top_lookups(self, schema_field, count):
1167 """
1168 Returns a list of {lookup, count} dictionaries representing the top
1169 Lookups for this QuerySet.
1170 """
1171 real_name = "db_attribute." + str(schema_field.real_name)
1172 if schema_field.is_many_to_many_lookup():
b3fcffa @slinkp Fix #146: breakage was introduced in changeset a11a8b00 when we added…
slinkp authored
1173 # First prepare a subquery to get a *single* count of
1174 # attribute rows that match each relevant m2m lookup
1175 # value. It's very important to get a single row here or
1176 # else we get a DataBaseError with "more than one row
1177 # returned by a subquery used as an expression". (Bug #146)
1178 clone = self.prepare_attribute_qs()
1179 clone = clone.filter(schema__id=schema_field.schema_id)
1180 # This is a regex search for the lookup id.
5c9826f initial import
Don Kukral authored
1181 clone = clone.extra(where=[real_name + " ~ ('[[:<:]]' || db_lookup.id || '[[:>:]]')"])
1182 # We want to count the current queryset and get a single
1183 # row for injecting into the subsequent Lookup query, but
1184 # we don't want Django's aggregation support to
1185 # automatically group by fields that aren't relevant and
1186 # would cause multiple rows as a result. So we call
1187 # `values()' on a field that we're already filtering by,
1188 # in this case, schema, as essentially a harmless identify
1189 # function.
b3fcffa @slinkp Fix #146: breakage was introduced in changeset a11a8b00 when we added…
slinkp authored
1190 # See http://docs.djangoproject.com/en/dev/topics/db/aggregation/#values
1191 clone = clone.values('schema')
1192
1193 # Fix #146: Having any `ORDER BY foo` in this subquery causes
1194 # Django to also add a `GROUP BY foo`, which potentially
1195 # returns multiple rows. So, remove the ordering.
1196 clone = clone.order_by()
1197 clone = clone.annotate(count=Count('schema'))
1198 # Unusual: We don't run the clone query, we just stuff its SQL
1199 # into our Lookup qs.
5c9826f initial import
Don Kukral authored
1200 qs = Lookup.objects.filter(schema_field__id=schema_field.id)
1201 qs = qs.extra(select={'lookup_id': 'id', 'item_count': clone.values('count').query})
1202 else:
b3fcffa @slinkp Fix #146: breakage was introduced in changeset a11a8b00 when we added…
slinkp authored
1203 # Counts of attribute rows matching each relevant Lookup.
1204 # Much easier when is_many_to_many_lookup == False :-)
5c9826f initial import
Don Kukral authored
1205 qs = self.prepare_attribute_qs().extra(select={'lookup_id': real_name})
1206 qs.query.group_by = [real_name]
1207 qs = qs.values('lookup_id').annotate(item_count=Count('id'))
b3fcffa @slinkp Fix #146: breakage was introduced in changeset a11a8b00 when we added…
slinkp authored
1208
1209 qs = qs.values('lookup_id', 'item_count').order_by('-item_count')
1210 ids_and_counts = [(v['lookup_id'], v['item_count']) for v in qs
1211 if v['item_count']]
1212 ids_and_counts = ids_and_counts[:count]
5c9826f initial import
Don Kukral authored
1213 lookup_objs = Lookup.objects.in_bulk([i[0] for i in ids_and_counts])
b3fcffa @slinkp Fix #146: breakage was introduced in changeset a11a8b00 when we added…
slinkp authored
1214 return [{'lookup': lookup_objs[i[0]], 'count': i[1]} for i in ids_and_counts
1215 if not None in i]
5c9826f initial import
Don Kukral authored
1216
1217 def text_search(self, schema_field, query):
1218 """
1219 Returns a QuerySet of NewsItems whose attribute for
1220 a given schema field matches a text search query.
1221 """
1222 clone = self.prepare_attribute_qs()
1223 query = query.lower()
3b0dc05 @slinkp Update stuff using Query.extra_where to use QuerySet.extra(where...).…
slinkp authored
1224
1225 clone = clone.extra(where=("db_attribute." + str(schema_field.real_name) + " ILIKE %s",),
1226 params=("%%%s%%" % query,))
5c9826f initial import
Don Kukral authored
1227 return clone
1228
94d3182 @slinkp More convenient by_request() API for restricting NewsItems to only sc…
slinkp authored
1229 def by_request(self, request):
1230 """
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1231 Returns a QuerySet that does additional request-specific
1232 filtering; currently this just uses
1233 get_schema_manager(request) to limit the schemas that are
94d3182 @slinkp More convenient by_request() API for restricting NewsItems to only sc…
slinkp authored
1234 visible during this request.
1235 """
1236 clone = self._clone()
1237 from ebpub.utils.view_utils import get_schema_manager
1238 allowed_schema_ids = get_schema_manager(request).allowed_schema_ids()
1239 return clone.filter(schema__id__in=allowed_schema_ids)
1240
1241
5c9826f initial import
Don Kukral authored
1242 class NewsItemManager(models.GeoManager):
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1243 """
1244 Available as :py:class:`NewsItem`.objects
1245 """
5c9826f initial import
Don Kukral authored
1246 def get_query_set(self):
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1247 """
1248 Returns a :py:class:`NewsItemQuerySet`
1249 """
5c9826f initial import
Don Kukral authored
1250 return NewsItemQuerySet(self.model)
1251
1252 def by_attribute(self, *args, **kwargs):
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1253 """
1254 See :py:meth:`NewsItemQuerySet.by_attribute`
1255 """
5c9826f initial import
Don Kukral authored
1256 return self.get_query_set().by_attribute(*args, **kwargs)
1257
1258 def text_search(self, *args, **kwargs):
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1259 """
1260 See :py:meth:`NewsItemQuerySet.text_search`
1261 """
5c9826f initial import
Don Kukral authored
1262 return self.get_query_set().text_search(*args, **kwargs)
1263
1264 def date_counts(self, *args, **kwargs):
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1265 """
1266 See :py:meth:`NewsItemQuerySet.date_counts`
1267 """
5c9826f initial import
Don Kukral authored
1268 return self.get_query_set().date_counts(*args, **kwargs)
1269
1270 def top_lookups(self, *args, **kwargs):
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1271 """
1272 See :py:meth:`NewsItemQuerySet.top_lookups`
1273 """
5c9826f initial import
Don Kukral authored
1274 return self.get_query_set().top_lookups(*args, **kwargs)
1275
94d3182 @slinkp More convenient by_request() API for restricting NewsItems to only sc…
slinkp authored
1276 def by_request(self, request):
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1277 """
1278 See :py:meth:`NewsItemQuerySet.by_request`
1279 """
94d3182 @slinkp More convenient by_request() API for restricting NewsItems to only sc…
slinkp authored
1280 return self.get_query_set().by_request(request)
1281
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1282
5c9826f initial import
Don Kukral authored
1283 class NewsItem(models.Model):
188c1d2 @slinkp Much commenting on how NewsItems work
slinkp authored
1284 """
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
1285 A NewsItem is broadly defined as "something with a date and a
1286 location." For example, it could be a local news article, a
1287 building permit, a crime report, or a photo.
1288
1289 For the big picture, see :ref:`newsitems`
1290
efc28d9 @slinkp * Add a foreign key between NewsItemImage and NewsItem.
slinkp authored
1291
188c1d2 @slinkp Much commenting on how NewsItems work
slinkp authored
1292 """
1293
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
1294 # We don't have a natural_key() method because we don't know for
1295 # sure that anything other than ID will be unique.
1296
19d0783 @slinkp Package docs: Big reorganization to move much of the ebpub docs into …
slinkp authored
1297 schema = models.ForeignKey(Schema, help_text=u'What kind of news is this and what extra fields does it have?')
1298 title = models.CharField(max_length=255, help_text=u'the "headline"')
f1a6469 @slinkp Allow empty description on NewsItems; some things have really nothing…
slinkp authored
1299 description = models.TextField(blank=True, default=u'')
4e12c6e @slinkp Add more help text to NewsItem fields... spread the knowledge
slinkp authored
1300 url = models.TextField(
08104bd @slinkp Allow empty description on NewsItems ... now with default.
slinkp authored
1301 blank=True, default=u'',
4e12c6e @slinkp Add more help text to NewsItem fields... spread the knowledge
slinkp authored
1302 help_text="link to original source for this news")
c59165d @slinkp Add a bit of help text to NewsItem models.
slinkp authored
1303 pub_date = models.DateTimeField(
1304 db_index=True,
21b06da @slinkp Better docs on spreadsheet loading, newsitem core fields
slinkp authored
1305 help_text='Date/time this Item was added to the OpenBlock site; default now.',
94c5500 @slinkp No reason to require fields that have sensible defaults.
slinkp authored
1306 default=datetime.datetime.now,
1307 blank=True,
1d1815b @slinkp Sensible defaults on DateFields and DateTimeFields. Not using 'auto_n…
slinkp authored
1308 )
c59165d @slinkp Add a bit of help text to NewsItem models.
slinkp authored
1309 item_date = models.DateField(
1310 db_index=True,
21b06da @slinkp Better docs on spreadsheet loading, newsitem core fields
slinkp authored
1311 help_text='Date (without time) this Item occurred, or failing that, the date of publication on the original source site; default today.',
94c5500 @slinkp No reason to require fields that have sensible defaults.
slinkp authored
1312 default=datetime.date.today,
1313 blank=True,
1d1815b @slinkp Sensible defaults on DateFields and DateTimeFields. Not using 'auto_n…
slinkp authored
1314 )
529a3ab @ltucker Add automatic last modification date to all newsitems
ltucker authored
1315
e7f7764 @slinkp * Added eb_widgets templatetags with get_locations_for_item tag,
slinkp authored
1316 # Automatic last modification tracking. Note: if changing only attributes, the
1317 # NewsItem should also be save()'d to update last_modification when complete.
529a3ab @ltucker Add automatic last modification date to all newsitems
ltucker authored
1318 last_modification = models.DateTimeField(db_index=True, auto_now=True)
1d1815b @slinkp Sensible defaults on DateFields and DateTimeFields. Not using 'auto_n…
slinkp authored
1319
4e12c6e @slinkp Add more help text to NewsItem fields... spread the knowledge
slinkp authored
1320 location = models.GeometryField(blank=True, null=True, spatial_index=True,
1321 help_text="Coordinates where this news occurred.")
1322 location_name = models.CharField(max_length=255,
1323 help_text="Human-readable address or name of place where this news item occurred.")
1324 location_object = models.ForeignKey(Location, blank=True, null=True,
e7f7764 @slinkp * Added eb_widgets templatetags with get_locations_for_item tag,
slinkp authored
1325 help_text="Optional reference to a Location where this item occurred, for use when we know the general area but not specific coordinates.",
1326 related_name='+')
1327
1328 location_set = models.ManyToManyField(
1329 Location, through='NewsItemLocation', blank=True, null=True,
1330 help_text="db.Location objects that intersect with our .location geometry. These are set automatically, do not try to assign to them.")
1331
5c9826f initial import
Don Kukral authored
1332 objects = NewsItemManager()
1333
953fef3 @slinkp More Attributes documentation
slinkp authored
1334 # Treat this like a dict. The related Schema and SchemaFields explain
1335 # what keys/ types of values you can set.
1336 # See the ``ebpub`` section of the docs for more information.
1337 # Note you do NOT need to save() the NewsItem after setting or modifying
1338 # this dictionary - but you do need to save() before the FIRST time you do so,
1339 # because the underlying Attribute instance needs a reference to the NewsItem's
1340 # primary key.
1341 attributes = AttributesDescriptor()
5309287 @slinkp Overhaul of olwidget integration.
slinkp authored
1342
1343 def clean(self):
07ab686 @slinkp Don't blow up if a NewsItem.location is null...that's allowed.
slinkp authored
1344 if self.location is None:
1345 if self.location_object is None:
1346 logger.warn(
1347 "Saving NewsItem with neither a location nor a location_object")
1348 else:
cc48c38 @slinkp Remove brain-dead checking the same condition twice
slinkp authored
1349 self.location = ensure_valid(flatten_geomcollection(self.location))
5309287 @slinkp Overhaul of olwidget integration.
slinkp authored
1350
a11a8b0 @slinkp Lots more columns, filters, and usable default ordering in admin UI; …
slinkp authored
1351 class Meta:
1352 ordering = ('title',)
1353
5c9826f initial import
Don Kukral authored
1354 def __unicode__(self):
92b5f2b @ltucker display "Untitled News Item" if item has no title, fixes non-linked i…
ltucker authored
1355 return self.title or 'Untitled News Item'
5c9826f initial import
Don Kukral authored
1356
528587e @slinkp Added get_absolute_url() method to models where it makes sense, so ad…
slinkp authored
1357 def get_absolute_url(self):
2d83842 @slinkp De-hardcode more model URLs. Refs #69
slinkp authored
1358 return urlresolvers.reverse('ebpub-newsitem-detail',
1359 args=[self.schema.slug, self.id], kwargs={})
5c9826f initial import
Don Kukral authored
1360
528587e @slinkp Added get_absolute_url() method to models where it makes sense, so ad…
slinkp authored
1361 # Backward compatibility.
1362 item_url = get_absolute_url
1363
5c9826f initial import
Don Kukral authored
1364 def item_url_with_domain(self):
2aa82d7 @slinkp on second thought, EB_DOMAIN and EB_FULL_DOMAIN have no reason to be …
slinkp authored
1365 return 'http://%s%s' % (settings.EB_DOMAIN, self.item_url())
5c9826f initial import
Don Kukral authored
1366
1367 def item_date_url(self):
5d323be @slinkp Remove one more ad-hoc URL generator. Refs #69
slinkp authored
1368 from ebpub.db.schemafilters import FilterChain
1369 chain = FilterChain(schema=self.schema)
1370 chain.add('date', self.item_date)
1371 return chain.make_url()
5c9826f initial import
Don Kukral authored
1372
1373 def location_url(self):
1374 if self.location_object_id is not None:
1375 return self.location_object.url()
37f0a78 @slinkp Add checkboxes to newsitem lists that allow linking to a rich map wit…
slinkp authored
1376 # TODO: look for a Block?
5c9826f initial import
Don Kukral authored
1377 return None
1378
1379 def attributes_for_template(self):
1380 """
1381 Return a list of AttributeForTemplate objects for this NewsItem. The
1382 objects are ordered by SchemaField.display_order.
1383 """
1384 fields = SchemaField.objects.filter(schema__id=self.schema_id).select_related().order_by('display_order')
3403be1 added check to validate fields exists
Everyblock User authored
1385 if not fields:
1386 return []
f65f953 @slinkp Fix two errors triggered when NewsItems are saved for a schema with n…
slinkp authored
1387 if not self.attributes:
1388 logger.warn("%s has fields in its schema, but no attributes!" % self)
76c50e7 @slinkp Work around issue where some NeighborNews examples didn't have an att…
slinkp authored
1389 # Hopefully we can cope with an empty dict.
1390 #return []
a7b3132 @slinkp Add tests for populate_attributes_if_needed. And fix #72. Mostly sa…
slinkp authored
1391 return [AttributeForTemplate(f, self.attributes) for f in fields]
5c9826f initial import
Don Kukral authored
1392
de7c1fe @slinkp Yet more de-hardcoded URLs.
slinkp authored
1393
5c9826f initial import
Don Kukral authored
1394 class AttributeForTemplate(object):
684c02e @slinkp Remove unneeded SchemaFieldInfo model. Refs #50. help_text wasn't eve…
slinkp authored
1395 def __init__(self, schema_field, attribute_row):
5c9826f initial import
Don Kukral authored
1396 self.sf = schema_field
b27f097 @slinkp More workarounds for missing Attribute rows.
slinkp authored
1397 if not schema_field.name in attribute_row:
1398 logger.warn("Attribute row %s is missing field %s" %
1399 (attribute_row, schema_field.name))
1400 self.raw_value = attribute_row.get(schema_field.name)
5c9826f initial import
Don Kukral authored
1401 self.schema_slug = schema_field.schema.slug
1402 self.is_lookup = schema_field.is_lookup
1403 self.is_filter = schema_field.is_filter
1404 if self.is_lookup:
a7b3132 @slinkp Add tests for populate_attributes_if_needed. And fix #72. Mostly sa…
slinkp authored
1405 # Earlier queries may have already looked up Lookup instances.
1406 # Don't do unnecessary work.
1407 if isinstance(self.raw_value, Lookup):
1408 self.values = [self.raw_value]
1409 elif (isinstance(self.raw_value, list) and self.raw_value
1410 and isinstance(self.raw_value[0], Lookup)):
1411 self.values = self.raw_values
64c6966 @ltucker fix bug in lookup resolution, check for None in addition to ''
ltucker authored
1412 elif self.raw_value is None or self.raw_value == '':
5c9826f initial import
Don Kukral authored
1413 self.values = []
1414 elif self.sf.is_many_to_many_lookup():
1415 try:
1416 id_values = map(int, self.raw_value.split(','))
1417 except ValueError:
1418 self.values = []
1419 else:
1420 lookups = Lookup.objects.in_bulk(id_values)
c0a14b4 @slinkp Fix KeyError when an Attribute references a non-existent Lookup.
slinkp authored
1421 self.values = [lookups[i] for i in id_values if i in lookups]
5c9826f initial import
Don Kukral authored
1422 else:
1423 self.values = [Lookup.objects.get(id=self.raw_value)]
1424 else:
1425 self.values = [self.raw_value]
1426
1427 def value_list(self):
1428 """
9b9be15 @slinkp more comments on non-obvious code
slinkp authored
1429 Returns a list of {value, url, description} dictionaries
1430 representing each value for this attribute.
5c9826f initial import
Don Kukral authored
1431 """
1432 from django.utils.dateformat import format, time_format
9b9be15 @slinkp more comments on non-obvious code
slinkp authored
1433 # Setting these to [None] ensures that zip() returns a list
1434 # of at least length one.
5c9826f initial import
Don Kukral authored
1435 urls = [None]
1436 descriptions = [None]
1437 if self.is_filter:
6a39ff1 @slinkp * Rename SchemaFilterChain to FilterChain - schema is optional now.
slinkp authored
1438 from ebpub.db.schemafilters import FilterChain
1439 chain = FilterChain(schema=self.sf.schema)
5c9826f initial import
Don Kukral authored
1440 if self.is_lookup:
de7c1fe @slinkp Yet more de-hardcoded URLs.
slinkp authored
1441 urls = [chain.replace(self.sf, look).make_url() if look else None
1442 for look in self.values]
1443 else:
1444 urls = [chain.replace(self.sf, self.raw_value).make_url()]
5c9826f initial import
Don Kukral authored
1445 if self.is_lookup:
1446 values = [val and val.name or 'None' for val in self.values]
1447 descriptions = [val and val.description or None for val in self.values]
1448 elif isinstance(self.raw_value, datetime.datetime):
1449 values = [format(self.raw_value, 'F j, Y, P')]
1450 elif isinstance(self.raw_value, datetime.date):
1451 values = [format(self.raw_value, 'F j, Y')]
1452 elif isinstance(self.raw_value, datetime.time):
1453 values = [time_format(self.raw_value, 'P')]
1454 elif self.raw_value is True:
1455 values = ['Yes']
1456 elif self.raw_value is False:
1457 values = ['No']
1458 elif self.raw_value is None:
1459 values = ['N/A']
1460 else:
1461 values = [self.raw_value]
1462 return [{'value': value, 'url': url, 'description': description} for value, url, description in zip(values, urls, descriptions)]
1463
dd75abd @slinkp whitespace
slinkp authored
1464
5c9826f initial import
Don Kukral authored
1465 class Attribute(models.Model):
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1466
188c1d2 @slinkp Much commenting on how NewsItems work
slinkp authored
1467 """
1468 Extended metadata for NewsItems.
1469
1470 Each row contains all the extra metadata for one NewsItem
1471 instance. The field names are generic, so in order to know what
1472 they mean, you must look at the SchemaFields for the Schema for
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1473 that NewsItem.
1474
1475 You don't normally access an Attribute instance directly.
1476 You usually go through the dictionary-like API provided by
1477 :ref:`newsitem_attributes`.
188c1d2 @slinkp Much commenting on how NewsItems work
slinkp authored
1478
1479 """
476e10c @slinkp Changing Attributes.newsitem from ForeignKey to OneToOneField. Not cl…
slinkp authored
1480 news_item = models.OneToOneField(NewsItem, primary_key=True, unique=True)
5c9826f initial import
Don Kukral authored
1481 schema = models.ForeignKey(Schema)
1482 # All data-type field names must end in two digits, because the code assumes this.
e229d7a @ltucker add a bit more wiggle room in the attributes for larger fields since …
ltucker authored
1483 varchar01 = models.CharField(max_length=4096, blank=True, null=True)
1484 varchar02 = models.CharField(max_length=4096, blank=True, null=True)
1485 varchar03 = models.CharField(max_length=4096, blank=True, null=True)
1486 varchar04 = models.CharField(max_length=4096, blank=True, null=True)
1487 varchar05 = models.CharField(max_length=4096, blank=True, null=True)
5c9826f initial import
Don Kukral authored
1488 date01 = models.DateField(blank=True, null=True)
1489 date02 = models.DateField(blank=True, null=True)
1490 date03 = models.DateField(blank=True, null=True)
1491 date04 = models.DateField(blank=True, null=True)
1492 date05 = models.DateField(blank=True, null=True)
1493 time01 = models.TimeField(blank=True, null=True)
1494 time02 = models.TimeField(blank=True, null=True)
1495 datetime01 = models.DateTimeField(blank=True, null=True)
1496 datetime02 = models.DateTimeField(blank=True, null=True)
1497 datetime03 = models.DateTimeField(blank=True, null=True)
1498 datetime04 = models.DateTimeField(blank=True, null=True)
1499 bool01 = models.NullBooleanField(blank=True)
1500 bool02 = models.NullBooleanField(blank=True)
1501 bool03 = models.NullBooleanField(blank=True)
1502 bool04 = models.NullBooleanField(blank=True)
1503 bool05 = models.NullBooleanField(blank=True)
1504 int01 = models.IntegerField(blank=True, null=True)
1505 int02 = models.IntegerField(blank=True, null=True)
1506 int03 = models.IntegerField(blank=True, null=True)
1507 int04 = models.IntegerField(blank=True, null=True)
1508 int05 = models.IntegerField(blank=True, null=True)
1509 int06 = models.IntegerField(blank=True, null=True)
1510 int07 = models.IntegerField(blank=True, null=True)
1511 text01 = models.TextField(blank=True, null=True)
e229d7a @ltucker add a bit more wiggle room in the attributes for larger fields since …
ltucker authored
1512 text02 = models.TextField(blank=True, null=True)
5c9826f initial import
Don Kukral authored
1513
1514 def __unicode__(self):
1515 return u'Attributes for news item %s' % self.news_item_id
1516
dd75abd @slinkp whitespace
slinkp authored
1517
5c9826f initial import
Don Kukral authored
1518 class LookupManager(models.Manager):
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
1519
b7ed000 @slinkp Fix natural keys on Lookup model
slinkp authored
1520 def get_by_natural_key(self, slug, schema__slug,
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
1521 schema_field__real_name):
b7ed000 @slinkp Fix natural keys on Lookup model
slinkp authored
1522 return self.get(slug=slug, schema_field__schema__slug=schema__slug,
baec0a7 @slinkp Use natural keys for serialization wherever possible; it makes fixtur…
slinkp authored
1523 schema_field__real_name=schema_field__real_name)
1524
5c9826f initial import
Don Kukral authored
1525 def get_or_create_lookup(self, schema_field, name, code=None, description='', make_text_slug=True, logger=None):
1526 """
1527 Returns the Lookup instance matching the given SchemaField, name and
1528 Lookup.code, creating it (with the given name/code/description) if it
1529 doesn't already exist.
1530
454f867 @slinkp Add a few more indexes; also comments
slinkp authored
1531 If ``code`` is not provided, ``name`` will be also used as code.
1532
1533 If ``make_text_slug`` is True (the default), then a slug will
1534 be created from the given name. If it's False, then the slug
1535 will be the Lookup's ID.
5c9826f initial import
Don Kukral authored
1536 """
1537 def log_info(message):
1538 if logger is None:
1539 return
1540 logger.info(message)
1541 def log_warn(message):
1542 if logger is None:
1543 return
1544 logger.warn(message)
1545 code = code or name # code defaults to name if it wasn't provided
1546 try:
1547 obj = Lookup.objects.get(schema_field__id=schema_field.id, code=code)
1548 except Lookup.DoesNotExist:
1549 if make_text_slug:
1550 slug = slugify(name)
1551 if len(slug) > 32:
4ababb0 @slinkp redundant conditional
slinkp authored
1552 log_warn("Trimming slug %r to %r in order to fit 32-char limit." % (slug, slug[:32]))
5c9826f initial import
Don Kukral authored
1553 slug = slug[:32]
1554 else:
1555 # To avoid integrity errors in the slug when creating the Lookup,
1556 # use a temporary dummy slug that's guaranteed not to be in use.
1557 # We'll change it back immediately afterward.
1558 slug = '__3029j3f029jf029jf029__'
1559 if len(name) > 255:
1560 old_name = name
1561 name = name[:250] + '...'
1562 # Save the full name in the description.
1563 if not description:
1564 description = old_name
1565 log_warn("Trimming name %r to %r in order to fit 255-char limit." % (old_name, name))
80bc249 @slinkp Remove neighbornews.FeaturedLookup, replace with a boolean on db.Lookup.
slinkp authored
1566 if Lookup.objects.filter(schema_field=schema_field, name=name).count():
1567 # Avoid integrity errors on 'name'.
1568 old_name = name
1569 name = name + ' b'
1570 log_warn("Munging name %r to %r in order to avoid dupe." % (old_name, name))
1571
5c9826f initial import
Don Kukral authored
1572 obj = Lookup(schema_field_id=schema_field.id, name=name, code=code, slug=slug, description=description)
1573 obj.save()
1574 if not make_text_slug:
1575 # Set the slug to the ID.
1576 obj.slug = obj.id
1577 obj.save()
1578 log_info('Created %s %r' % (schema_field.name, name))
1579 return obj
1580
80bc249 @slinkp Remove neighbornews.FeaturedLookup, replace with a boolean on db.Lookup.
slinkp authored
1581 def featured_lookups_for(self, newsitem, attribute_key):
1582 """
3832f61 @slinkp Enhanced API docs for ebpub.db.models; remove docs for some stuff tha…
slinkp authored
1583