Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Overview
--------
The point of this commit is to switch from
`django_hstore.fields.DictionaryField` to
`django.contrib.postgres.fields.HStoreField`.
The former is incompatible with later releases of django,
and OpenTreeMap needs to migrate from the current 1.8 to 1.11
for stability.
The latter is the sanctioned django interface to PostgreSQL's hstore
going forward.
The latter is also less functional than the former, prompting a series
of other changes.
Changes were also made for the sake of maintenance simplicity.
The key changes here are in `treemap.udf`, `treemap.audit`, `*.models`, and
`treemap.search`.
**treemap.udf**
Entry points:
- `UDFModel`, a base class for models that need user defined fields
- `UserDefinedFieldDefinition`, defines the name, data type, and model type
for a user defined field, for a given treemap instance, as a
scalar or a collection
- `UserDefinedCollectionValue`, an HStore collection value for a
specific `UserDefinedFieldDefinition` on a specific model instance
`UDFModel` user guide:
Classes that want custom fields subclass `UDFModel`.
The `UDFModel` `HStoreField` is named `hstore_udfs`, but unless you are
getting deep into udf code, you should use `udfs`, which is an ordinary
attribute on the model instance, not directly associated with any
django `Field`.
To assign a custom field,
`my_model_instance.udfs['custom field name'] = value`
There are three ways to retrieve it:
`my_model_instance.udfs['custom field name']`
`getattr(my_model_instance, 'custom field name')`
`getattr(my_model_instance, 'udf:custom field name'`)
All of the above work for both scalar and collection custom fields.
Filter using the `hstore_udf` transform.
In addition to the transforms and lookups described at
https://docs.djangoproject.com/en/1.8/ref/contrib/postgres/fields/,
`UDFModel` implements `__int` and `__float` transforms, for use
before magnitude comparisons (`__gt`, `__gte`, `__lt`, `__lte`).
`udfs` perpetually syncs to `hstore_udfs`.
Values in `udfs` are always typed for the app, and
values in `hstore_udfs` are always typed for the db.
Since they are perpetually clean, code that assigns to `udfs`
needs to be prepared for a `ValidationError` at the time of the
assignment, not just at `save`.
In practice, this is not a problem, because the web frontend
takes care to send valid data in PUTs and POSTs.
**treemap.audit and \*.models**
- Models can use a plain `GeoManager` now
- In order to reduce code coupling and assure that initialization happens in a
rational order, `populate_previous_state` is now the responsibility of the
leaf subclass of `UserTrackable`.
- Species is no longer a `UDFModel` because it never employed udfs,
and removing unused code makes maintenance simpler.
Before this change, `populate_previous_state` was getting called between
initialization of different levels of model class. It calls `as_dict`, which
raises exceptions if not all fields have been initialized,
so now `populate_previous_state` is the responsibility of the leaf subclass.
This reduces the coupling between auditing and custom fields.
That tight coupling was why previous versions of `udfs.py` had to
inject a descriptor into the django model instantiation procedure.
Descriptors are fine as long as they work, but when they don't, they
are a headache to debug. Better to reduce the amount of injection
into django's procedure.
**treemap.search**
- Use the new `hstore_udf` filter transform
- Additional parser steps to add the udf `__float` filter transform
Specific Changes
----------------
- Replace UDF specific GeoHStoreUDFManager with generic
GeoManager in treemap and stormwater models
- Add django.contrib.postgres to settings
- Migration to give the `udfs` column a default
- Migration to demote Species from UDFModel
- `treemap.udf`
- Eliminate
- UDF specific manager and descriptor
- `quotesingle`
- `UDF_ORDER_PATTERN`
- `UDFBaseContains(Lookup)`
- `UDFContains` and `UDFIcontains`
- `UDFQuery(Query)`
- `UDFQuerySet(HStoreGeoQuerySet)`
- `GeoHStoreUDFManager(HStoreGeoManager)`
- unused `'user'` datatype
- Alias `UDFField` to `HStoreField` for the sake of earlier migrations
- Rename the `HStoreField` to `hstore_udfs` with a column param
to point it at the `udfs` column in `treemap_mapfeature` table
- Dynamically make a `udfs` model instance attribute in `UDFModel.__init__`
- Migration to support the above
- The auto-generated migration used `RemoveField` and `AddField`,
which lost all the data so the migration here has been hacked to use
`RenameField` and `AlterField`, and tested for data retention.
- `UserDefinedCollectionValue.data` now based on
`django.contrib.postgres.fields.HStoreField`
- If a collection field choice is removed,
delete all UDCVs that name that choice,
not just the choice field in the UDCV
- Use `feature_type` attribute if it exists, because the only classname
available might be the generic `MapFeature`.
- Have a stackoverflow reference in case ordering
by an hstore key ever becomes important
- Perhaps it's a design flaw that `udfs` perpetually cleans and
`hstore_udfs` perpetually reverse cleans, and both should be
delayed to `save_with_user`, but there are disadvantages to
allowing dirty values, too.
- Transforms based on
`django.contrib.postgres.fields.hstore.KeyTransform`,
which is registered on `HStoreField`
- Search now works for both collection and scalar UDFs
- `treemap.search`
- Removed 'IN' and 'ISNULL' predicates from collection search
because they are never used in production
- Removed 'WITHIN_RADIUS' predicate from scalar search
because it is never used in production
- Removed tests that no longer apply, or that test code that
is never used in production
- Modified integer and float udf tests so that they fail if the emitted
SQL does a lexical comparison to demonstrate the problem
- Add a `'multichoice'` test to `test_udfs`
- Fix form_extras to tolerate a missing UDF key
- Test for `is None` wherever a falsey value could be legitimate
- Adapt `treemap.views.map_feature.update_map_feature` to include
`ValidationError` `message_dict`s from setting a udf to an invalid value
in its `errors`.
Testing Notes
-------------
- Combines well with other searches
- ... unless you are bitten by #3080
--
Connects to #1968- Loading branch information
1 parent
87b715d
commit b9ea43a