Update to Django 1.9 and store parameters as JSON object

.travis.yml

@@ -14,11 +14,11 @@ install:
   - popd
   - source activate aei_dropq
 
-before_script:
-  - python manage.py syncdb --noinput
-  - python manage.py makemigrations taxbrain
-  - python manage.py migrate taxbrain
-  - python manage.py migrate --fake
+addons:
+    postgresql: "9.4"
 
+before_script:
+  - psql -c 'create database test_db;' -U postgres
+
 script:
   - py.test webapp/apps/

PARAMETERS.md

@@ -0,0 +1,45 @@
+Parameter Policy
+====================
+
+Goal: document parameter handling approach
+
+Parameter Parsing
+-------------------
+
+Ideally, the string received from the user input is parsed into the the
+Python type it most closely matches. That means that `'True'` parses
+to `True`, `'1'` to `1`, `'20000.0'` to `20000.0`, and so on. It is then
+up to the upstream package to perform type and range validation on the
+converted object and provide the appropriate error messages.
+
+However, due to issues with the interface, the user's input has to be massaged
+into the correct format. For instance, places where we expect a boolean
+string display `1.0` as a default value instead of `True`. Thus, the value
+parsing logic must be capable of parsing `'1.0'`, `'1'`, or `'True'` to `True`.
+There are other issues such as values that should only be integers (e.g. age)
+are displayed as floating point values. Once we display more accurate
+default values, we should remove this functionality in order to simplify the
+codebase. Note that there will always be some element of string massaging
+since PolicyBrain users tend to be less technical and thus, may not realize
+the significance of entering `True` versus `true` or `TRUE`.
+
+
+Parameter Deprecation
+---------------------
+
+Our primary goal is to preserve the results and relevant information for all
+runs. The goal of allowing the user to edit and re-submit previous runs is a
+lower priority than the first goal. The underlying modeling packages change
+overtime which means that parameter names change, their underlying definitions
+change, they are split in to multiple components, and some are removed.
+PolicyBrain only allows the user to view and submit parameters that are
+supported by the current version of the underlying package. Thus, when a user
+tries to access a run submitted on an older version of the underlying package
+via the edit parameters page, some of the parameters may not exist on the page
+anymore. The PolicyBrain policy in this situation is to allow the user to view
+and edit the parameters that still exist and for the parameters that do not,
+display an error message at the top of the page for each missing parameter that
+says:
+
+    Field {parameter name} has been deprecated. Refer to the Tax-Caclulator
+    documentation for a sensible replacement.

RELEASES.md

@@ -4,6 +4,26 @@ Go
 [here](https://github.com/OpenSourcePolicyCenter/PolicyBrain/pulls?q=is%3Apr+is%3Aclosed)
 for a complete commit history.
 
+Release 1.4.0 on 2018-02-26
+----------------------------
+**Major Changes**
+- None
+
+**Minor Changes**
+- [#798](https://github.com/OpenSourcePolicyCenter/PolicyBrain/pull/798) - Add examples for get_default_policy_param_name - Hank Doupe
+- [#800](https://github.com/OpenSourcePolicyCenter/PolicyBrain/pull/800) - Refactor warning/error parsing - Hank Doupe
+- [#808](https://github.com/OpenSourcePolicyCenter/PolicyBrain/pull/808) - Upgrade to Tax-Calculator 0.15.0 - Hank Doupe
+- [#809](https://github.com/OpenSourcePolicyCenter/PolicyBrain/pull/809) - Add Legend to Bubble Plot - Anderson Frailey
+- [#810](https://github.com/OpenSourcePolicyCenter/PolicyBrain/pull/810) - Small changes to .gitignore and install script - Ben Bolte
+- [#822](https://github.com/OpenSourcePolicyCenter/PolicyBrain/pull/822) - Update to Django 1.9 and store parameters as JSON object - Hank Doupe
+
+**Bug Fixes**
+- [#797](https://github.com/OpenSourcePolicyCenter/PolicyBrain/pull/797) - Remove Expanded-Income/or/Adjusted-Income row buttons - Sean Wang
+- [#799](https://github.com/OpenSourcePolicyCenter/PolicyBrain/pull/799) - .travis.yml update to fix build errors - Hank Doupe
+- [#807](https://github.com/OpenSourcePolicyCenter/PolicyBrain/pull/807) - Fix "Vizualization" typo in Gallery - Max Ghenis
+- [#814](https://github.com/OpenSourcePolicyCenter/PolicyBrain/pull/814) - Resolve more backwards compatibility issues related to old taxcalc tables - Hank Doupe
+- [#818](https://github.com/OpenSourcePolicyCenter/PolicyBrain/pull/818) - Changed all instances of TJ Alumbaugh's email address - Ben Bolte
+
 Release 1.3.0 on 2017-12-23
 ----------------------------
 **Major Changes**

conda-requirements.txt

@@ -1,6 +1,6 @@
 nomkl
-taxcalc==0.15.1
-btax==0.1.9
+taxcalc==0.15.2
+btax==0.2.2
 numba>=0.33.0
 pandas>=0.22.0
 gevent

deploy/fab/reset_server.sh

@@ -78,9 +78,6 @@ redis-cli FLUSHALL
 cd ${DEP}/../Tax-Calculator && git fetch origin
 conda list | grep 'taxcalc' | awk -F' ' '{print $2}' | xargs -n 1 git checkout
 cp ~/deploy/puf.csv.gz ./ && gunzip -k puf.csv.gz
-cd taxcalc/tests
-echo $rs Test the correct puf is here - Tax-Calculator
-py.test -m "requires_pufcsv"
 cd $DEP/taxbrain_server/tests
 echo $rs Test the mock celery - mock flask tests in deploy
 MOCK_CELERY=1 TAX_ESTIMATE_PATH=$OGUSA_PATH py.test -p no:django -v

deploy/taxbrain_server/celery_tasks.py

@@ -156,7 +156,7 @@ def ogusa_async(user_mods, ogusa_params, guid):
 def btax_async(user_mods):
     print("user mods: ", user_mods)
     results = {}
-    tables = runner_json_tables(**user_mods)
+    tables = json.loads(runner_json_tables(**user_mods))
     if tables.get("json_table"):
         results.update(tables["json_table"])
         if tables.get("dataframes"):
@@ -171,7 +171,8 @@ def btax_async(user_mods):
     results['dropq_version'] = vinfo['version']
     binfo = btax._version.get_versions()
     results['btax_version'] = binfo['version']
-    return json.dumps(results)
+    json_res = json.dumps(results)
+    return json_res
 
 
 @celery_app.task

deploy/taxbrain_server/flask_server.py

@@ -102,8 +102,9 @@ def btax_endpoint():
     raw_results = btax_async.delay(user_mods)
     RUNNING_JOBS[raw_results.id] = raw_results
     length = client.llen(queue_name) + 1
-    results = {'job_id':str(raw_results), 'qlength':length}
-    return str(json.dumps(results))
+    results = {'job_id': str(raw_results), 'qlength':length}
+    json_res = json.dumps(results)
+    return json_res
 
 
 @app.route("/elastic_gdp_start_job", methods=['POST'])

requirements.txt

@@ -1,4 +1,4 @@
-Django==1.8.2
+Django==1.9
 argparse==1.2.1
 dj-database-url==0.3.0
 dj-static==0.0.6
@@ -11,12 +11,10 @@ sendgrid-django
 psycopg2>=2.5.4
 django-user-accounts
 django-uuidfield==0.5.0
-django-queryset-csv==0.2.10
 newrelic==2.58.2.45
 pdfkit==0.4.1
 django-hermes
 mock
-jsonfield
 eventlet
 django-debug-toolbar==1.2.1
 django_import_export

webapp/apps/btax/migrations/0001_initial.py

@@ -3,9 +3,8 @@
 
 from django.db import models, migrations
 import datetime
-import jsonfield.fields
 from django.conf import settings
-import uuidfield.fields
+import uuid
 import webapp.apps.taxbrain.models
 
 
@@ -22,7 +21,7 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('model_pk', models.IntegerField(default=None, null=True)),
                 ('exp_comp_datetime', models.DateTimeField(default=datetime.datetime(2015, 1, 1, 0, 0))),
-                ('uuid', uuidfield.fields.UUIDField(null=True, default=None, editable=False, max_length=32, blank=True, unique=True)),
+                ('uuid', models.UUIDField(null=True, default=None, editable=False, max_length=32, blank=True, unique=True)),
                 ('btax_vers', models.CharField(default=None, max_length=50, null=True, blank=True)),
                 ('taxcalc_vers', models.CharField(default=None, max_length=50, null=True, blank=True)),
             ],
@@ -93,7 +92,7 @@ class Migration(migrations.Migration):
                 ('job_ids', webapp.apps.taxbrain.models.SeparatedValuesField(default=None, null=True, blank=True)),
                 ('jobs_not_ready', webapp.apps.taxbrain.models.SeparatedValuesField(default=None, null=True, blank=True)),
                 ('first_year', models.IntegerField(default=None, null=True)),
-                ('tax_result', jsonfield.fields.JSONField(default=None, null=True, blank=True)),
+                ('tax_result', models.TextField(default=None, null=True, blank=True)),
                 ('creation_date', models.DateTimeField(default=datetime.datetime(2015, 1, 1, 0, 0))),
             ],
             options={

webapp/apps/btax/migrations/0003_auto_20180205_2151.py

@@ -0,0 +1,26 @@
+# -*- coding: utf-8 -*-
+# Generated by Django 1.9 on 2018-02-05 21:51
+from __future__ import unicode_literals
+
+from django.db import migrations, models
+import uuid
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('btax', '0002_btaxoutputurl_webapp_vers'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='btaxoutputurl',
+            name='uuid',
+            field=models.UUIDField(blank=True, default=uuid.uuid4, editable=False, null=True, unique=True),
+        ),
+        migrations.AlterField(
+            model_name='btaxsaveinputs',
+            name='tax_result',
+            field=models.TextField(blank=True, default=None, null=True),
+        ),
+    ]

webapp/apps/btax/models.py

@@ -1,4 +1,5 @@
 import re
+import uuid
 
 from django.db import models
 from django.core import validators
@@ -7,13 +8,11 @@
 from django.core.validators import MinValueValidator, MaxValueValidator, RegexValidator
 from django.contrib.auth.models import User
 
-from uuidfield import UUIDField
-from jsonfield import JSONField
+from django.contrib.postgres.fields import JSONField
 import datetime
 
 from ..taxbrain.models import (SeparatedValuesField,
-                               CommaSeparatedField,
-                               convert_to_floats)
+                               CommaSeparatedField)
 
 
 
@@ -117,7 +116,7 @@ class BTaxSaveInputs(models.Model):
     first_year = models.IntegerField(default=None, null=True)
 
     # Result
-    tax_result = JSONField(default=None, blank=True, null=True)
+    tax_result = models.TextField(default=None, blank=True, null=True)
     # Creation DateTime
     creation_date = models.DateTimeField(default=datetime.datetime(2015, 1, 1))
 
@@ -138,7 +137,7 @@ class BTaxOutputUrl(models.Model):
     model_pk = models.IntegerField(default=None, null=True)
     # Expected Completion DateTime
     exp_comp_datetime = models.DateTimeField(default=datetime.datetime(2015, 1, 1))
-    uuid = UUIDField(auto=True, default=None, null=True)
+    uuid = models.UUIDField(default=uuid.uuid4, null=True, editable=False, max_length=32, blank=True, unique=True)
     btax_vers = models.CharField(blank=True, default=None, null=True, max_length=50)
     taxcalc_vers = models.CharField(blank=True, default=None, null=True, max_length=50)
     webapp_vers = models.CharField(blank=True, default=None, null=True,

webapp/apps/btax/tests/test_views.py

@@ -4,7 +4,6 @@
 
 from ..models import BTaxSaveInputs, BTaxOutputUrl
 from ...taxbrain.models import WorkerNodesCounter
-from ..models import convert_to_floats
 from ..compute import (DropqComputeBtax, MockComputeBtax,
                        MockFailedComputeBtax, NodeDownComputeBtax)
 import taxcalc

webapp/apps/btax/urls.py

@@ -4,9 +4,9 @@
                     edit_btax_results, generate_mock_results)
 
 
-urlpatterns = patterns('',
+urlpatterns = [
     url(r'^$', btax_results, name='btax_tax_form'),
     url(r'^(?P<pk>\d+)/', output_detail, name='btax_output_detail'),
     url(r'^edit/(?P<pk>\d+)/', edit_btax_results, name='btax_edit_btax_results'),
     url(r'^mock-ccc-results', generate_mock_results, name='btax_generate_mock_results')
-)
+]

webapp/apps/btax/views.py

@@ -19,7 +19,7 @@
 from ipware.ip import get_real_ip
 
 from django.core import serializers
-from django.core.context_processors import csrf
+from django.template.context_processors import csrf
 from django.core.exceptions import ValidationError
 from django.contrib.auth.decorators import login_required, permission_required
 from django.http import HttpResponseRedirect, HttpResponse, Http404, JsonResponse
@@ -31,8 +31,6 @@
 from django.contrib.auth.models import User
 from django import forms
 
-from djqscsv import render_to_csv_response
-
 from .forms import BTaxExemptionForm, has_field_errors
 from .models import BTaxSaveInputs, BTaxOutputUrl
 from .helpers import (get_btax_defaults,
@@ -42,8 +40,7 @@
                       make_bool, convert_val)
 from ..taxbrain.helpers import (format_csv,
                                 is_wildcard)
-from ..taxbrain.views import (benefit_switch_fixup,
-                              denormalize, normalize)
+from ..taxbrain.views import denormalize, normalize
 from .compute import DropqComputeBtax, MockComputeBtax, JobFailError
 
 from ..constants import (METTR_TOOLTIP, METR_TOOLTIP, COC_TOOLTIP, DPRC_TOOLTIP,
@@ -101,15 +98,18 @@ def btax_results(request):
     """
     no_inputs = False
     start_year = START_YEAR
-    REQUEST = request.REQUEST
     if request.method=='POST':
         print 'POST'
         # Client is attempting to send inputs, validate as form data
         # Need need to the pull the start_year out of the query string
         # to properly set up the Form
         has_errors = make_bool(request.POST['has_errors'])
-        start_year = REQUEST['start_year']
-        fields = dict(REQUEST)
+        fields = dict(request.GET)
+        fields.update(dict(request.POST))
+        fields = {k: v[0] if isinstance(v, list) else v for k, v in fields.items()}
+        start_year = fields.get('start_year', START_YEAR)
+        # TODO: migrate first_year to start_year to get rid of weird stuff like
+        # this
         fields['first_year'] = fields['start_year']
         btax_inputs = BTaxExemptionForm(start_year, fields)
         btax_inputs = make_bool_gds_ads(btax_inputs)
@@ -355,8 +355,8 @@ def output_detail(request, pk):
     if model.tax_result:
         exp_num_minutes = 0.25
         JsonResponse({'eta': exp_num_minutes, 'wait_interval': 15000}, status=202)
-
-        tables = url.unique_inputs.tax_result[0]
+        tax_result = url.unique_inputs.tax_result
+        tables = json.loads(tax_result)[0]
         first_year = url.unique_inputs.first_year
         created_on = url.unique_inputs.creation_date
         tables["tooltips"] = {
@@ -365,7 +365,7 @@ def output_detail(request, pk):
             "coc": COC_TOOLTIP,
             "dprc": DPRC_TOOLTIP,
         }
-        bubble_js, bubble_div, cdn_js, cdn_css = bubble_plot_tabs(model.tax_result[0]['dataframes'])
+        bubble_js, bubble_div, cdn_js, cdn_css = bubble_plot_tabs(tables['dataframes'])
 
         inputs = url.unique_inputs
         is_registered = True if request.user.is_authenticated() else False
@@ -416,8 +416,8 @@ def output_detail(request, pk):
 
 
         if all([job == 'YES' for job in jobs_ready]):
-            model.tax_result = dropq_compute.btax_get_results(normalize(job_ids))
-
+            tax_result = dropq_compute.btax_get_results(normalize(job_ids))
+            model.tax_result = json.dumps(tax_result)
             model.creation_date = datetime.datetime.now()
             print 'ready'
             model.save()