Upgrade to Django 1.11 and Python 3.6

[maurizi]

Overview

This PR upgrades to Django 1.11 and Python 3.6, and additionally updates a variety of Django related dependencies.

Demo

N/A

Notes

I wasn't originally intending to update Python at the same time as Django, but we don't have any Django 1.11 / Python 3.5 Docker images so I updated Python at the same time, which doesn't seem to have caused any fallout.

As these updates touch every part of the application, I added support for generating code coverage reports so I could determine which parts of the application were covered by the automated test suite and which needed more dedicated manual testing. Based on that, I manually tested the following parts of the application:

• Registration
• Forgot password workflow
• Creating a superuser
• The Django Admin site
• The user profile page
• The change password workflow for logged in users
• Integration with the Lab (charts, creating projects, login)
• Integration with Temperate (charts, top concern calculations, creating organizations)
• The /city, /city/<pk>/ and /city/<pk>/map-cell/ endpoints

I still need to test email and static file hosting in the staging environment (I was unable to test these locally).

I intentionally didn't go any further with adding code coverage past being able to use it locally. If we want to add integration with Jenkins or an external service I'd like to add that in later.

Testing Instructions

• scripts/update
• Uh... Test everything?
• Running scripts/test should generate an htmlcov directory with code coverage results

Checklist

• Does the python linter pass?
• Do tests pass?
• [ ] Has the API documentation been updated, or does this PR not require it?

Closes #411
Closes #879

[flibbertigibbet]

After destroying the VM and running ./scripts/setup, docker-compose seems to be unavailable inside the new VM.

vagrant@vagrant-ubuntu-trusty-64:~/climate-change-api$ ./scripts/update 
./scripts/update: line 59: docker-compose: command not found

[flibbertigibbet]

🎊

[flibbertigibbet]

🎆

[ddohler]

I was able to replicate the issue with docker-compose not being available on a new VM. It looks like it's a problem with Python 2.7 and TLS inside the VM:

    default: TASK [azavea.python-security : Install Python TLS SNI dependencies] ************
    default: Monday 08 April 2019  18:17:10 +0000 (0:00:00.034)       0:00:05.476 ********** 
    default: fatal: [localhost]: FAILED! => 
{"changed": false, "cmd": "/usr/local/bin/pip2 install urllib3[secure]==1.22.*", "failed": true, "msg": "stdout: Collecting urllib3[secure]==1.22.*\n  Downloading https://files.pythonhosted.org/packages/63/cb/6965947c13a94236f6d4b8223e21beb4d576dc72e8130bd7880f600839b8/urllib3-1.22-py2.py3-none-any.whl (132kB)\nCollecting idna>=2.0.0; python_version <= \"2.7\" and extra == \"secure\" (from urllib3[secure]==1.22.*)\n  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)\nCollecting pyOpenSSL>=0.14; python_version <= \"2.7\" and extra == \"secure\" (from urllib3[secure]==1.22.*)\n  Downloading https://files.pythonhosted.org/packages/01/c8/ceb170d81bd3941cbeb9940fc6cc2ef2ca4288d0ca8929ea4db5905d904d/pyOpenSSL-19.0.0-py2.py3-none-any.whl (53kB)\nRequirement already satisfied: cryptography>=1.3.4; python_version <= \"2.7\" and extra == \"secure\" in /usr/local/lib/python2.7/dist-packages (from urllib3[secure]==1.22.*) (2.6.1)\nCollecting certifi; extra == \"secure\" (from urllib3[secure]==1.22.*)\n  Downloading https://files.pythonhosted.org/packages/60/75/f692a584e85b7eaba0e03827b3d51f45f571c2e793dd731e598828d380aa/certifi-2019.3.9-py2.py3-none-any.whl (158kB)\nRequirement already satisfied: ipaddress; python_version <= \"2.7\" and extra == \"secure\" in /usr/local/lib/python2.7/dist-packages (from urllib3[secure]==1.22.*) (1.0.22)\nRequirement already satisfied: six>=1.5.2 in /usr/lib/python2.7/dist-packages (from pyOpenSSL>=0.14; python_version <= \"2.7\" and extra == \"secure\"->urllib3[secure]==1.22.*) (1.5.2)\nRequirement already satisfied: enum34; python_version < \"3\" in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.3.4; python_version <= \"2.7\" and extra == \"secure\"->urllib3[secure]==1.22.*) (1.1.6)\nRequirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.3.4; python_version <= \"2.7\" and extra == \"secure\"->urllib3[secure]==1.22.*) (1.12.2)\nRequirement already satisfied: asn1crypto>=0.21.0 in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.3.4; python_version <= \"2.7\" and extra == \"secure\"->urllib3[secure]==1.22.*) (0.24.0)\nRequirement already satisfied: pycparser in /usr/local/lib/python2.7/dist-packages (from cffi!=1.11.3,>=1.8->cryptography>=1.3.4; python_version <= \"2.7\" and extra == \"secure\"->urllib3[secure]==1.22.*) (2.19)\nInstalling collected packages: idna, pyOpenSSL, certifi, urllib3\n  Found existing installation: pyOpenSSL 0.13\n\n:stderr: /usr/local/lib/python2.7/dist-packages/cryptography/hazmat/primitives/constant_time.py:26: CryptographyDeprecationWarning: Support for your Python version is deprecated. The next version of cryptography will remove support. Please upgrade to a 2.7.x release that supports hmac.compare_digest as soon as possible.\n  utils.PersistentlyDeprecated2018,\nDEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.\n/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:354: SNIMissingWarning: An HTTPS request has been made, but the SNI (Server Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n  SNIMissingWarning\n/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:150: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n  InsecurePlatformWarning\n/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:150: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings\n  InsecurePlatformWarning\nCannot uninstall 'pyOpenSSL'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.\n"}

This causes provisioning to fail before it gets to installing docker-compose. I confirmed that this happens on develop, so it's not directly related to this PR.

[maurizi]

@flibbertigibbet I was able to replicate the issue you encountered and fix it by upgrading the Vagrant VM to a newer version in 488efd1

I then encountered a different error in provisioning the PostgreSQL APT keys which I worked-around in d9260d7 fixed in 5bb5760 by upgrading Ansible to 2.4

[flibbertigibbet]

The only issues I encountered in testing the API with either the lab or in Temperate were related to having outdated fixtures. Manually adding scenario alias text and running generate_historic instead of using fixtures was enough to work around the fixture issues.