Django Upgrade (1.8 to 1.11)

[dongbohu]

According to Django's official doc:
https://www.djangoproject.com/download/#supported-versions
Djanog 1.11.x seems to be a reasonable version to upgrade to for two reasons:

• It is an LTS that will be supported until at least April 2020.
• It is the last version that supports Python 2.7.

Once Django is upgrade to 1.11, many other Django-related packages should be upgraded too. I am going to use this issue to keep track of the upgrade info for all backend packages that depend upon Django.

[dongbohu]

Django related:

• Django 1.11.16 -->

• Django-haystack 2.8.1 (latest)
  https://django-haystack.readthedocs.io/en/v2.8.1/installing_search_engines.html) -->

• elasticsearch 2.x (ES binding on PyPI) -->

• Elasticsearch 2.X (backend)

• django-allauth?

• django-appconf?

• django-braces?

• django-celery-transactions?

• django-fixtureless?

• django-genes (in house)

• django-oauth-toolkit?

• django-organisms (in house)

• django-taggit?

• django-tastypie?

• django-transaction-hooks?

• django-uuidfield

Notes:

django-haystack-es (Django Haystack backend for Elasticsearch 5) in PyPI:
https://pypi.org/project/django-haystack-es/

defusedxml and lxml are optional packages for django-tastypie, see:

• https://bitbucket.org/greenelab/tribe/commits/c06a6cdab326a6452bf6f375ea3271679b969f2c
• https://django-tastypie.readthedocs.io/en/latest/#format-support

[dongbohu]

Celery related:

• Celery 4.x: the last version that supports Python 2.x
  http://docs.celeryproject.org/en/latest/history/whatsnew-4.0.html#django-support

• celery-haystack (a sub-project maintained by django-haystack team):
  https://github.com/django-haystack/celery-haystack
  https://pypi.org/project/celery-haystack/ (last updated in PyPI in November 2015)

• Celery dependencies:
  http://docs.celeryproject.org/en/stable/faq.html#does-celery-have-many-dependencies

  • kombu (depends on amqp)
  • billiard
  • pytz (also a dependency of Django)
  • django-celery (seems to be replaced by django-celery-result now)
    • https://pypi.org/project/django_celery_results/
    • http://docs.celeryproject.org/en/stable/faq.html?highlight=django-celery-result#what-purpose-does-the-database-tables-created-by-django-celery-results-have

[dongbohu]

django-haystack 1.X -> 2.X:
https://django-haystack.readthedocs.io/en/master/searchqueryset_api.html#field-lookups

The "content" filter became the new default filter as of Haystack v2.X (the default in 
Haystack v1.X was "exact"). This changed because "exact" caused problems and 
was unintuitive for new people trying to use Haystack. "content" is a much more 
natural usage.

If you had an app built on Haystack v1.X & are upgrading, you’ll need to sanity-check
& possibly change any code that was relying on the default. The solution is just to 
add "__exact" to any “bare” field in a ".filter(...)" clause.

Files affected:

• tribe/genesets/api/resources.py: line 478

[dongbohu]

The following packages in requirements/base.txt may be removed after we upgrade Django to 1.9+:

• django-transction-hooks: https://django-transaction-hooks.readthedocs.io/en/latest/
• django-uuidfield: obsolete since Django 1.8+, see: Django 1.8+ dcramer/django-uuidfield#63 (comment)
• shortuuid: ?

[dongbohu]

According to Celery 3.1 documentation:
http://docs.celeryproject.org/en/3.1/django/first-steps-with-django.html#using-the-django-orm-cache-as-a-result-backend
django-celery (aka. djcelery) is needed only when we save the celery task results back to Django's backend database. But on Tribe we are only using Celery to update ELasticsearch index. So django-celery may not be necessary.

Here are the files that refers to djcelery in Tribe requirement packages:

• celery/bin/beat.py
• celery/events/snapshot.py
• celery/loaders/__init__.py
• celery_haystack/test_settings.py
• raven/contrib/django/models.py
• raven/contrib/django/celery/models.py

Confirmed that none of the djcelery* tables in Postgres database has any real data.

[dongbohu]

django.contrib.auth.middleware.SessionAuthenticationMiddleware should be removed from MIDDLEWARE_CLASSES when upgrading to Django 1.10+:
https://docs.djangoproject.com/en/2.1/releases/1.10/#features-removed-in-1-10

Session verification is enabled regardless of whether or not 
'django.contrib.auth.middleware.SessionAuthenticationMiddleware' is in 
'MIDDLEWARE_CLASSES'. 'SessionAuthenticationMiddleware' no longer 
has any purpose and can be removed from 'MIDDLEWARE_CLASSES'.

[dongbohu]

When Django is upgraded to 1.9+, this error shows up:

File "/home/tribe/tribe/versions/models.py", line 29, in <module>
    class FrozenSetField(models.TextField):
  File "/home/tribe/tribe/versions/models.py", line 34, in FrozenSetField
    __metaclass__ = models.SubfieldBase
AttributeError: 'module' object has no attribute 'SubfieldBase'

Here is a solution:
https://stackoverflow.com/questions/35166085/how-to-deal-with-subfieldbase-has-been-deprecated-use-field-from-db-value-inst

[dongbohu]

Some package version requirements:

• Python 2.x requires Django 1.11.x
• django-celery 3.2.2 requires celery<4.0,>=3.1.15
• django-fixtureless has to be 1.5.1 because Factory has been removed since 1.6.x.
• django-tastypie has to be 0.13.1 to avoid this error:

File ".../.virtualenvs/tribe/local/lib/python2.7/site-packages/tastypie/fields.py", 
line 744, in contribute_to_class
    related_field = getattr(self._resource._meta.object_class, self.attribute, None)
TypeError: Error when calling the metaclass bases
    getattr(): attribute name must be string

See this issue: django-tastypie/django-tastypie#1581

Here is a new version of requirements/base.txt with Django 1.11.20, but it has two unsolved issues:

• django-tastypie 0.14.2 is not compatible (see description above)
• Elasticsearch web UI doesn't seem to be working

amqp==1.4.9
anyjson==0.3.3
billiard==3.3.0.23
celery==3.1.26.post2
celery-haystack==0.10
certifi==2018.11.29
chardet==3.0.4
contextlib2==0.5.5
defusedxml==0.5.0
Django==1.11.20
django-allauth==0.38.0
django-appconf==1.0.2
django-celery==3.2.2
django-fixtureless==1.5.1
django-genes==0.18
django-haystack==2.8.1
django-oauth-toolkit==1.1.2
django-organisms==0.6
django-taggit==0.24.0
django-tastypie==0.14.2
dogslow==1.2
elasticsearch==2.4.1
funcsigs==1.0.2
idna==2.8
kombu==3.0.37
lxml==4.3.1
mock==2.0.0
oauthlib==3.0.1
pbr==5.1.2
psycopg2-binary==2.7.7
python-dateutil==2.8.0
python-mimeparse==1.6.0
python-openid==2.2.5
pytz==2018.9
raven==6.10.0
requests==2.21.0
requests-oauthlib==1.2.0
six==1.12.0
urllib3==1.24.1
vine==1.2.0

"gunicorn 19.9.0" is not listed here because it is included in requirements/prod.txt.

[dongbohu]

Installing this patch:

pip -e git+https://github.com/django-tastypie/django-tastypie@6cb7da757fd28e0541adefd7a7261df452123750#egg=TastyPie

seems to fix the tastypie issues on Django 1.11.20,

Edit: The patch has been merged into django-tastypie's master branch by the following PR:
django-tastypie/django-tastypie#1583

But now the following two test cases failed:

======================================================================
FAIL: testGOBPQuery (genesets.tests.GenesetUnregisteredTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tribe/tribe/genesets/tests.py", line 267, in testGOBPQuery
    self.assertEqual(len(self.deserialize(resp)['objects']), 1)
AssertionError: 0 != 1

======================================================================
FAIL: testSearchIndexUpdate (genesets.tests.GenesetUnregisteredTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/tribe/tribe/genesets/tests.py", line 300, in testSearchIndexUpdate
    self.assertEqual(len(self.deserialize(resp)['objects']), 2)
AssertionError: 0 != 2

The reason seems to be because in:

@override_settings(HAYSTACK_CONNECTIONS=TEST_INDEX, CELERY_ALWAYS_EAGER=True,
                   CELERY_EAGER_PROPAGATES_EXCEPTIONS=True)

CELERY_ALWAYS_EAGER=True is not effective.

If we add:

call_command('update_index', interactive=False, verbosity=0)

in the two failed test cases, they will pass. But why?

[dongbohu]

PR #41 and #42 have removed most django-1.7 deprecated features. Once the new django-tastypie is released with the bug fix (django-tastypie/django-tastypie#1583), we need to update base.txt and figure out how to fix the two failed test cases above.

[dongbohu]

Fixed by PR #52.