Django upgrade #3073

ubernostrum · 2015-02-11T05:21:52Z

This is the tracking PR for upgrading kuma to Django 1.7. This PR contains our project meeting notes & tasks.

The process:

git fetch mozilla
get checkout mozilla/django-upgrade-1116335
git submodule sync && git submodule update --init
find . -name "*.pyc" | xargs rm -f
(vagrant-only) vagrant provision
Either:

django-compat-lint (bash script) kuma with the 1.7 ruleset
./manage.py test 2> test-stderr.txt

Analyze common errors
Pick an error to fix
Add yourself to a "Tasks for code" item below

Tasks for code:

Tasks for deploy:
Anytime:

with Sheeri - Add utf8_distinct_ci collation to MySQL (bug 1161107)

May 26

With Jake and cyliang: Create new python 2.7 virtualenv (bug 1148435)
With Jake and cyliang: Install dependencies from this branch's compiled.txt requirements file into the new virtualenv

June 2

Ping VM team re: snapshot & potential restore
Ping MDN teams re: stage server freeze
Deploy branch to stage (logs)
- http://developeradm.private.scl3.mozilla.com/chief/developer.stage/logs/3c95d5eabe720b5ca9e25ececfba7da9786325e3.1433259513
  - error: err: fatal: destination path 'vendor/src/elasticsearch-dsl-py' already exists and is not an empty directory. (cause)
  - fix: rm -rf vendor/src/elasticsearch-dsl-py
- http://developeradm.private.scl3.mozilla.com/chief/developer.stage/logs/3c95d5eabe720b5ca9e25ececfba7da9786325e3.1433259741
  - error: ImportError: cannot import name widgets
  - fix: find . -name "*.pyc" | xargs rm -f
- http://developeradm.private.scl3.mozilla.com/chief/developer.stage/logs/3c95d5eabe720b5ca9e25ececfba7da9786325e3.1433262124
  - error: ImportError: No module named Image
  - fix: add VENV_DIR to the commander_settings.py and 2d8b7d7
- http://developeradm.private.scl3.mozilla.com/chief/developer.stage/logs/2d8b7d7fac66d413e26397960a43793ce8a1a94f.1433271998
  - error: ImportError: No module named git
  - fix: 0836005
- http://developeradm.private.scl3.mozilla.com/chief/developer.stage/logs/0836005e29b798bea544be625c88c903a0b40919.1433276127
  - error: django.db.utils.OperationalError: (1060, "Duplicate column name 'user_id'")
  - fix: Continue with fake migrations below ...
With Jake and cyliang: Convert web heads Apache to use python 2.7 path
- error: https://cliang.pastebin.mozilla.org/8835477
  - fix: Hrm. Didn't make it from the admin host to the webhead. Will try to force the files out ... W00t. that did it.
- error: all-servers.conf file moved. (Was in kuma/etc/apache/all-servers.conf; now in kuma/provisioning/roles/kuma/files/apache2/all-servers.conf)
  - fix: Altering include line
- error: https://cliang.pastebin.mozilla.org/8835480
  - fix: Remove https://github.com/mozilla/kuma/blame/master/wsgi/kuma.wsgi#L36-L41 (0276e9f)
With Jake and cyliang: Manually do fake migrations during the first deployment
Deploy branch to stage again
- http://developeradm.private.scl3.mozilla.com/chief/developer.stage/logs/0276e9ff5b338e845943d8d7cc952676e7afb822.1433282257
  - error: Unknown command: 'update_badges'
  - fix: 8068ff6
Update cron jobs to use new python2.7 venv

June 2-4

Stage Test, test, test with writers, WebQA, devs, et al.
- Bug 1171977 - ISE when submitting a demo on staging
Fix Travis builds on branch

June 9

Merge branch into master

June 9

Repeat deployment steps on Prod (logs)
- http://developeradm.private.scl3.mozilla.com/chief/developer.prod/logs/ca49d1fc849d92445c1abf98415a32efdc3aa3d2.1433879425
  - error: SyntaxError: invalid syntax
  - fix: push again because only now the new chief deploy file was on the admin node
- http://developeradm.private.scl3.mozilla.com/chief/developer.prod/logs/ca49d1fc849d92445c1abf98415a32efdc3aa3d2.1433879541
  - error: django.db.utils.OperationalError: (1060, "Duplicate column name 'user_id'")
  - fix: run fake and real migrations
- With @cyliang run migrations
  - migrate --fake wiki 0001 -> DONE
  - migrate wiki 0002 -> DONE
  - migrate --fake -> DONE
- http://developeradm.private.scl3.mozilla.com/chief/developer.prod/logs/a1d3727bf7f8eb147e4a71d4f5294470d402121f.1433880741
  - error: Syntax error in WSGI script while starting httpd caused by 06b3163
  - fix: a1d3727
- Increased errors on search page with search filter tags
  - error: https://rpm.newrelic.com/accounts/263620/applications/3172075/traced_errors/3066f1-ceb7f59a-0ee5-11e5-a48a-f8bc124250a8?original_error_id=3066f1-efed656b-0ee5-11e5-a48a-f8bc124250a8
  - fix: 73cdf4f

Tasks for post-deploy:

groovecoder · 2015-02-18T16:03:39Z

@ubernostrum - how's this branch going? Are you updating more libraries from vendor/src to 1.6-compatible versions?

groovecoder · 2015-02-26T19:08:01Z

Ping @ubernostrum - 8 days without an update here. 😦

jezdez · 2015-03-03T16:42:08Z

Ok, I've found a series of issues that need to be addressed before moving along with this. That said, I think this looks great in general, lots of changes, but all worth it. We may end up in the present eventually!!

ubernostrum · 2015-03-05T22:10:10Z

I've cleaned up most of the submodule version issues (and even removed one!), but there are a few things that need further poking or decisions:

Decide on a safe version of django-rest-framework.
Try out mainline bleach instead of Les' fork.
Decide if we really do need httplib2.
Decide if we're going to get rid of grappelli.
Get me to do a proper PyPI release of soapbox.

groovecoder · 2015-03-12T22:49:53Z

I investigated DRF and emailed the summary to dev-mdn (May also want to watch encode/django-rest-framework#2681 and encode/django-rest-framework#2684.)

I suggest we update to 2.3.14 immediately, and probably 2.4.4 soon after.

groovecoder · 2015-03-12T22:51:23Z

Also, mdn-dev and devdoc-writers confirm we can remove grappelli.

groovecoder · 2015-03-18T22:09:37Z

#3109 is now merged, so this could rebase on master and then it's ready to run tests?

WARNING: This commit breaks compatibility with Django 1.4. Many of the warnings from the linter are actually spurious/overzealous reminders about checking the existence of cleaned_data (bad), where we actually were checking for the presence of specific elements in cleaned_data (good), but which django-compat-lint can't handle. As such they were left as-is, and only the actually-relevant changes (which are fewer in number) have been made.

This clears all the non-spurious warnings/errors from the lint output; the primary change here is transaction handling, with the changes falling into three main areas: 1. Replacing old transaction decorators with the new 'atomic' one, since it provides the behavior we wanted. 2. Removing use of 'transaction.autocommit', since autocommit is now the default behavior of Django. 3. Rewriting the page-move task -- which is probably the trickiest transaction-related code -- to temporarily disable autocommit, manually manage the page-move transaction, and turn autocommit back on afterward.

groovecoder · 2015-06-01T22:30:10Z

I incorporated the fix for amqp connection in the vm. I ran the new intern client-side test suite against this branch and fixed a bug I found in the newsfeed template. (I also sent a PR to improve the intern test suite messages.)

Only thing I know we have to do now is fix Travis CI. I'll work on that tomorrow.

groovecoder · 2015-06-03T04:47:24Z

I'm using #3232 against this branch to let me test Travis builds without committing here (yet).

I couldn't find a path to follow with running ansible connections to Travis CI localhost, so I started a python script that re-uses those same Jinja2 templates as ansible to install the Index.xml file into MySQL.

ubernostrum · 2015-06-05T10:37:52Z

When trying to follow vagrant install instructions, getting this at the submodule init step:

$ git submodule update --init --recursive
Submodule 'testdata' (https://github.com/html5lib/html5lib-tests.git) registered for path 'html5lib/tests/testdata'
Cloning into 'html5lib/tests/testdata'...
remote: Counting objects: 3953, done.
remote: Total 3953 (delta 0), reused 0 (delta 0), pack-reused 3953
Receiving objects: 100% (3953/3953), 4.91 MiB | 2.74 MiB/s, done.
Resolving deltas: 100% (2028/2028), done.
Checking connectivity... done.
Submodule path 'vendor/src/html5lib/html5lib/tests/testdata': checked out 'f6a1b202de14fc057b196044c5ebef4672be3dd0'
fatal: Needed a single revision
Unable to find current revision in submodule path 'vendor/src/urlobject'

Not sure if that's an actual issue with submodules, or with the wifi I'm on (which has been messing with connections all week).

groovecoder · 2015-06-05T14:22:09Z

I destroyed and re-built my vm last night and had no issue. Probably wifi connection.

jezdez · 2015-06-09T11:18:42Z

2924329 is wrong. settings_test.py is committed to the repo (until we merge the settings-refactor), so overwriting it with a different file from Ansible will result in an unclean git index (aka you have to commit it). Since we're deprecating non-Vagrant installations (at least from a supported maintenance perspective).

jezdez · 2015-06-09T11:19:22Z

woops, part of the sentence missing.

..I'm going to hardcode that path in the settings_test.py.

…e MySQL collation installation steps.

jezdez · 2015-06-09T11:35:29Z

So apparently there are problems with MySQL on Travis, travis-ci/travis-ci#2353 (reported by the author of the Ansible roles we're using for MySQL). So I'm going to uninstall the Travis provided MySQL and install the community server from MySQL itself (https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/index.html). Because I haven't explored a rabbit cave in recent weeks

… django-upgrade-1116335

jezdez · 2015-06-09T11:36:59Z

Ran 625 tests in 214.740s

OK (SKIP=10)

groovecoder · 2015-06-09T14:27:38Z

Another merge from master.

Ran 625 tests in 162.653s

OK (SKIP=10)

jezdez · 2015-06-09T19:27:17Z

Django upgrade

ubernostrum added not ready labels Feb 11, 2015

ubernostrum force-pushed the django-upgrade-1116335 branch from 26df89b to de74f1e Compare February 25, 2015 02:53

jezdez assigned ubernostrum Mar 3, 2015

ubernostrum added 19 commits March 30, 2015 23:42

Bug 1116335: Upgrade django-taggit for Django 1.6 compatibility.

027a39a

Bug 1116335: Upgrade Django to 1.6.10.

0d57c69

Bug 1116335: Move and upgrade httplib2.

632eb97

Bug 1116335: Update django-rest-framework to 3.0.5.

e3ffa40

Bug 1116335: Update django-mozilla-product-details to latest HEAD.

6192d81

Bug 1116335: Update jingo to 0.7.

0421224

Bug 1116335: Update django-grappelli to 2.6.3.

c9c03de

Bug 1116335: Update django-statici18n to 1.1.1.

463acc3

Bug 1116335: Update django-tidings to 0.4.

9a220d2

Bug 1116335: Update django-waffle to 0.10.1.

4fee1e0

Bug 1116335: Update django-constance to 1.0.1.

2e543c7

Bug 1116335: Update django-appconf to 1.0.1.

bbfecbe

Bug 1116335: Update django-smuggler to 0.6.0.

0fee1a8

Bug 1116335: Update commonware to 0.4.3.

ef77bcf

Bug 1116335: Update jingo-minify to latest HEAD.

900c193

Bug 1116335: Update django-allauth to 0.19.1.

3c63bf6

Bug 1116335: Update django-recaptcha to 1.0.3.

61c7b70

Bug 1116335: Re-lint all of kuma through to Django 1.7.

09e5560

ubernostrum force-pushed the django-upgrade-1116335 branch from b8b8f83 to 09e5560 Compare June 1, 2015 18:47

groovecoder added 2 commits June 1, 2015 14:00

bug 1116335 - tcp_listener vars for rabbitmq

880cf5f

bug 1116335 - fix update_feeds and newsfeed template

3c95d5e

groovecoder added 4 commits June 2, 2015 14:01

bug 1116335 - update chief to use venv

2d8b7d7

bug 1116335 - restore GitPython for compress_assets

0836005

bug 1116335 - update kuma.wsgi for django 1.7

0276e9f

bug 1116335 - remove update_badges from chief_deploy

8068ff6

jezdez and others added 3 commits June 8, 2015 15:03

Bug 1116335 - Further cleanup of WSGI file.

06b3163

Merge branch 'master' into django-upgrade-1116335

e5caa0c

bug 1116335 - small flake8 fixes for Travis

3b70379

Bug 1116335 - Added an Ansible playbook for Travis so we can reuse th…

ed67af1

…e MySQL collation installation steps.

Merge branch 'django-upgrade-1116335' of github.com:mozilla/kuma into…

5c89cc2

… django-upgrade-1116335

Merge branch 'master' into django-upgrade-1116335

c565bf9

jezdez changed the title ~~(DO NOT MERGE) Django upgrade~~ Django upgrade Jun 9, 2015

jezdez added a commit that referenced this pull request Jun 9, 2015

Merge pull request #3073 from mozilla/django-upgrade-1116335

ca49d1f

Django upgrade

jezdez merged commit ca49d1f into master Jun 9, 2015

groovecoder deleted the django-upgrade-1116335 branch June 10, 2015 13:49

jezdez removed the not ready label Jun 15, 2015

groovecoder mentioned this pull request Nov 12, 2015

bug 1139490 - remove GitPython #3648

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Django upgrade #3073

Django upgrade #3073

ubernostrum commented Feb 11, 2015

groovecoder commented Feb 18, 2015

groovecoder commented Feb 26, 2015

jezdez commented Mar 3, 2015

ubernostrum commented Mar 5, 2015

groovecoder commented Mar 12, 2015

groovecoder commented Mar 12, 2015

groovecoder commented Mar 18, 2015

groovecoder commented Jun 1, 2015

groovecoder commented Jun 3, 2015

ubernostrum commented Jun 5, 2015

groovecoder commented Jun 5, 2015

jezdez commented Jun 9, 2015

jezdez commented Jun 9, 2015

jezdez commented Jun 9, 2015

jezdez commented Jun 9, 2015

groovecoder commented Jun 9, 2015

jezdez commented Jun 9, 2015

Django upgrade #3073

Django upgrade #3073

Conversation

ubernostrum commented Feb 11, 2015

groovecoder commented Feb 18, 2015

groovecoder commented Feb 26, 2015

jezdez commented Mar 3, 2015

ubernostrum commented Mar 5, 2015

groovecoder commented Mar 12, 2015

groovecoder commented Mar 12, 2015

groovecoder commented Mar 18, 2015

groovecoder commented Jun 1, 2015

groovecoder commented Jun 3, 2015

ubernostrum commented Jun 5, 2015

groovecoder commented Jun 5, 2015

jezdez commented Jun 9, 2015

jezdez commented Jun 9, 2015

jezdez commented Jun 9, 2015

jezdez commented Jun 9, 2015

groovecoder commented Jun 9, 2015

jezdez commented Jun 9, 2015