New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Django upgrade #3073

Merged
merged 223 commits into from Jun 9, 2015

Conversation

Projects
None yet
4 participants
@ubernostrum
Contributor

ubernostrum commented Feb 11, 2015

This is the tracking PR for upgrading kuma to Django 1.7. This PR contains our project meeting notes & tasks.

The process:

  1. git fetch mozilla
  2. get checkout mozilla/django-upgrade-1116335
  3. git submodule sync && git submodule update --init
  4. find . -name "*.pyc" | xargs rm -f
  5. (vagrant-only) vagrant provision
  6. Either:
  1. Analyze common errors
  2. Pick an error to fix
  3. Add yourself to a "Tasks for code" item below

Tasks for code:

  • @groovecoder verify updated django-statici18n usage
  • @groovecoder - throw new enhanced intern tests at the branch
  • @jezdez - convert puppet provisioning to ansible
    • debug rabbitmq install
  • @jezdez - Update kuma setup docs
    • Remove host machine install docs
    • Convert vagrant/vm docs from puppet to ansible (e.g., install ansible on host machine)
    • mark submodules as deprecated
    • pip install compiled.txt
  • @openjck and/or @darkwing and/or others - spot-check ansible vm build

Tasks for deploy:
Anytime:

  • with Sheeri - Add utf8_distinct_ci collation to MySQL (bug 1161107)

May 26

  • With Jake and cyliang: Create new python 2.7 virtualenv (bug 1148435)
  • With Jake and cyliang: Install dependencies from this branch's compiled.txt requirements file into the new virtualenv

June 2

June 2-4

  • Stage Test, test, test with writers, WebQA, devs, et al.
  • Fix Travis builds on branch

June 9

  • Merge branch into master

June 9

Tasks for post-deploy:

  • @jezdez - Clean out db tables for removed apps and/or libraries, including stale content type records
    • @jezdez - Audit tables and list tables for deletion; ping Sheeri and delete
  • @groovecoder - notify dev-events team that MDN events page is gone
  • @groovecoder - notify mdn lists that features are gone: demo comments
  • @jezdez - send email to dev-mdn re: big dev environment changes (e.g., new vendor scheme and pip requirements usage)
  • @groovecoder - schedule mdn-dev (+ Cloud Ops?) post-mortem on the update project, including:
    • Policy for on-going updates to MDN python dependencies to prevent another huge gap
      • requires.io
    • Discuss pip-ifying dependencies, peep-like checks, installing from tarballs, adding pip install compiled to chief script
    • Resuming django-pipeline (and removing GitPython in the process)
    • Discuss a new tagging system
    • Discuss html5lib usage
    • Discuss PostgreSQL
    • Discuss regular db audit and maintenance
  • @groovecoder - file sec bug to update django-rest-framework
  • @jezdez - re-base #3062
@groovecoder

This comment has been minimized.

Member

groovecoder commented Feb 18, 2015

@ubernostrum - how's this branch going? Are you updating more libraries from vendor/src to 1.6-compatible versions?

@groovecoder

This comment has been minimized.

Member

groovecoder commented Feb 26, 2015

Ping @ubernostrum - 8 days without an update here. 😦

@jezdez

This comment has been minimized.

Member

jezdez commented Mar 3, 2015

Ok, I've found a series of issues that need to be addressed before moving along with this. That said, I think this looks great in general, lots of changes, but all worth it. We may end up in the present eventually!!

@ubernostrum

This comment has been minimized.

Contributor

ubernostrum commented Mar 5, 2015

I've cleaned up most of the submodule version issues (and even removed one!), but there are a few things that need further poking or decisions:

  • Decide on a safe version of django-rest-framework.
  • Try out mainline bleach instead of Les' fork.
  • Decide if we really do need httplib2.
  • Decide if we're going to get rid of grappelli.
  • Get me to do a proper PyPI release of soapbox.
@groovecoder

This comment has been minimized.

Member

groovecoder commented Mar 12, 2015

I investigated DRF and emailed the summary to dev-mdn (May also want to watch encode/django-rest-framework#2681 and encode/django-rest-framework#2684.)

I suggest we update to 2.3.14 immediately, and probably 2.4.4 soon after.

@groovecoder

This comment has been minimized.

Member

groovecoder commented Mar 12, 2015

Also, mdn-dev and devdoc-writers confirm we can remove grappelli.

@groovecoder

This comment has been minimized.

Member

groovecoder commented Mar 18, 2015

#3109 is now merged, so this could rebase on master and then it's ready to run tests?

ubernostrum added some commits Jan 23, 2015

Bug 1116337: Clear Django 1.5 lint output.
WARNING: This commit breaks compatibility with Django 1.4.

Many of the warnings from the linter are actually spurious/overzealous
reminders about checking the existence of cleaned_data (bad), where we
actually were checking for the presence of specific elements in
cleaned_data (good), but which django-compat-lint can't handle. As
such they were left as-is, and only the actually-relevant changes
(which are fewer in number) have been made.
Bug 1116340: Clear Django 1.6 lint output.
This clears all the non-spurious warnings/errors from the lint output;
the primary change here is transaction handling, with the changes
falling into three main areas:

1. Replacing old transaction decorators with the new 'atomic' one,
   since it provides the behavior we wanted.

2. Removing use of 'transaction.autocommit', since autocommit is now
   the default behavior of Django.

3. Rewriting the page-move task -- which is probably the trickiest
   transaction-related code -- to temporarily disable autocommit,
   manually manage the page-move transaction, and turn autocommit back
   on afterward.
@groovecoder

This comment has been minimized.

Member

groovecoder commented Jun 1, 2015

I incorporated the fix for amqp connection in the vm. I ran the new intern client-side test suite against this branch and fixed a bug I found in the newsfeed template. (I also sent a PR to improve the intern test suite messages.)

Only thing I know we have to do now is fix Travis CI. I'll work on that tomorrow.

@jezdez

This comment has been minimized.

Member

jezdez commented on kuma/feeder/management/commands/update_feeds.py in 3c95d5e Jun 2, 2015

If it's a logger this should be log.debug("Feed %s changed, updating db", feed)

@groovecoder

This comment has been minimized.

Member

groovecoder commented Jun 3, 2015

I'm using #3232 against this branch to let me test Travis builds without committing here (yet).

I couldn't find a path to follow with running ansible connections to Travis CI localhost, so I started a python script that re-uses those same Jinja2 templates as ansible to install the Index.xml file into MySQL.

@jezdez

This comment has been minimized.

Member

jezdez commented on 2d8b7d7 Jun 4, 2015

Ough, what a mess

@jezdez

This comment has been minimized.

Member

jezdez commented on requirements/packages.txt in 0836005 Jun 4, 2015

This is a no-go. We're not to add heavily out-dated versions. Please fix.

This comment has been minimized.

Member

groovecoder replied Jun 4, 2015

I'll work on an update. This was to make the stage push possible.

This comment has been minimized.

Member

groovecoder replied Jun 4, 2015

Every release after 0.1.7 breaks jingo-minify during compress_assets. So I propose we defer this and remove GitPython altogether when we change from jingo-minify to django-pipeline.

This comment has been minimized.

Contributor

ubernostrum replied Jun 5, 2015

I've submitted a pull request to jingo-minify to remove the GitPython dependency entirely.

This comment has been minimized.

Member

groovecoder replied Jun 8, 2015

Note: @jezdez decided to keep the old GitPython in to keep jingo-minify working without waiting on a new release. We'll remove GitPython completely when we remove jingo-minify and switch to django-pipeline.

This comment has been minimized.

Member

groovecoder replied Jun 8, 2015

@jezdez and I decided ...

@ubernostrum

This comment has been minimized.

Contributor

ubernostrum commented Jun 5, 2015

When trying to follow vagrant install instructions, getting this at the submodule init step:

$ git submodule update --init --recursive
Submodule 'testdata' (https://github.com/html5lib/html5lib-tests.git) registered for path 'html5lib/tests/testdata'
Cloning into 'html5lib/tests/testdata'...
remote: Counting objects: 3953, done.
remote: Total 3953 (delta 0), reused 0 (delta 0), pack-reused 3953
Receiving objects: 100% (3953/3953), 4.91 MiB | 2.74 MiB/s, done.
Resolving deltas: 100% (2028/2028), done.
Checking connectivity... done.
Submodule path 'vendor/src/html5lib/html5lib/tests/testdata': checked out 'f6a1b202de14fc057b196044c5ebef4672be3dd0'
fatal: Needed a single revision
Unable to find current revision in submodule path 'vendor/src/urlobject'

Not sure if that's an actual issue with submodules, or with the wifi I'm on (which has been messing with connections all week).

@groovecoder

This comment has been minimized.

Member

groovecoder commented Jun 5, 2015

I destroyed and re-built my vm last night and had no issue. Probably wifi connection.

@jezdez

This comment has been minimized.

Member

jezdez commented Jun 9, 2015

2924329 is wrong. settings_test.py is committed to the repo (until we merge the settings-refactor), so overwriting it with a different file from Ansible will result in an unclean git index (aka you have to commit it). Since we're deprecating non-Vagrant installations (at least from a supported maintenance perspective).

@jezdez

This comment has been minimized.

Member

jezdez commented Jun 9, 2015

woops, part of the sentence missing.

..I'm going to hardcode that path in the settings_test.py.

@jezdez

This comment has been minimized.

Member

jezdez commented Jun 9, 2015

So apparently there are problems with MySQL on Travis, travis-ci/travis-ci#2353 (reported by the author of the Ansible roles we're using for MySQL). So I'm going to uninstall the Travis provided MySQL and install the community server from MySQL itself (https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/index.html). Because I haven't explored a rabbit cave in recent weeks

@jezdez

This comment has been minimized.

Member

jezdez commented Jun 9, 2015

Ran 625 tests in 214.740s

OK (SKIP=10)
@groovecoder

This comment has been minimized.

Member

groovecoder commented Jun 9, 2015

Another merge from master.

Ran 625 tests in 162.653s

OK (SKIP=10)

@jezdez jezdez changed the title from (DO NOT MERGE) Django upgrade to Django upgrade Jun 9, 2015

@jezdez

This comment has been minimized.

Member

jezdez commented Jun 9, 2015

giphy

jezdez added a commit that referenced this pull request Jun 9, 2015

@jezdez jezdez merged commit ca49d1f into master Jun 9, 2015

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

@groovecoder groovecoder deleted the django-upgrade-1116335 branch Jun 10, 2015

@jezdez jezdez removed the not ready label Jun 15, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment