Upgrade

[huynhsontung]

Change log

• Dockerfile no longer create log folder or run the server, docker-compose will handle that
• Python 3.7
• No more base app, base's urls is now in history.historyofcg.urls
• Tuples in settings are converted to list
• All South's migrations were removed due to incompatibility with Django's built-in migration. As part new migration system, fixtures are converted to a dedicated migration - 0002_initial_data
• No more implicit context passing in views, all contexts are explicitly defined
• No need to explicitly pass request as context to template, template middleware will handle that
• Deleted render_to decorator as it uses render_to_response which is deprecated
• No more password_change view override
• manage.py no longer requires explicitly setting --settings and will use settings.development by default
• Many other backward incompatible and deprecation changes

Package removed/replaced

• setuptools
• django-ses
• static
• django-toolbelt
• wsgiref
• django-discover-runner
• virtualenv
• Pillow
• simplejson (replaced with json)
• dj-static (replace with whitenoise)
• boto (replaced with boto3)
• South (replaced with Django built-in migration)

[Gowiem]

@huynhsontung Hey man, I pushed to staging: https://histcg-staging.herokuapp.com/

Got any idea what that is about? I tried running migrations again using heroku run --app=histcg-staging python history/manage.py migrate --fake-initial but that doesn't seem to have done the trick. What're we missing there? Let me know and we'll get this wrapped up. Thanks!

[huynhsontung]

Hey @Gowiem, it's weird that I don't have this problem locally. This has nothing to do with migrations and just Django fails to import a module. I have updated the python path to this module. Can you try deploying again?

[Gowiem]

@huynhsontung Looking solid now. I just did a quick test and I'm liking it. I'm going to copy of the Prod DB to stage and have the site owner give it a run through and then we'll get this merged. Thanks for the hard work!

[Gowiem]

@huynhsontung I copied over the prod data to stage and ran migrate w/ fake-initial, but ran into the following:

Running python history/manage.py migrate --fake-initial on ⬢ histcg-staging... up, run.2425 (Free)
*** MANAGE PY MAIN - START ***
*** MANAGE PY MAIN - BEFORE READ_ENV ***
*** MANAGE PY MAIN - AFTER READ_ENV ***
*** RUNNING STAGING SETTINGS ***
System check identified some issues:

WARNINGS:
historyofcg.Page.connections: (fields.W340) null has no effect on ManyToManyField.
Operations to perform:
  Apply all migrations: admin, auth, contenttypes, historyofcg, sessions, sites
Running migrations:
  Applying historyofcg.0002_initial_data...Traceback (most recent call last):
  File "history/manage.py", line 44, in <module>
    execute_from_command_line(sys.argv)
  File "/app/.heroku/python/lib/python3.7/site-packages/django/core/management/__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "/app/.heroku/python/lib/python3.7/site-packages/django/core/management/__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/app/.heroku/python/lib/python3.7/site-packages/django/core/management/base.py", line 328, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/app/.heroku/python/lib/python3.7/site-packages/django/core/management/base.py", line 369, in execute
    output = self.handle(*args, **options)
  File "/app/.heroku/python/lib/python3.7/site-packages/django/core/management/base.py", line 83, in wrapped
    res = handle_func(*args, **kwargs)
  File "/app/.heroku/python/lib/python3.7/site-packages/django/core/management/commands/migrate.py", line 233, in handle
    fake_initial=fake_initial,
  File "/app/.heroku/python/lib/python3.7/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/app/.heroku/python/lib/python3.7/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/app/.heroku/python/lib/python3.7/site-packages/django/db/migrations/executor.py", line 246, in apply_migration
    self.record_migration(migration)
  File "/app/.heroku/python/lib/python3.7/site-packages/django/db/migrations/executor.py", line 261, in record_migration
    self.recorder.record_applied(migration.app_label, migration.name)
  File "/app/.heroku/python/lib/python3.7/site-packages/django/db/migrations/recorder.py", line 85, in record_applied
    self.ensure_schema()
  File "/app/.heroku/python/lib/python3.7/site-packages/django/db/migrations/recorder.py", line 62, in ensure_schema
    if self.has_table():
  File "/app/.heroku/python/lib/python3.7/site-packages/django/db/migrations/recorder.py", line 56, in has_table
    return self.Migration._meta.db_table in self.connection.introspection.table_names(self.connection.cursor())
  File "/app/.heroku/python/lib/python3.7/site-packages/django/db/backends/base/introspection.py", line 48, in table_names
    return get_names(cursor)
  File "/app/.heroku/python/lib/python3.7/site-packages/django/db/backends/base/introspection.py", line 43, in get_names
    return sorted(ti.name for ti in self.get_table_list(cursor)
  File "/app/.heroku/python/lib/python3.7/site-packages/django/db/backends/postgresql/introspection.py", line 54, in get_table_list
    """.format('c.relispartition' if self.connection.features.supports_table_partitions else 'FALSE'))
  File "/app/.heroku/python/lib/python3.7/site-packages/django/db/backends/utils.py", line 100, in execute
    return super().execute(sql, params)
  File "/app/.heroku/python/lib/python3.7/site-packages/django/db/backends/utils.py", line 68, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/app/.heroku/python/lib/python3.7/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/app/.heroku/python/lib/python3.7/site-packages/django/db/backends/utils.py", line 80, in _execute
    self.db.validate_no_broken_transaction()
  File "/app/.heroku/python/lib/python3.7/site-packages/django/db/backends/base/base.py", line 449, in validate_no_broken_transaction
    "An error occurred in the current transaction. You can't "
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

Seems the site data migrated and is working correctly however... Can you give me some insight into this error and what we need to do to properly properly migrate production when we get to that point?

[huynhsontung]

@Gowiem Does it happen every time you run migrate (without --fake-initial)?

[Gowiem]

@huynhsontung Yeah, I ran with --fake-initial twice and happened on both runs. Now I just ran without --fake-initial and it happened on that command as well.

[huynhsontung]

@Gowiem I found the problem and fixed it. Can you try again?

[Gowiem]

@huynhsontung That looks to have done the trick -- Thanks for the quick turnaround!

[Gowiem]

@huynhsontung While I'm thinking about it, can you do a README update as part of this work? Clear the old README cruft away and start fresh. I'd just like you to provide the high-level docker / compose requirements and workflow that you've been using to accomplish these upgrades so that is up-to-date for any future work.

[huynhsontung]

@Gowiem Sure I can take a look at README later tonight

[Gowiem]

@huynhsontung README looks great -- Thanks for the update!

[Gowiem]

@huynhsontung Getting the following error when signing up a new user on Stage. I imagine resetting passwords is broken as well..

Have you seen this? Is this something to do with misconfigured code or the email service we're using? Give me your thoughts and lets get this fixed up - Thanks man.

[huynhsontung]

@Gowiem Yes this is because the registration backend needs to send activation mail. I believe you don't have mail password for noreply@historyofcg.com set on staging?

https://github.com/huynhsontung/history_of_cg/blob/95eb0bb00a9b3938d22ccaac57445f2ce62debdf/history/settings/default.py#L47

Moreover, the email template is hard-coded to production URL so even when you get the email server working, it will not be able to confirm an account on staging and that account will stay inactive (unable to log in).

So you resolve this by first setting EMAIL_HOST_PASSWORD then either manually activate the account in admin dashboard or rewrite the email logic to support dynamic host address.

Hope that helps.

[Gowiem]

@huynhsontung Got it -- that makes sense. Thanks for the info!