Resolve social auth migration issue in production

[raymondberg]

We seem to be having the same problem in production as documented here: omab/python-social-auth#682

This our output when migrating:

Operations to perform:
  Synchronize unmigrated apps: django_gravatar, django_ical, rest_framework, main, honeypot, django_extensions, admin_tools, humanize, about, interval, django_forms_bootstrap, thumbnail, ckeditor, nocaptcha_recaptcha, theming, messages, django_bleach, storages, tinymce, gunicorn, staticfiles, envelope, contact
  Apply all migrations: profiles, flatpages, admin, sessions, announcements, menu, contenttypes, sponsors, dashboard, auth, sites, social_django, subgroups, flatblocks, meetings
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying meetings.0009_auto_20191231_1735... OK
  Applying meetings.0010_auto_20191231_1757... OK
  Applying profiles.0002_auto_20191231_1735... OK
  Applying social_django.0001_initial...Traceback (most recent call last):
  File "/home/app/.local/lib/python3.6/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
psycopg2.ProgrammingError: relation "social_auth_association" already exists


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/home/app/.local/lib/python3.6/site-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/home/app/.local/lib/python3.6/site-packages/django/core/management/__init__.py", line 346, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/app/.local/lib/python3.6/site-packages/django/core/management/base.py", line 394, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/app/.local/lib/python3.6/site-packages/django/core/management/base.py", line 445, in execute
    output = self.handle(*args, **options)
  File "/home/app/.local/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 222, in handle
    executor.migrate(targets, plan, fake=fake, fake_initial=fake_initial)
  File "/home/app/.local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 110, in migrate
    self.apply_migration(states[migration], migration, fake=fake, fake_initial=fake_initial)
  File "/home/app/.local/lib/python3.6/site-packages/django/db/migrations/executor.py", line 148, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/home/app/.local/lib/python3.6/site-packages/django/db/migrations/migration.py", line 115, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/home/app/.local/lib/python3.6/site-packages/django/db/migrations/operations/models.py", line 59, in database_forwards
    schema_editor.create_model(model)
  File "/home/app/.local/lib/python3.6/site-packages/django/db/backends/base/schema.py", line 286, in create_model
    self.execute(sql, params or None)
  File "/home/app/.local/lib/python3.6/site-packages/django/db/backends/base/schema.py", line 111, in execute
    cursor.execute(sql, params)
  File "/home/app/.local/lib/python3.6/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/app/.local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/app/.local/lib/python3.6/site-packages/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/app/.local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/app/.local/lib/python3.6/site-packages/django/db/backends/utils.py", line 62, in execute
    return self.cursor.execute(sql)
django.db.utils.ProgrammingError: relation "social_auth_association" already exists

[raymondberg]

For now, the workaround is to apply migrations from apps individually in production.

[elmq0022]

Any chance of just "faking" the migration that fails? The fields are there, so no need to create them.

https://blog.dbrgn.ch/2014/10/30/migrating-django-social-auth/

[elmq0022]

I made some progress with this issue. We won't be able to "fake" the migration. The database and migrations are inconsistent.

Results after faking the first migration:

social_django
[X] 0001_initial (2 squashed migrations)
[ ] 0002_add_related_name (2 squashed migrations)
[ ] 0003_alter_email_max_length (2 squashed migrations)
[ ] 0004_auto_20160423_0400 (2 squashed migrations)
[ ] 0005_auto_20160727_2333 (1 squashed migrations)
[ ] 0006_partial
[ ] 0007_code_timestamp
[ ] 0008_partial_timestamp

Operations to perform:
  Apply all migrations: social_django
 Running migrations:
  Rendering model states... DONE
  Applying social_django.0002_add_related_name... OK
  Applying social_django.0003_alter_email_max_length...Traceback (most recent call last):
  File "/home/app/.local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
psycopg2.ProgrammingError: relation "social_auth_code" does not exist

The value for social_auth_code does not exist.

[elmq0022]

I have a preliminary solution 😃. There seems to be just one table missing from the initial migration.

Step 1 - fake the initial migration

python manage.py migrate --fake social_django 0001_initial

Step 2 - run the following SQL to create the missing table, constraints, and indexes

CREATE TABLE "social_auth_code" (
    "id" serial NOT NULL PRIMARY KEY, 
    "email" varchar(75) NOT NULL, 
    "code" varchar(32) NOT NULL, 
    "verified" boolean NOT NULL
);

ALTER TABLE "social_auth_code" ADD CONSTRAINT "social_auth_code_email_116e94140c869d48_uniq" UNIQUE ("email", "code");
CREATE INDEX "social_auth_code_c1336794" ON "social_auth_code" ("code");
CREATE INDEX "social_auth_code_code_6dfe12a8de3ca34c_like" ON "social_auth_code" ("code" varchar_pattern_ops);

ALTER TABLE "social_auth_usersocialauth" ADD CONSTRAINT "social_auth_usersocialauth_provider_30b1032e78c8ec2e_uniq" UNIQUE ("provider", "uid");
ALTER TABLE "social_auth_usersocialauth" ADD CONSTRAINT "social_auth_usersocial_user_id_64dcff6fa70c5823_fk_auth_user_id" FOREIGN KEY ("user_id") REFERENCES "auth_user" ("id") DEFERRABLE INITIALLY DEFERRED;

ALTER TABLE "social_auth_nonce" ADD CONSTRAINT "social_auth_nonce_server_url_6a92d04f9c043c6c_uniq" UNIQUE ("server_url", "timestamp", "salt");

Step 3 - migrate as usual

python manage.py migrate

I'll do some additional checks, but I think we're back in business. Hopefully, this won't mess up the existing data.

[elmq0022]

@raymondberg and @smandekar1,

I was able to create a couple new oauth accounts just fine this morning. Could one of you verify the steps above before we run it in production?

Also, I have a new pull request to add a make psql shell, #243. The command runs:

docker-compose exec db psql chipy chipy

You could use this psql shell to run the SQL on your dev machine. This sql was generated from:

python manage.py sqlmigrate social_django 0001_initial

and then trimmed back to the missing table.

It's extremely similar to omab/python-social-auth#682, but we have a slightly different set of SQL commands to run.

[smandekar1]

I was getting this after step 1. running off the chipy master. I tried migrating contenttypes alone and then step 1 worked.

  File "/home/app/.local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/home/app/.local/lib/python3.6/site-packages/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/app/.local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/app/.local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "django_content_type" does not exist

step 2 returned django.db.utils.ProgrammingError: relation "social_auth_partial" already exists

[elmq0022]

@smandekar1 you will need to start with a clean database and back on tag 1.12.1 prior to Django social auth to python social auth migration. Then mock the upgrade.

You could nuke the current volume, docker-compse volume rm chipy... And docker-compse image rm chipy... Then rebuild the environment.

Alternatively build out a new environment typing the commands in the make file but adding the -p option. Example docker-compse -p up - - build -d. -p let's you provide a name other than that of the parent directory.

Rebuilding a docker-compose env does not delete the data from the volume by design.

Step 2 is running SQL directly on postgresql and does not involve social auth partial. So the error you got looks odd to me.

[elmq0022]

Maybe we can test it on your machine Thursday night before running it on the production site.

[smandekar1]

Sounds good, we can test it on this. I did delete the database, but I didn't delete the image and start on 1.12.1

[raymondberg]

Truly awesome. Thanks @elmq0022 and @smandekar1

[raymondberg]

Forgot to say, this is what we wound up with in prod:

  No migrations to apply.