SQL migration guide in upgrade docs is not clear

[clandmeter]

First of all, I dont have any experiance with django so sorry if I'm taking wrong assumptions.

I'm are running an older master checked out at 20150420 with Django v1.5 which I tried to update. I've read the documentation on how to upgrade but it seems not very clear coming from and older version than 1.0. Here are the steps I followed:

manage.py migrate

This will run ok until it will error on 0001_initial (some table already exists).
I converted this file to sql with the following cmd:

manage.py sqlmigrate patchwork 0001_initial

After looking at the generated SQL, it looks I'm only missing the part defined at 015-add-patch-tags.sql so i have manually imported it into my db.

Then looking at my db, I'm also missing all other parts from the following migration scripts, so I generated the missing SQL and imported it into my db.

This ends up in a working upgraded patchwork except runserver complained about some outstanding migration which i resolved by running:

manage.py migrate --fake

Now my question is, is this the correct procedure or did I miss/forget something?
And will I in the future be able to just run:

manage.py migrate

[stephenfin]

Looks like those docs need even more refinement alright. Unfortunately, how you went about this is not how I would do it, thus, I can't tell you whether your DB is consistent or not. I'll describe the process flow that I'd use, were I upgrading a pre-v1.0 instance. After that, I'd appreciate it if I could get a dump of your schema that I could compare to the "expected" version.

To upgrade pre-v1.0 to v1.1:

1. Identify the last SQL migration you applied. Apply any other migrations, found in lib/sql/migration, you might have missed. Once done, you'll no longer need to use this directory.
2. Fake apply the initial migration. This will be done automatically is using Django 1.7.x. If using Django 1.8+, you can do this with ./manage.py migrate 0001_initial --fake-initial.
3. Apply all other migrations. You can do this with ./manage.py migrate.

This step works assuming you haven't any out-of-tree changes to your schema. If you have, that's a different ballgame and will require a custom solution.

[clandmeter]

I'm trying to follow the steps you mention. I guess the migration information is stored in the database, and if i just revert the database to the previous state, the migration starts from zero.

as you suggested I run ./manage.py migrate 0001_initial --fake and get:

CommandError: App '0001_initial' does not have migrations (you cannot selectively sync unmigrated apps)

Then I remember i run ./manage.py migrate --fake-initial patchwork but the I get:

Operations to perform:
  Apply all migrations: patchwork
Running migrations:
  Rendering model states... DONE
  Applying contenttypes.0001_initial... FAKED
  Applying auth.0001_initial... FAKED
  Applying patchwork.0001_initial... FAKED
  Applying patchwork.0002_fix_patch_state_default_values... OK
  Applying patchwork.0003_add_check_model... OK
  Applying patchwork.0004_add_delegation_rule_model... OK
  Applying patchwork.0005_unselectable_maintainer_projects... OK
/usr/lib/python2.7/site-packages/django/db/backends/mysql/base.py:124: Warning: Field 'name' doesn't have a default value
  return self.cursor.execute(query, args)

please remember I was using django 1.5 and now 1.8. (1.5 doesn't have any migration support right?).

PS which version of the database schema would you like, the initial one or the one after upgrading?
PS2 I didn't do any "out-of-tree changes" its just a default install with no modifications whatsoever.

[stephenfin]

Yes, that should have read python manage.py migrate --fake-initial - my mistake. Corrected original comment.

Could you give me the schema of your DB before applying Django migrations, but after applying any SQL migrations? Looks like your schema is not what Django is expecting, thus there may be a need to manually fix it before applying the migrations.

Off the top of my head, you probably should migrate all tables (i.e. ./manage.py migrate --fake-initial without the patchwork argument). In addition, it might also be worth apply the migrations step by step to see which one in particular is failing (I guess 0006)?

[clandmeter]

db after applying the 015 patch (think this is the only missing one).

http://sprunge.us/dPYO

[clandmeter]

I don't see any 0006 patch, so i guess its coming from the 0005?
When i try to generate SQL from the 0005 migration script it outputs nothing.

ls /usr/share/webapps/patchwork/patchwork/migrations |grep -v pyc

0001_initial.py
0002_fix_patch_state_default_values.py
0003_add_check_model.py
0004_add_delegation_rule_model.py
0005_unselectable_maintainer_projects.py
__init__.py

How does one run the migrations step by step?

[stephenfin]

After comparing your schema to mine, I see no significant differences: some columns are bigger/smaller, while others are in different positions. This is to be expected though, seeing as mine were produced by Django migrate rather than SQL scripts.

Regarding this output:

/usr/lib/python2.7/site-packages/django/db/backends/mysql/base.py:124: Warning: Field 'name' doesn't have a default value
  return self.cursor.execute(query, args)

This seems to be a warning, rather than an error. Are you able to use the instance once done? If so, I think you can ignore this.

[clandmeter]

This seems to work now on my existing DB:

I am using MySQL so i need to fix the double quotes in the SQL migration files by adding this to the top of it:

SET SQL_MODE=ANSI_QUOTES;

Then import the missing sql files and run the migration as followed:

python manage.py migrate --fake-initial
Operations to perform:
  Synchronize unmigrated apps: staticfiles, humanize
  Apply all migrations: sessions, admin, sites, auth, patchwork, contenttypes
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying contenttypes.0001_initial... FAKED
  Applying auth.0001_initial... FAKED
  Applying admin.0001_initial... FAKED
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying patchwork.0001_initial... FAKED
  Applying patchwork.0002_fix_patch_state_default_values... OK
  Applying patchwork.0003_add_check_model... OK
  Applying patchwork.0004_add_delegation_rule_model... OK
  Applying patchwork.0005_unselectable_maintainer_projects... OK
  Applying sessions.0001_initial... FAKED
  Applying sites.0001_initial... FAKED

No further migrations seem necessary.

I think the error happends when you first run the migration differently, and want to run it afterwards with the fake switches.

[daxtens]

@stephenfin - this has been sitting around for a while and predates 2.0 - are there any doc or migration changes still needed?

[stephenfin]

Nope, I think we're good. I removed the manual migration information for the 2.1 documentation, so I think we can close this unless other issues are experienced.