Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New migrations failing on startup #96

Closed
Serneum opened this issue Dec 18, 2021 · 8 comments
Closed

New migrations failing on startup #96

Serneum opened this issue Dec 18, 2021 · 8 comments

Comments

@Serneum
Copy link

Serneum commented Dec 18, 2021

It looks like my Codex server has been down for a few days, likely around the time the new version was released. I checked the logs and found this:

2021-12-18 16:32:11 EST INFO     Loaded config from /config/hypercorn.toml
2021-12-18 16:32:12 EST WARNING  no such table: codex_comic_folders
No changes detected in app 'codex'
Operations to perform:
  Apply all migrations: admin, auth, codex, contenttypes, sessions
Running migrations:
Traceback (most recent call last):
  File "/usr/local/bin/codex", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/codex/run.py", line 67, in main
    update_db()
  File "/usr/local/lib/python3.10/site-packages/codex/run.py", line 34, in update_db
    call_command("migrate")
  File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 187, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 417, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 90, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 253, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 126, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 156, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 235, in apply_migration
    with self.connection.schema_editor(atomic=migration.atomic) as schema_editor:
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/sqlite3/schema.py", line 35, in __exit__
    self.connection.check_constraints()
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 346, in check_constraints
    raise IntegrityError(
django.db.utils.IntegrityError: The row in table 'codex_comic_characters' with primary key '15653' has an invalid foreign key: codex_comic_characters.comic_id contains a value '1674' that does not have a corresponding value in codex_comic.id.
  Applying codex.0006_update_default_names_and_remove_duplicate_comics...%

Is the best option here to just blow away my database and rebuild it? I had put in a lot of manual work to get some issues to be recognized properly and would not like to do that again, but I could. I also think that was just fixing the metadata of the issues themselves, so I suspect they'd import properly this time.

@ajslater
Copy link
Owner

ajslater commented Dec 18, 2021 via email

@Serneum
Copy link
Author

Serneum commented Dec 19, 2021

I did a docker-compose pull codex and got a new image. I got the same error, so I tried explicitly setting the tag to 0.7.3 instead of relying on latest just to be safe, did another pull, and then started up the app again. Same issue from the look of it

2021-12-18 21:06:31 EST INFO     Loaded config from /config/hypercorn.toml
2021-12-18 21:06:32 EST WARNING  no such table: codex_comic_folders
No changes detected in app 'codex'
Operations to perform:
  Apply all migrations: admin, auth, codex, contenttypes, sessions
Running migrations:
Traceback (most recent call last):
  File "/usr/local/bin/codex", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/codex/run.py", line 67, in main
    update_db()
  File "/usr/local/lib/python3.10/site-packages/codex/run.py", line 34, in update_db
    call_command("migrate")
  File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 187, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 417, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 90, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 253, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 126, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 156, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 235, in apply_migration
    with self.connection.schema_editor(atomic=migration.atomic) as schema_editor:
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/sqlite3/schema.py", line 35, in __exit__
    self.connection.check_constraints()
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 346, in check_constraints
    raise IntegrityError(
django.db.utils.IntegrityError: The row in table 'codex_comic_characters' with primary key '15653' has an invalid foreign key: codex_comic_characters.comic_id contains a value '1674' that does not have a corresponding value in codex_comic.id.
  Applying codex.0006_update_default_names_and_remove_duplicate_comics...%

I also tried with the CODEX_SKIP_INTEGRITY_CHECK and it looks the same for the most part, though it did print out that it was skipping the integrity check

2021-12-18 21:08:46 EST INFO     Loaded config from /config/hypercorn.toml
2021-12-18 21:08:47 EST INFO     Skipping integrity check
No changes detected in app 'codex'
Operations to perform:
  Apply all migrations: admin, auth, codex, contenttypes, sessions
Running migrations:
Traceback (most recent call last):
  File "/usr/local/bin/codex", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/codex/run.py", line 67, in main
    update_db()
  File "/usr/local/lib/python3.10/site-packages/codex/run.py", line 34, in update_db
    call_command("migrate")
  File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 187, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 417, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 90, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 253, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 126, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 156, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 235, in apply_migration
    with self.connection.schema_editor(atomic=migration.atomic) as schema_editor:
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/sqlite3/schema.py", line 35, in __exit__
    self.connection.check_constraints()
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 346, in check_constraints
    raise IntegrityError(
django.db.utils.IntegrityError: The row in table 'codex_comic_characters' with primary key '15653' has an invalid foreign key: codex_comic_characters.comic_id contains a value '1674' that does not have a corresponding value in codex_comic.id.
  Applying codex.0006_update_default_names_and_remove_duplicate_comics...%

@ajslater
Copy link
Owner

ajslater commented Dec 19, 2021

Please try Codex 0.7.4.
I've rewritten (and simplified) the section of the integrity checker that I believe was causing your crash to handle additional database corruptions.

I only have a limited number of corrupt databases to test with so if this version doesn't fix your issue, consider sending me your database itself to aj@slater.net so I may be assured of fixing your issues in the next release. If you don't feel like doing another log trace will also help.

As a practical matter to get up and running more quickly, if v0.7.4 doesn't do the trick you might also try the emergency database repair procedure documented in the Troubleshooting section of the README.

@Serneum
Copy link
Author

Serneum commented Dec 20, 2021

Here's my log with 0.7.4. I'll see what I can do about getting you a copy of my database

2021-12-19 22:18:20 EST INFO     Loaded config from /config/hypercorn.toml
2021-12-19 22:18:20 EST INFO     Skipping integrity check
No changes detected in app 'codex'
Operations to perform:
  Apply all migrations: admin, auth, codex, contenttypes, sessions
Running migrations:
Traceback (most recent call last):
  File "/usr/local/bin/codex", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/codex/run.py", line 67, in main
    update_db()
  File "/usr/local/lib/python3.10/site-packages/codex/run.py", line 34, in update_db
    call_command("migrate")
  File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 187, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 417, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 90, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 253, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 126, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 156, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 235, in apply_migration
    with self.connection.schema_editor(atomic=migration.atomic) as schema_editor:
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/sqlite3/schema.py", line 35, in __exit__
    self.connection.check_constraints()
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 346, in check_constraints
    raise IntegrityError(
django.db.utils.IntegrityError: The row in table 'codex_comic_characters' with primary key '15653' has an invalid foreign key: codex_comic_characters.comic_id contains a value '1674' that does not have a corresponding value in codex_comic.id.
  Applying codex.0006_update_default_names_and_remove_duplicate_comics...%

@Serneum
Copy link
Author

Serneum commented Dec 20, 2021

Just for good measure, I also tried the database rebuild:

2021-12-19 22:26:03 EST INFO     Loaded config from /config/hypercorn.toml
2021-12-19 22:26:03 EST VERBOSE  max_db_ops limit is 100000
2021-12-19 22:26:03 EST DEBUG    Parsed browserChoices
2021-12-19 22:26:03 EST DEBUG    Parsed readerChoices
2021-12-19 22:26:03 EST DEBUG    Parsed websocketMessages
2021-12-19 22:26:03 EST WARNING  REBUILDING DATABASE!!
2021-12-19 22:26:08 EST INFO     Backed up old db to /config/db.sqlite3.backup
2021-12-19 22:26:08 EST INFO     Rebuilt database.
2021-12-19 22:26:08 EST INFO     Skipping integrity check
2021-12-19 22:26:08 EST DEBUG    sqlite journal_mode=wal
No changes detected in app 'codex'
Operations to perform:
  Apply all migrations: admin, auth, codex, contenttypes, sessions
Running migrations:
Traceback (most recent call last):
  File "/usr/local/bin/codex", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/site-packages/codex/run.py", line 67, in main
    update_db()
  File "/usr/local/lib/python3.10/site-packages/codex/run.py", line 34, in update_db
    call_command("migrate")
  File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 187, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 417, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 90, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 253, in handle
    post_migrate_state = executor.migrate(
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 126, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 156, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 235, in apply_migration
    with self.connection.schema_editor(atomic=migration.atomic) as schema_editor:
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/sqlite3/schema.py", line 35, in __exit__
    self.connection.check_constraints()
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 346, in check_constraints
    raise IntegrityError(
django.db.utils.IntegrityError: The row in table 'codex_userbookmark' with primary key '518' has an invalid foreign key: codex_userbookmark.comic_id contains a value '4732' that does not have a corresponding value in codex_comic.id.
  Applying codex.0006_update_default_names_and_remove_duplicate_comics...usermod: no changes

@ajslater
Copy link
Owner

A fix is currently making its way through ci:
#98

@ajslater
Copy link
Owner

ajslater commented Dec 21, 2021

Release v0.7.5, now on docker hub and elsewhere, contains a fix for the specific issue you ran into. Give it a whirl. You will have to run the integrity checker normally for the migrations to succeed.

@Serneum
Copy link
Author

Serneum commented Dec 21, 2021

That looks like it did it. Thanks for all of your help

@Serneum Serneum closed this as completed Dec 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants