Updating Inventree to 0.12.8. fails at datatbase migration

[MatjazSi]

Deployment Method

• Installer
• Docker Development
• Docker Production
• Bare metal Development
• Bare metal Production
• Digital Ocean image
• Other (please provide a link Steps to Reproduce

Describe the problem*

Hi,
When trying to update Inventree to the latest version (0.12.8) I get an error in the database migration step (see log output section)

Steps to Reproduce

Followed instructions on this page

1. Stop server:
   sudo docker-compose down

2. Backup DB:
   sudo docker-compose run inventree-server invoke backup

3. Download new version:
   sudo docker-compose pull

Migtrate DB to new version:
sudo docker-compose run inventree-server invoke update

This is whwere error happens

Relevant log output

Running InvenTree database migrations...
========================================
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
No changes detected
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Operations to perform:
  Apply all migrations: InvenTree, account, admin, auth, authtoken, build, common, company, contenttypes, django_q, error_report, exchange, flags, label, order, otp_static, otp_totp, part, plugin, report, sites, socialaccount, stock, taggit, user_sessions, users
Running migrations:
  Applying InvenTree.0001_initial...Traceback (most recent call last):
  File "/root/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "django_migrations_pkey"
DETAIL:  Key (id)=(431) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inventree/InvenTree/manage.py", line 23, in <module>
    execute_from_command_line(sys.argv)
  File "/root/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/root/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/root/.local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/root/.local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/root/.local/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/root/.local/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/root/.local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/root/.local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/root/.local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 229, in apply_migration
    self.record_migration(migration)
  File "/root/.local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 244, in record_migration
    self.recorder.record_applied(migration.app_label, migration.name)
  File "/root/.local/lib/python3.9/site-packages/django/db/migrations/recorder.py", line 87, in record_applied
    self.migration_qs.create(app=app, name=name)
  File "/root/.local/lib/python3.9/site-packages/django/db/models/query.py", line 453, in create
    obj.save(force_insert=True, using=self.db)
  File "/root/.local/lib/python3.9/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/root/.local/lib/python3.9/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/root/.local/lib/python3.9/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/root/.local/lib/python3.9/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/root/.local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/root/.local/lib/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/root/.local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/root/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/root/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/root/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/root/.local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/root/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "django_migrations_pkey"
DETAIL:  Key (id)=(431) already exists.

ERROR: 1

[SchrodingersGat]

@MatjazSi have not seen this one before!

Can you provide a list of the migrations that have been run against your database:

> docker compose run inventree-server sh
...
$ cd InvenTree
$ python ./manage.py show migrations

[MatjazSi]

Hi @SchrodingersGat
Because I really needed my Inventree I managed to restore it to the old working version (0.9.0) using Digital Ocean backups and backed-up database.
I will however make a copy of this server and try to update the Inventree there. I will try the above command and will post the results here.

[SchrodingersGat]

@MatjazSi good to hear that you could restore from backup. Please do let us know when you have some further information, so we can fix whatever is going on here

[MatjazSi]

@SchrodingersGat I created a copy of my server on Digital Ocean, try to update it got the same error as above.
Then I executed the comands in your previous post and got this:

# python ./manage.py show migrations
Traceback (most recent call last):
  File "/root/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedTable: relation "label_buildlinelabel" does not exist
LINE 1: SELECT (1) AS "a" FROM "label_buildlinelabel" WHERE "label_b...
                               ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inventree/InvenTree/./manage.py", line 23, in <module>
    execute_from_command_line(sys.argv)
  File "/root/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/root/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 395, in execute
    django.setup()
  File "/root/.local/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/root/.local/lib/python3.9/site-packages/django/apps/registry.py", line 122, in populate
    app_config.ready()
  File "/home/inventree/InvenTree/label/apps.py", line 41, in ready
    self.create_labels()  # pragma: no cover
  File "/home/inventree/InvenTree/label/apps.py", line 109, in create_labels
    self.create_labels_category(
  File "/home/inventree/InvenTree/label/apps.py", line 144, in create_labels_category
    self.create_template_label(model, src_dir, ref_name, label)
  File "/home/inventree/InvenTree/label/apps.py", line 180, in create_template_label
    if model.objects.filter(label=filename).exists():
  File "/root/.local/lib/python3.9/site-packages/django/db/models/query.py", line 808, in exists
    return self.query.has_results(using=self.db)
  File "/root/.local/lib/python3.9/site-packages/django/db/models/sql/query.py", line 561, in has_results
    return compiler.has_results()
  File "/root/.local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1145, in has_results
    return bool(self.execute_sql(SINGLE))
  File "/root/.local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/root/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/root/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/root/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/root/.local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/root/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "label_buildlinelabel" does not exist
LINE 1: SELECT (1) AS "a" FROM "label_buildlinelabel" WHERE "label_b...
                               ^

# 

[SchrodingersGat]

@MatjazSi those secondary error message would indicate that the database tables have not been created at all - are you running an update on an existing database? Or perhaps the virtual environment is not set correctly?

Regarding the pkey errors, this link indicates that running the command again may result in a pass.

Try: invoke migrate a couple of times to see if you get different results.

This is certainly an odd one, would like to get to the bottom of it

[MatjazSi]

@SchrodingersGat Yes, I belive I am running the update on the existing database. How do I check if the virtual environment is correct?

When I try to run migrate multiple times, that doesn't help, all I get is that the record ID in the error gets increased:

DETAIL: Key (id)=(433) already exists.

[SchrodingersGat]

Because I really needed my Inventree I managed to restore it to the old working version (0.9.0)

@MatjazSi so you were updating from 0.9.0 to 0.12.8? There have been a lot of changes since then, I wonder if a staged update would work better?

1. Start at 0.9.0 (and ensure you backup a copy of your data)
2. Update to 0.10.0 and run invoke update
3. Update to 0.11.0 and run invoke update
4. Update to 0.12.0 and run invoke update
5. Update to 0.12.8 and run invoke update

If you can run these steps at least we can potentially focus in on where the migration issue is ocurring

[MatjazSi]

@SchrodingersGat Thanks for the advice. I tried to update to 0.10.0 But unfortunatly got a similar error then before:

Running InvenTree database migrations...
========================================
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[PLUGIN] Encountered an error with /root/.local/lib/python3.9/site-packages/django/apps/registry.py:
App registry isn't ready yet.
fatal: not a git repository (or any of the parent directories): .git
Error initializing plugin `InvenTreeCoreNotificationsPlugin`: duplicate key value violates unique constraint "plugin_pluginconfig_pkey"
DETAIL:  Key (id)=(1) already exists.

[PLUGIN] Encountered an error with /root/.local/lib/python3.9/site-packages/django/db/backends/utils.py:
duplicate key value violates unique constraint "plugin_pluginconfig_pkey"
DETAIL:  Key (id)=(1) already exists.

fatal: not a git repository (or any of the parent directories): .git
Error initializing plugin `InvenTreeCoreNotificationsPlugin`: duplicate key value violates unique constraint "plugin_pluginconfig_pkey"
DETAIL:  Key (id)=(2) already exists.

[PLUGIN] Encountered an error with /root/.local/lib/python3.9/site-packages/django/db/backends/utils.py:
duplicate key value violates unique constraint "plugin_pluginconfig_pkey"
DETAIL:  Key (id)=(2) already exists.

fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
Migrations for 'part':
  part/migrations/0095_alter_partpricing_currency.py
    - Alter field currency on partpricing
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
fatal: not a git repository (or any of the parent directories): .git
Operations to perform:
  Apply all migrations: account, admin, auth, authtoken, build, common, company, contenttypes, django_q, error_report, exchange, label, order, otp_static, otp_totp, part, plugin, report, sites, socialaccount, stock, user_sessions, users
Running migrations:
  Applying build.0037_build_priority...Traceback (most recent call last):
  File "/root/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "django_migrations_pkey"
DETAIL:  Key (id)=(432) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inventree/InvenTree/manage.py", line 23, in <module>
    execute_from_command_line(sys.argv)
  File "/root/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/root/.local/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/root/.local/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/root/.local/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/root/.local/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/root/.local/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/root/.local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/root/.local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/root/.local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 229, in apply_migration
    self.record_migration(migration)
  File "/root/.local/lib/python3.9/site-packages/django/db/migrations/executor.py", line 244, in record_migration
    self.recorder.record_applied(migration.app_label, migration.name)
  File "/root/.local/lib/python3.9/site-packages/django/db/migrations/recorder.py", line 87, in record_applied
    self.migration_qs.create(app=app, name=name)
  File "/root/.local/lib/python3.9/site-packages/django/db/models/query.py", line 453, in create
    obj.save(force_insert=True, using=self.db)
  File "/root/.local/lib/python3.9/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/root/.local/lib/python3.9/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/root/.local/lib/python3.9/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/root/.local/lib/python3.9/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/root/.local/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/root/.local/lib/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/root/.local/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/root/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/root/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/root/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/root/.local/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/root/.local/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "django_migrations_pkey"
DETAIL:  Key (id)=(432) already exists.

ERROR: 1

[SchrodingersGat]

@MatjazSi did you ever get anywhere with this one?

I have opened a PR which may address some of these issues, but I do not think it is the whole picture.

Some references which may be helpful if more work is needed:

• https://stackoverflow.com/questions/51071385/django-db-utils-integrityerror-duplicate-key-value-violates-unique-constraint
• https://stackoverflow.com/questions/45975694/django-integrityerror-duplicate-key-value-violates-unique-constraint-django-co

[MatjazSi]

@SchrodingersGat I am still not able to update to any version. I will try the fix in the first link, but I am quite a noob when it comes to Docker, so can you please tell me how to login to dDango shell in Docker?

[SchrodingersGat]

Try docker compose run inventree-server sh which will get you a shell inside the container

[Sundypha]

I had a similar experience. From 0.10.1 I was able to upgrade to 0.11.0. From 0.11.0 to 0.12.0 calling invoke upgrade from the sh I get a RuntimeError: can't start new thread. Running python ./manage.py show migrations I get the label_buildlinelabel error from #5804 (comment).
If you want to test a few things let me know I have no important data in it.

[SchrodingersGat]

calling invoke upgrade from the sh I get a RuntimeError: can't start new thread.

@Sundypha can you provide a full error log please?

[Sundypha]

Ran from docker-compose:

user@host:/usr/docker/composes/inventree$ sudo docker-compose run inventree-server invoke update
[+] Running 1/0
 ✔ Container inventree-db  Running                                                                                                       0.0s
Loading config file : /home/inventree/data/config.yaml
Installing required python packages from 'requirements.txt'
Traceback (most recent call last):
  File "/usr/local/bin/invoke", line 8, in <module>
    sys.exit(program.run())
  File "/usr/local/lib/python3.9/site-packages/invoke/program.py", line 398, in run
    self.execute()
  File "/usr/local/lib/python3.9/site-packages/invoke/program.py", line 583, in execute
    executor.execute(*self.tasks)
  File "/usr/local/lib/python3.9/site-packages/invoke/executor.py", line 140, in execute
    result = call.task(*args, **call.kwargs)
  File "/usr/local/lib/python3.9/site-packages/invoke/tasks.py", line 138, in __call__
    result = self.body(*args, **kwargs)
  File "/home/inventree/tasks.py", line 285, in update
    install(c)
  File "/usr/local/lib/python3.9/site-packages/invoke/tasks.py", line 138, in __call__
    result = self.body(*args, **kwargs)
  File "/home/inventree/tasks.py", line 128, in install
    c.run('pip3 install --upgrade pip')
  File "/usr/local/lib/python3.9/site-packages/invoke/context.py", line 104, in run
    return self._run(runner, command, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/invoke/context.py", line 113, in _run
    return runner.run(command, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/invoke/runners.py", line 395, in run
    return self._run_body(command, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/invoke/runners.py", line 449, in _run_body
    thread.start()
  File "/usr/local/lib/python3.9/threading.py", line 899, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

ulimit inside container (0.12.0):

root@6d9deea2707b:/home/inventree# ulimit
unlimited

OS: Ubuntu server 20.04.6 LTS
Docker version 20.10.7, build f0df350
Docker Compose version v2.17.3

[SchrodingersGat]

@Sundypha this seems like a different error, perhaps related to docker itself. Can you try updating docker maybe?

I am running:

docker 24.0.7
docker compose v2.23.3

[Sundypha]

@SchrodingersGat I can confirm. I migrated the data to a host with a newer docker runtime and there it worked flawlessly. Even migrating from v0.10.1 directly to v0.12.0. It's weird that the invoke error didn't pop up during a search together with docker. Since I had another docker container (not related to InvenTree) which also uses invoke had the same problem I figured it must be the docker runtime as you guessed right.

But this covers only the invoke update part of the error. The python ./manage.py show migrations and its label_buildlinelabel still is there when trying to go from v0.10.1 to v0.12.0 directly. But I think this is not an issue worth following up on when the regular path of upgrading with invoke update works.

[SchrodingersGat]

@Sundypha this "extra" bug should be addressed in #6062 :)

[MatjazSi]

@SchrodingersGat sorry for the late response, was pretty busy the last few weeks.

I followed your advice and tried to reset the id as suggested in this post: Link

Then I tried to update (directly to 0.13.0) and got this:

Python version 3.10.13 - /usr/local/bin/python3
No changes detected
Python version 3.10.13 - /usr/local/bin/python3
Operations to perform:
  Apply all migrations: InvenTree, account, admin, auth, build, common, company, contenttypes, django_q, error_report, exchange, flags, label, order, otp_static, otp_totp, part, plugin, report, sites, socialaccount, stock, taggit, user_sessions, users
Running migrations:
  Applying part.0109_auto_20230517_1048...Traceback (most recent call last):
  File "/root/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.InFailedSqlTransaction: current transaction is aborted, commands ignored until end of transaction block


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inventree/InvenTree/manage.py", line 23, in <module>
    execute_from_command_line(sys.argv)
  File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/root/.local/lib/python3.10/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/root/.local/lib/python3.10/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/root/.local/lib/python3.10/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/root/.local/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/root/.local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/root/.local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/root/.local/lib/python3.10/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/root/.local/lib/python3.10/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/root/.local/lib/python3.10/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/home/inventree/InvenTree/part/migrations/0109_auto_20230517_1048.py", line 31, in update_template_units
    for template in PartParameterTemplate.objects.all():
  File "/root/.local/lib/python3.10/site-packages/django/db/models/query.py", line 280, in __iter__
    self._fetch_all()
  File "/root/.local/lib/python3.10/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/root/.local/lib/python3.10/site-packages/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/root/.local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/root/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/root/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/root/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute
    with self.db.wrap_database_errors:
  File "/root/.local/lib/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/root/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.InternalError: current transaction is aborted, commands ignored until end of transaction block

root@test:/home/inventree#

[ChristianSchindler]

I have a similar problem from upgrade from 0.12.10 to -> 0.13.0

/home/inventree/InvenTree #  python ./manage.py show migrations
Python version 3.10.13 - /usr/local/bin/python
2024-01-08 22:19:30,427 INFO DB_ENGINE: django.db.backends.postgresql
2024-01-08 22:19:30,427 INFO DB_NAME: inventree
2024-01-08 22:19:30,427 INFO DB_HOST: postgres-pod
2024-01-08 22:19:30,432 INFO MEDIA_ROOT: '/home/inventree/data/media'
2024-01-08 22:19:30,432 INFO STATIC_ROOT: '/home/inventree/data/static'
Loading InvenTree plugins
Installing plugins from plugins file
Running pip command: /usr/local/bin/python -m pip install -r /home/inventree/data/plugins.txt
Plugin Registry: Reloading plugins - Force: True, Full: True, Collect: True
Collecting plugins
/root/.local/lib/python3.10/site-packages/allauth/exceptions.py:9: UserWarning: allauth.exceptions is deprecated, use allauth.core.exceptions
  warnings.warn("allauth.exceptions is deprecated, use allauth.core.exceptions")
Collected 10 plugins
Start unloading plugins
Loading InvenTree plugins
Finished unloading plugins
Loading plugins
Found 9 active plugins
activate_integration_schedule failed, database not ready
Plugin Registry: Loaded 9 plugins
Error: Could not find permission matching 'report.view_stocklocationreport'
Error: Could not find permission matching 'stock.view_stocklocationtype'
Error: Could not find permission matching 'report.change_stocklocationreport'
Error: Could not find permission matching 'stock.delete_stocklocationtype'
Error: Could not find permission matching 'stock.change_stocklocationtype'
Error: Could not find permission matching 'stock.add_stocklocationtype'
Error: Could not find permission matching 'stock.view_stocklocationtype'
Error: Could not find permission matching 'report.add_stocklocationreport'
Error: Could not find permission matching 'report.view_stocklocationreport'
Error: Could not find permission matching 'report.delete_stocklocationreport'
Failed to remove obsolete tasks - database not ready
Traceback (most recent call last):
  File "/root/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedColumn: column django_q_schedule.intended_date_kwarg does not exist
LINE 1: ..._schedule"."task", "django_q_schedule"."cluster", "django_q_...
                                                             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inventree/InvenTree/InvenTree/apps.py", line 79, in remove_obsolete_tasks
    Schedule.objects.filter(func__in=obsolete).delete()
  File "/root/.local/lib/python3.10/site-packages/django/db/models/query.py", line 745, in delete
    collector.collect(del_query)
  File "/root/.local/lib/python3.10/site-packages/django/db/models/deletion.py", line 243, in collect
    new_objs = self.add(objs, source, nullable,
  File "/root/.local/lib/python3.10/site-packages/django/db/models/deletion.py", line 107, in add
    if not objs:
  File "/root/.local/lib/python3.10/site-packages/django/db/models/query.py", line 284, in __bool__
    self._fetch_all()
  File "/root/.local/lib/python3.10/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/root/.local/lib/python3.10/site-packages/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/root/.local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/root/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/root/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/root/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute
    with self.db.wrap_database_errors:
  File "/root/.local/lib/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/root/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column django_q_schedule.intended_date_kwarg does not exist
LINE 1: ..._schedule"."task", "django_q_schedule"."cluster", "django_q_...
                                                             ^

Starting background tasks...
Traceback (most recent call last):
  File "/root/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UndefinedColumn: column django_q_schedule.intended_date_kwarg does not exist
LINE 1: ..._schedule"."task", "django_q_schedule"."cluster", "django_q_...
                                                             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/inventree/InvenTree/./manage.py", line 23, in <module>
    main()
  File "/home/inventree/InvenTree/./manage.py", line 19, in main
    execute_from_command_line(sys.argv)
  File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 395, in execute
    django.setup()
  File "/root/.local/lib/python3.10/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/root/.local/lib/python3.10/site-packages/django/apps/registry.py", line 122, in populate
    app_config.ready()
  File "/home/inventree/InvenTree/InvenTree/apps.py", line 48, in ready
    self.start_background_tasks()
  File "/home/inventree/InvenTree/InvenTree/apps.py", line 92, in start_background_tasks
    for existing_task in Schedule.objects.all():
  File "/root/.local/lib/python3.10/site-packages/django/db/models/query.py", line 280, in __iter__
    self._fetch_all()
  File "/root/.local/lib/python3.10/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/root/.local/lib/python3.10/site-packages/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/root/.local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/root/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/root/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/root/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute
    with self.db.wrap_database_errors:
  File "/root/.local/lib/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/root/.local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: column django_q_schedule.intended_date_kwarg does not exist
LINE 1: ..._schedule"."task", "django_q_schedule"."cluster", "django_q_...

[opajonk]

I ran into this yesterday for an upgrade from 0.12.10 --> 0.13.1 as well.

[SchrodingersGat]

@opajonk can you please share your full error trace also. We are getting different errors being reported here.

[SchrodingersGat]

@ChristianSchindler it looks like you have missed the invoke update step - what is the output of your invoke update command?

[ChristianSchindler]

@SchrodingersGat the update gives me the following error:

pg_dump: detail: server version: 16.0; pg_dump version: 15.5

  File "/root/.local/lib/python3.10/site-packages/dbbackup/utils.py", line 120, in wrapper
    func(*args, **kwargs)
  File "/root/.local/lib/python3.10/site-packages/dbbackup/management/commands/dbbackup.py", line 93, in handle
    self._save_new_backup(database)
  File "/root/.local/lib/python3.10/site-packages/dbbackup/management/commands/dbbackup.py", line 106, in _save_new_backup
    outputfile = self.connector.create_dump()
  File "/root/.local/lib/python3.10/site-packages/dbbackup/db/base.py", line 92, in create_dump
    return self._create_dump()
  File "/root/.local/lib/python3.10/site-packages/dbbackup/db/postgresql.py", line 112, in _create_dump
    stdout, stderr = self.run_command(cmd, env=self.dump_env)
  File "/root/.local/lib/python3.10/site-packages/dbbackup/db/base.py", line 171, in run_command
    raise exceptions.CommandConnectorError(

Traceback (most recent call last):
  File "/home/inventree/InvenTree/manage.py", line 23, in <module>
    execute_from_command_line(sys.argv)
  File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/root/.local/lib/python3.10/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/root/.local/lib/python3.10/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/root/.local/lib/python3.10/site-packages/dbbackup/utils.py", line 120, in wrapper
    func(*args, **kwargs)
  File "/root/.local/lib/python3.10/site-packages/dbbackup/management/commands/dbbackup.py", line 93, in handle
    self._save_new_backup(database)
  File "/root/.local/lib/python3.10/site-packages/dbbackup/management/commands/dbbackup.py", line 106, in _save_new_backup
    outputfile = self.connector.create_dump()
  File "/root/.local/lib/python3.10/site-packages/dbbackup/db/base.py", line 92, in create_dump
    return self._create_dump()
  File "/root/.local/lib/python3.10/site-packages/dbbackup/db/postgresql.py", line 112, in _create_dump
    stdout, stderr = self.run_command(cmd, env=self.dump_env)
  File "/root/.local/lib/python3.10/site-packages/dbbackup/db/base.py", line 171, in run_command
    raise exceptions.CommandConnectorError(
dbbackup.db.exceptions.CommandConnectorError: Error running:  pg_dump --dbname=postgresql://inventreeuser:****@postgres-pod:5432/inventree --format=custom
pg_dump: error: aborting because of server version mismatch
pg_dump: detail: server version: 16.0; pg_dump version: 15.5

But without a backup, it works. Maybe you could implement that you support more PostgreSQL versions because I use the db for several things.
And is it possible to automate the invoke update on every creation of the containers?

[opajonk]

@opajonk can you please share your full error trace also. We are getting different errors being reported here.

D'oh, nevermind. I missed the invoke update command.

To be honest, this happens quite easily - is it not somehow possible to "update if required" when the server container starts?

[SchrodingersGat]

I do not think that enforcing invoke update on container launch is a good idea. There are a lot of expensive operations which occur on an update, we should not be doing that every time. Also there are different setup methods apart from docker, we would have to then handle those too.

It is not too much of a burden to run a single command which handles the database update, especially when the admin has already made a decision to update.

[opajonk]

Well, of course some kind of "am I already on the current version?" check would be a good idea :-) but yeah, an invoke update is also OK. Just not what one would expect.

[matmair]

@opajonk superusers can access this in the version info screen already. In 0.14.0 there will also be an email to superusers if a new version is detected.

[opajonk]

@opajonk superusers can access this in the version info screen already. In 0.14.0 there will also be an email to superusers if a new version is detected.

I mean of course an automated check on container boot, checking for "is my persistency on the current version of the container that is being booted?". If not, run "invoke update". If yes, just skip that and boot the container immediately.

[ChristianSchindler]

I have created an issue for improving the docker deployment #6303

[ChristianSchindler]

pg_dump: detail: server version: 16.0; pg_dump version: 15.5

  File "/root/.local/lib/python3.10/site-packages/dbbackup/utils.py", line 120, in wrapper
    func(*args, **kwargs)
  File "/root/.local/lib/python3.10/site-packages/dbbackup/management/commands/dbbackup.py", line 93, in handle
    self._save_new_backup(database)
  File "/root/.local/lib/python3.10/site-packages/dbbackup/management/commands/dbbackup.py", line 106, in _save_new_backup
    outputfile = self.connector.create_dump()
  File "/root/.local/lib/python3.10/site-packages/dbbackup/db/base.py", line 92, in create_dump
    return self._create_dump()
  File "/root/.local/lib/python3.10/site-packages/dbbackup/db/postgresql.py", line 112, in _create_dump
    stdout, stderr = self.run_command(cmd, env=self.dump_env)
  File "/root/.local/lib/python3.10/site-packages/dbbackup/db/base.py", line 171, in run_command
    raise exceptions.CommandConnectorError(

Traceback (most recent call last):
  File "/home/inventree/InvenTree/manage.py", line 23, in <module>
    execute_from_command_line(sys.argv)
  File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/root/.local/lib/python3.10/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/root/.local/lib/python3.10/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/root/.local/lib/python3.10/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/root/.local/lib/python3.10/site-packages/dbbackup/utils.py", line 120, in wrapper
    func(*args, **kwargs)
  File "/root/.local/lib/python3.10/site-packages/dbbackup/management/commands/dbbackup.py", line 93, in handle
    self._save_new_backup(database)
  File "/root/.local/lib/python3.10/site-packages/dbbackup/management/commands/dbbackup.py", line 106, in _save_new_backup
    outputfile = self.connector.create_dump()
  File "/root/.local/lib/python3.10/site-packages/dbbackup/db/base.py", line 92, in create_dump
    return self._create_dump()
  File "/root/.local/lib/python3.10/site-packages/dbbackup/db/postgresql.py", line 112, in _create_dump
    stdout, stderr = self.run_command(cmd, env=self.dump_env)
  File "/root/.local/lib/python3.10/site-packages/dbbackup/db/base.py", line 171, in run_command
    raise exceptions.CommandConnectorError(
dbbackup.db.exceptions.CommandConnectorError: Error running:  pg_dump --dbname=postgresql://inventreeuser:****@postgres-pod:5432/inventree --format=custom
pg_dump: error: aborting because of server version mismatch
pg_dump: detail: server version: 16.0; pg_dump version: 15.5

But without a backup, it works. Maybe you could implement that you support more PostgreSQL versions because I use the db for several things. And is it possible to automate the invoke update on every creation of the containers?

@matmair @opajonk can we update pg_dump to 16, or is there something that would not work?

[SchrodingersGat]

@ChristianSchindler are you running a "standard" docker setup? We pin postgres to version 13 - so why are you seeing postgres 16?

[ChristianSchindler]

No not really. I use only one db for all my services and that’s at the moment postgrsql 16.

[matmair]

InvenTree will not work with Postgres 16 currently