Upgradee v3.7.7->v4.0.1/v4.0.2 produces DB key error, unsure how to correct

[BloodyIron]

Deployment Type

Self-hosted

NetBox Version

v4.0.2

Python Version

3.11

Steps to Reproduce

I ran the ./upgrade.sh as netbox user, and the error happened at step:

Removing stale content types (python3 netbox/manage.py remove_stale_contenttypes --no-input)...

And the error that was spat out:

Removing stale content types (python3 netbox/manage.py remove_stale_contenttypes --no-input)...
Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 299, in _commit
    return self.connection.commit()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/psycopg/connection.py", line 896, in commit
    self.wait(self._commit_gen())
  File "/opt/netbox/venv/lib/python3.11/site-packages/psycopg/connection.py", line 969, in wait
    return waiting.wait(gen, self.pgconn.socket, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "psycopg_c/_psycopg/waiting.pyx", line 205, in psycopg_c._psycopg.wait_c
  File "/opt/netbox/venv/lib/python3.11/site-packages/psycopg/connection.py", line 538, in _commit_gen
    yield from self._exec_command(b"COMMIT")
  File "/opt/netbox/venv/lib/python3.11/site-packages/psycopg/connection.py", line 472, in _exec_command
    raise e.error_from_result(result, encoding=pgconn_encoding(self.pgconn))
psycopg.errors.ForeignKeyViolation: update or delete on table "django_content_type" violates foreign key constraint "django_admin_log_content_type_id_c4bce8eb_fk_django_co" on table "django_admin_log"
DETAIL:  Key (id)=(80) is still referenced from table "django_admin_log".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/netbox/netbox/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/contrib/contenttypes/management/commands/remove_stale_contenttypes.py", line 101, in handle
    ct.delete()
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/base.py", line 1183, in delete
    return collector.delete()
           ^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/deletion.py", line 459, in delete
    with transaction.atomic(using=self.using, savepoint=False):
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/transaction.py", line 263, in __exit__
    connection.commit()
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 323, in commit
    self._commit()
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 298, in _commit
    with debug_transaction(self, "COMMIT"), self.wrap_database_errors:
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/backends/base/base.py", line 299, in _commit
    return self.connection.commit()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/psycopg/connection.py", line 896, in commit
    self.wait(self._commit_gen())
  File "/opt/netbox/venv/lib/python3.11/site-packages/psycopg/connection.py", line 969, in wait
    return waiting.wait(gen, self.pgconn.socket, timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "psycopg_c/_psycopg/waiting.pyx", line 205, in psycopg_c._psycopg.wait_c
  File "/opt/netbox/venv/lib/python3.11/site-packages/psycopg/connection.py", line 538, in _commit_gen
    yield from self._exec_command(b"COMMIT")
  File "/opt/netbox/venv/lib/python3.11/site-packages/psycopg/connection.py", line 472, in _exec_command
    raise e.error_from_result(result, encoding=pgconn_encoding(self.pgconn))
django.db.utils.IntegrityError: update or delete on table "django_content_type" violates foreign key constraint "django_admin_log_content_type_id_c4bce8eb_fk_django_co" on table "django_admin_log"
DETAIL:  Key (id)=(80) is still referenced from table "django_admin_log".

Expected Behavior

I expected the upgrade to not throw an error.

Observed Behavior

Well it threw an error.

[BloodyIron]

To add some context, this is a system I inherited, I did not set it up. It has been set up to weekly pull from the repo, install django-auth-ldap, requirements.txt, then run migrate, and run collectstatic.

Suddenly with v4.0.x coming out, this broke, naturally.

I have been following the documentation, and I am unsure what is going on with this log error. I am about to try starting the application anyways, as I don't know if this error is a blocking error and on my end I need to bring this application back online.

I have not been able to find ANY resources on the internet on appropriate next steps for this. So I know I'm taking a risk turning this on and moving on anyways, but I have a hunch this probably needs to be fixed (by me running some commands), but I genuinely cannot see how to help myself in this scenario. Please help! :(

[jeffgdotorg]

Hi, thanks for reporting a problem you're having with NetBox.

From your description and context, it seems very likely that some (no longer valid) assumption baked into your maintenance scripts, rather than a bug in NetBox itself, is the proximate cause of the message you're seeing. I'm therefore converting this issue to a discussion for exposure to a wider audience with operatinoal troubleshooting skills. If it turns out to be down to a bug after all, we'll handle it as one.

[BloodyIron]

What a horribly reasonable thing to do! ;P Thanks for your engagement.

[KrisLowet]

I had the same problem when upgrading from 3.7.8 to 4.0.2.
Debian 12, Python 3.11.
I fixed it by installing python3-psycopg2 via apt.

[candlerb]

You should not have to install any python packages via apt or via pip globally. Netbox will install all required packages into its own virtualenv, and won't use any packages outside of it.

If you needed to install packages outside, it means an error in the way you're running netbox (check path to ExecStart in netbox.service)

[BloodyIron]

I would like to add that it is preferable from a SysAdmin/ITSEC perspective to have any/all packages installed via the package manager (apt) and not PIP. I know that currently the requirements.txt is one core method relied upon for this tool, but the problem with that is that PIP packages are non-visible to Linux package managers. This means that said PIP-installed packages can generally be expected to either never be updated, or only updated in very-low frequency regards. This causes ITSEC problems (as updating software is typically the first thing anyone should do to keep systems secure), as well as a lack of visibility from any central-management of systems and their installed packages.

So if there's any possibility to move more "stuff" out of requirements.txt and into the responsibility of the package manager of the OS, then that would be preferable in my professional experience.

That being said, I do agree with the premise that this particular package "shouldn't necessarily be required to fix this" in concept. But in-practice... I dunno 🤷