django.db.utils.IntegrityError when upgrading to pulpcore 3.22

[daviddavis]

Version
pulpcore 3.22.0

Describe the bug
When upgrading to pulpcore 3.22, the pulp_labels migration fails with:

django.db.utils.IntegrityError: null value in column "pulp_labels" of relation "core_repository" violates not-null constraint
DETAIL:  Failing row contains (bbb38cf3-c632-4d51-a91f-94b23c7d3544, 2023-01-10 14:06:50.628502+00, 2023-01-10 14:06:50.632373+00, test, null, 1, file.file, null, null, f, null).

To Reproduce

1. In pulpcore < 3.22, create 2 repos: one with labels, one without.
2. Upgrade to pulpcore 3.22

Expected behavior
No migration failure.

[daviddavis]

Full stacktrace below.

I believe the issue is that the subquery in this line will return null if there's no labels for a model which violates the non-null constraint on pulp_labels:

3d67cc1#diff-1a15d2d8642c07f47172cac10d8c48b6a9229b6081510f66f9e32ba95659bf5bR411

psycopg2.errors.NotNullViolation: null value in column "pulp_labels" of relation "core_repository" violates not-null constraint
DETAIL:  Failing row contains (bbb38cf3-c632-4d51-a91f-94b23c7d3544, 2023-01-10 14:06:50.628502+00, 2023-01-10 14:06:50.632373+00, tester2, null, 1, f
ile.file, null, null, f, null).


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/pulp/bin/pulpcore-manager", line 33, in <module>
    sys.exit(load_entry_point('pulpcore', 'console_scripts', 'pulpcore-manager')())
  File "/home/vagrant/devel/pulpcore/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/core/management/commands/migrate.py", line 268, in handle
    emit_post_migrate_signal(
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/core/management/sql.py", line 42, in emit_post_migrate_signal
    models.signals.post_migrate.send(
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/dispatch/dispatcher.py", line 180, in send
    return [
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
    (receiver, receiver(signal=self, sender=sender, **named))
  File "/home/vagrant/devel/pulpcore/pulpcore/app/apps.py", line 411, in _migrate_remaining_labels
    model.objects.update(pulp_labels=label_subq)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/models/query.py", line 783, in update
    rows = query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/models/sql/compiler.py", line 1567, in execute_sql
    aux_rows = query.get_compiler(self.using).execute_sql(result_type)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/pulp/lib64/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: null value in column "pulp_labels" of relation "core_repository" violates not-null constraint
DETAIL:  Failing row contains (bbb38cf3-c632-4d51-a91f-94b23c7d3544, 2023-01-10 14:06:50.628502+00, 2023-01-10 14:06:50.632373+00, tester2, null, 1, f
ile.file, null, null, f, null).

[mdellweg]

Thank you for finding this!
It's probably better to limit the update to all objects that have actual labels assigned.