Getting 'null value in column \"image_manifest_id\" violates not-null constraint' when syncing openstack container repos

[ipanova]

Version
pulp-container 2.8.4

Describe the bug

 Failed to sync the openstack container repositories with the following error:
----------------------------------
  error:
    traceback: |2
        File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 342, in _perform_task
          result = func(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulp_container/app/tasks/synchronize.py", line 44, in synchronize
          return dv.create()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 151, in create
          loop.run_until_complete(pipeline)
        File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
          return future.result()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
          await asyncio.gather(*futures)
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
          await self.run()
        File "/usr/lib/python3.6/site-packages/pulp_container/app/tasks/sync_stages.py", line 457, in run
          objs=manifest_to_list_list, ignore_conflicts=True, batch_size=1000
        File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
          return getattr(self.get_queryset(), name)(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 474, in bulk_create
          ids = self._batched_insert(objs_without_pk, fields, batch_size, ignore_conflicts=ignore_conflicts)
        File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1211, in _batched_insert
          self._insert(item, fields=fields, using=self.db, ignore_conflicts=ignore_conflicts)
        File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
          return query.get_compiler(using=using).execute_sql(return_id)
        File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
          cursor.execute(sql, params)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
          return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
          return executor(sql, params, many, context)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
          return self.cursor.execute(sql, params)
        File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
          raise dj_exc_value.with_traceback(traceback) from exc_value
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
          return self.cursor.execute(sql, params)
    description: |
      null value in column "image_manifest_id" violates not-null constraint
      DETAIL:  Failing row contains (105527, amd64, linux, , , , , null, d55659df-811f-452e-a9a1-cc07205daa9b).
  worker: "/pulp/api/v3/workers/275806b5-c56b-42e8-b170-09bce2b290e9/"

To Reproduce
Failed to sync the following but not limited to the below RHOSP repositories.

openstack-cinder-api
openstack-neutron-server
openstack-neutron-dhcp-agent
openstack-neutron-server

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Looking at the hotfix PR (#568). It seems like we also need a "await list_dc.resolution()"
after line 121 (https://github.com/pulp/pulp_container/pull/535/files#diff-b702cbd3ebdff19bf8e4dfc14688aa3fb36e1a47ef6cf84d24e0f75b61247d36R121)
to make sure "list_dc" is resolved before man_dc.

sync_stages.py has been refactored in upstream Pulp "#608" which fixes the issue
https://bugzilla.redhat.com/show_bug.cgi?id=2154734

[ipanova]

This issue persists until the refactor of sync stages which landed in 2.12

[ipanova]

Consider backports to:
python3-pulp-container-2.8.4-0.1.el7pc.noarch
python3-pulp-container-2.9.2-1.el7pc.noarch
python39-pulp-container-2.10.9-1.el8pc.noarch

[pulpbot]

https://bugzilla.redhat.com/show_bug.cgi?id=2174984