Glance Upload Internal Server Error

[runlevel-six]

I pulled down the Glance changes made yesterday (Thursday, September 8), but when I use the container image tag referenced (quay.io/vexxhost/glance:677c89c23631e9083261a1a18ed438d8966e0de2 ) I get a failure when attempting to upload a new OS image to Glance:

If I use the previous image (us-docker.pkg.dev/vexxhost-infra/openstack/glance:22.1.1.dev2-1), everything works fine.

I see this result whether I let Atmosphere attempt to deploy an image defined in my inventory or if I run a manual upload via the CLI:

openstack image create \
  --disk-format qcow2 \
  --container-format bare \
  --public \
  --file ~/Downloads/cirros-0.3.5-x86_64-disk.img \
  cirros

Here are the relevant logs:

❯ kubectl -n openstack logs --follow -l application=glance,component=api
2022-09-09 12:54:21.801 9 ERROR glance.api.v2.image_data [req-0b4745cd-261c-4616-87ca-38aea8bc7056 b19af6e18b774b1b9de6f05d65131d43 325b64f1784b41328921a79e292cbe9b - default default] Failed to upload image data due to internal error
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi [req-0b4745cd-261c-4616-87ca-38aea8bc7056 b19af6e18b774b1b9de6f05d65131d43 325b64f1784b41328921a79e292cbe9b - default default] Caught error: 'NoneType' object has no attribute 'Rados': AttributeError: 'NoneType' object has no attribute 'Rados'
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi Traceback (most recent call last):
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance/common/wsgi.py", line 1353, in __call__
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     action_result = self.dispatch(self.controller, action,
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance/common/wsgi.py", line 1397, in dispatch
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     return method(*args, **kwargs)
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance/common/utils.py", line 416, in wrapped
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     return func(self, req, *args, **kwargs)
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 300, in upload
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     self._restore(image_repo, image)
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     self.force_reraise()
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     raise self.value
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 165, in upload
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     image.set_data(data, size, backend=backend)
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance/domain/proxy.py", line 208, in set_data
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     self.base.set_data(data, size, backend=backend, set_active=set_active)
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance/notifier.py", line 501, in set_data
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     _send_notification(notify_error, 'image.upload', msg)
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     self.force_reraise()
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     raise self.value
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance/notifier.py", line 447, in set_data
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     self.repo.set_data(data, size, backend=backend,
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance/api/policy.py", line 273, in set_data
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     return self.image.set_data(*args, **kwargs)
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance/quota/__init__.py", line 322, in set_data
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     self.image.set_data(data, size=size, backend=backend,
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance/location.py", line 567, in set_data
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     self._upload_to_store(data, verifier, backend, size)
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance/location.py", line 473, in _upload_to_store
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     loc_meta) = self.store_api.add_to_backend_with_multihash(
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance_store/backend.py", line 490, in add_to_backend_with_multihash
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     return store_add_to_backend_with_multihash(
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance_store/backend.py", line 467, in store_add_to_backend_with_multihash
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     (location, size, checksum, multihash, metadata) = store.add(
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance_store/driver.py", line 279, in add_adapter
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     metadata_dict) = store_add_fun(*args, **kwargs)
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance_store/capabilities.py", line 176, in op_checker
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     return store_op_fun(store, *args, **kwargs)
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance_store/_drivers/rbd.py", line 536, in add
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     with self.get_connection(conffile=self.conf_file,
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     return next(self.gen)
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi   File "/var/lib/openstack/lib/python3.8/site-packages/glance_store/_drivers/rbd.py", line 288, in get_connection
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi     client = rados.Rados(conffile=conffile, rados_id=rados_id)
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi AttributeError: 'NoneType' object has no attribute 'Rados'
2022-09-09 12:54:21.833 9 ERROR glance.common.wsgi 

I did some research and it looks like an issue associated with the container missing the python3-rados or python3-ceph-common packages but I cannot say for sure as from what I can tell python3-ceph-common is installed. I can say that us-docker.pkg.dev/vexxhost-infra/openstack/glance:22.1.1.dev2-1 works fine.

[mnaser]

Good catch, let me have a look at this.

[mnaser]

@runlevel-six thanks for the report, hopefully once we land #25 we'll be able to avoid all those things we haven't been catching.

I just tested and the image should have RADOS right now.

[runlevel-six]

Tested successfully. Thanks!