Upstream Image Encryption Standardization #560
Comments
josephineSei
added
SCS-VP10
|
I have discussed and outlined the spec with @markus-hentsch and currently writing it. |
|
On the user side we'd need to be able to create LUKS-encrypted files. Nova devs made us aware of the QEMU tooling being able to do so allegedly. Here is the Nova implementation:
Here's a shortened example: echo "muchsecretsuchwow" > secret_file.key
qemu-img convert -f raw -O luks --object secret,id=sec,file=secret_file.key -o key-secret=sec \
-o cipher-alg=aes-256 -o cipher-mode=xts -o hash-alg=sha256 \
-o ivgen-alg=plain64 -o ivgen-hash-alg=sha256 \
$INPUT_FILE $OUTPUT_LUKS_FILEIt works within a Docker container running Ubuntu LTS but interestingly it doesn't on my NixOS setup, failing with: Versions of qemu-img are 6.2.0 (Ubuntu) and 8.1.5 (NixOS) respectively. I wonder if there is another system dependency in play here or if it's the version difference. We should investigate because this impacts the ability to create images on the client side. EDIT: Tried it on |
|
As I was adding restore instructions to the user data backup guide in SovereignCloudStack/docs#176 I reproduced the process of creating volumes from previously encrypted (LUKS) images originating from Cinder itself on my DevStack and noticed that Cinder does not verify that the target volume actually has an encrypted type: $ file image.raw
image.raw: LUKS encrypted file, ver 1 [aes, xts-plain64, sha256]
$ file image.key
image.key: data
$ openstack secret store --algorithm aes --bit-length 256 --mode cbc \
--secret-type symmetric --file image.key --name restored-image-key
$ openstack secret list -f value -c "Secret href" -c "Name"
http://10.0.1.116/key-manager/v1/secrets/6ea6b0a8-de50-45b8-90b7-9470c4dd201a restored-image-key
$ export SECRET_ID=6ea6b0a8-de50-45b8-90b7-9470c4dd201a
$ openstack image create --file image.raw \
--property cinder_encryption_key_id=$SECRET_ID \
--property cinder_encryption_key_deletion_policy=on_image_deletion \
restored-image
$ openstack volume create --size 1 \
--image restored-image volume-restored-notype
$ openstack volume show -f value -c type volume-restored-notype
lvmdriver-1 # <- this is an *unencrypted* volume type! (which contains LUKS blocks now)
$ openstack volume create --size 1 \
--image restored-image --type lvmdriver-1-LUKS \
volume-restored-lukstype
$ openstack volume show -f value -c type volume-restored-lukstype
lvmdriver-1-LUKS
$ openstack server create \
--volume volume-restored-notype \
... server-from-untyped-volume
$ openstack server create \
--volume volume-restored-lukstype \
... server-from-luks-volumeThis results in the server It seems that this is an oversight in Cinder in its current implementation? |
|
I tested this with a simple encrypted volume to encrypted image to volume: A seemingly unencrypted volume is created from the encrypted image. But as far as we know, there is no decrypting mechanism implemented in Cinder to go from such a Cinder-specific LUKS encryption in the image to an unencrypted volume. We should definitely file a bug in Cinder for this one. |
How to test, if a server bootedWhen creating a server from a volume, like my above case, it will be shown as active even though this cannot be true To show the boot log of a server the following command can be used: The log will then look something like: But if created from a server with an unencrypted volume from an encrypted image the log file remains empty: |
|
I uploaded the new spec to gerrit: https://review.opendev.org/c/openstack/glance-specs/+/915726 |
|
I reported the bug uncovered in #560 (comment) at https://bugs.launchpad.net/cinder/+bug/2061154 |
I raised this in the IRC, the following was the answer: |
|
We currently evaluate whether we will also need a spec for Cinder. What we definitely need is a blueprint for Cinder to track all development. I have written a blueprint and tried to include all possible points, where there will be implementation needed in Cinder: |
|
We got a review for the Spec (I am currently updating it) and there is one last question to discuss: We wanted to introduce a new container format "encrypted" - so it would be easily visible to anyone, that this image is encrypted. To identify what the underlying format is, we introduced a property: "os_decrypt_container_format". Now Nova and Cinder both would like to have the container format showing up in the original property and Cinder additionally would need some parameter to know whether the encrypted image is compressed or not. The thing here is, that we could check whether encrypted images are always qcow or raw when the container_format and decrypt_container_format is set. The metadata could be set after creating an image in the upload step. So it may be allowed to create an encrypted image with a format neither cinder nor nova could use. We would like to avoid such bad user experience. So we want to ask the following questions in the Glance meeting this thursday:
|
|
From the Cinder meeting today I got the wish to also create a small Cinder spec: https://etherpad.opendev.org/p/cinder-dalmatian-meetings |
|
I created the Cidner spec today: https://review.opendev.org/c/openstack/cinder-specs/+/919499 |
|
We got reviews on the spec, and Markus and I are going through them and answering questions. |
|
I adjusted the Glance spec and removed the container_format 'encrypted', because this was one of the most discussed parts of the spec: https://review.opendev.org/c/openstack/glance-specs/+/915726 |
|
I got feedback on the Glance patch with the question about what to do when the image conversion plugin is activated. Image conversion as I understand it, is not something that can be triggered by a CLI command. Instead when the plugin is activated ALL images that are created will be automatically converted to the ( through the config ) specified format after uploading and before storing it. This creates a few questions regarding encrypted images:
As this optional feature of Glance may already have problems with 1, and we would need to at least forbid uploading encrypted images when the target format is vmdk (maybe also when the target format is qcow2) this would be a lot more of implementation work. So we will for now render this out of scope for the spec. Maybe this can be done after the image encryption is in place and if it is needed by operators and users. |
Seems like this is part of the "Interoperable Image Import"1. Which references the following methods2:
This makes me wonder if this is applicable to images uploaded to Glance by Nova or Cinder at all. This is limited to images from external sources I think. Nonetheless, at the latest when the user is initiating such an image upload we need to make sure that we account for this concerning the image encryption. Footnotes |
|
I am looking into the Cinder Side a little bit more because, we got a comment from Dan Smith regarding image conversion and Glance checking a few Image parameters: The image conversion does only seem to be triggered when a user is uploading an image, not if Cinder or Nova upload one. Forbidding that would put more responsibility on the user side, but we also do that with the image encryption. The check for the virtual size can be omitted, because we only allow 2 types of encrypted images: qcow2 and raw and we want to introduce the "os_decrypt_size" parameter that should discribe the size of the unencrypted image. We may even mandate using this parameter. The format for encrypted raw images is only checkable, when decrypting the image. So while this is a valid point to discuss, as Glance will reject images, that do not have the format, they say they have, the Glance team did not wanted to have the power to encrypt or decrypt images back when we discussed gpg-based image encryption. I doubt that this has changed and i also do not see a good reason, Glance should be able to do this. |
|
I edited the glance spec to forbid image conversion in a more explicit way and included some more comments on the glance spec. |
|
I adjusted the key-managing part of the spec to clarify the behavior of deleting keys and added a part to put images into ERROR state, when they are encrypted and image conversion is enabled and need to be done. |
|
I attended the Glance meeting and asked for more reviews to the spec, so it can be merged. |
|
I adjusted the glance spec as there were a few open questions: https://review.opendev.org/c/openstack/glance-specs/+/915726 |
|
There were a few more questions on a spec. One thing is, that with the standardization we would also OPTIONALLY allow users to set a parameter that advises glance to delete a key on image deletion. This might be something to discuss in a Glance meeting. I alsoe edited the work items and added some more for tests, documentation and a migration script for old cinder-specific parameters. |
Converting between qcow2+LUKS and LUKS images using
|
Relation to OSSA-2024-001Since special qcow2 functionalities are the source of the recent OSSA-2024-001 vulnerability, we will also be affected if we opt to use The mitigation added by OpenStack is implemented as an image format inspection in all relevant services (Glance, Cinder, Nova). We can re-use this. As an example, consider the "cirros.raw.luks.qcow2" qcow2+LUKS file created above and the >>> from format_inspector import QcowInspector
>>> i = QcowInspector.from_file("cirros.raw.luks.qcow2")
>>> i
<format_inspector.QcowInspector object at 0x7068ea532900>
>>> i.has_header
True
>>> i.has_backing_file
False
>>> i.has_unknown_features
False
>>> i.safety_check()
TrueThe qcow2+LUKS file created by |
|
I was able to discuss this also with the Glance team in the Glance channel and meeting today, and they and fungi do not see a problem, when the metadata is not encrypted. We can continue with our effort on standardizing the image encryption upstream |
|
Glance spec is finally merged now: https://review.opendev.org/c/openstack/glance-specs/+/915726 |
|
The cinder Spec was also merged: https://review.opendev.org/c/openstack/cinder-specs/+/919499. The following steps need to be implemented:
Cinder:
|
|
I spoke with Markus, about all steps that need to be implemented or written (Glance documentation and security guide). We added some steps. Markus may start implementing, when I am on vacation. |
Currently there is the possibility in Cinder to encrypt volumes and in Nova to use qcow2 encrypted images (still under development).
Both can lead to and use LUKS-encrpyted images, but those are different and not aligned:
@markus-hentsch also found out in #541 that uploading a LUKS-encrypted image (that was created from a volume) to another cloud in combination with setting a few parameteres (cinder_encryption_key, etc...) will result in an image that can be used to create an encrpyted and functional volume.
As a user it would be good to have a streamlined operation to use encrypted images in openstack for both volumes and ephemeral storage and to also allow interoperability between clouds.
Therefore we need and will propose standardized parameters to describe and detect an encrypted image, which might be similar to the parameters described here: https://specs.openstack.org/openstack/cinder-specs/specs/zed/image-encryption.html
But will use the LUKS encryption.
So those encrypted images could be natively mounted in Nova or just formed into a volume (raw LUKS images can be directly used, qcow images need to be flattened).
With such a way encrypted backup images can be easily downloaded and transferred to another cloud.
This is a result from a lengthy discussion at the PTG with Nova, Cinder and Glance ( https://etherpad.opendev.org/p/dalmatian-ptg-cinder#L376 )
Followup tasks may be to implement re-encryption to fully change keys for LUKS volumes and images.
The text was updated successfully, but these errors were encountered: