Proposal to add Encrypted Layer Mediatype

[lumjjb]

This is a first draft of modifications to the OCI spec on Encrypted Container Images. Work around this has been on-going for quite a while (since the start of the issue). The current design is based on the discussions we've had in the issue and with the runtime implementation in containerd.

Abstract

We would like to propose a new media type for encrypted layers of a container image. This addition would facilitate the ecosystem for encrypted container images. This allows users with stricter trust requirements to be able ensure end-to-end encryption from build to runtime. In addition, it allows users to use a centralized managed repository (i.e. Docker Hub) without any risk of their images being compromised.

Issue Details

Based on the discussion of #747

Implementations

The specification and libraries to perform encryption/decryption

The implementation to perform encryption and decryption of this spec is in:

• https://github.com/containers/ocicrypt
  • func EncryptLayer(ec *config.EncryptConfig, encOrPlainLayerReader io.Reader, desc ocispec.Descriptor) (io.Reader, EncryptLayerFinalizer, error)
  • func DecryptLayer(dc *config.DecryptConfig, encLayerReader io.Reader, desc ocispec.Descriptor, unwrapOnly bool) (io.Reader, digest.Digest, error)

Container Runtimes

Implemented and upstreamed for containerd stack

• containerd (1.3+)
• https://github.com/containerd/imgcrypt

Implemented and upstreamed for RedHat stack

• cri-o 1.17+
• buildah 1.5+
• skopeo

Registry

Current docker distribution supports: https://github.com/docker/distribution. Tested with v2.7.1.

Build toolchain

The plan for this is to have docker CLI, skopeo and buildah. In the meantime, we also support encryption of images in containerd via imgcrypt ctr client.

Current implementations for RedHat stack

• buildah 1.5+
• skopeo

Current implementations for containerd stack

• ctr-enc
• To integrate into docker CLI, we are currently waiting on Proposal: containerd image integration moby/moby#38043. Tracking with Creating support for Encrypted Container Images moby/buildkit#714

Kubernetes Integration

It is important to note that kubernetes integration is not required to use encrypted container images. i.e. in cri-o you are able to specify a directory containing keys.

A way to do this is via an operator k8s-enc-image-operator

Signed-off-by: Brandon Lum lumjjb@gmail.com

[stevvooe]

If I'm understanding this correctly, this structure ends up in the annotations below. It might be helpful to describe the annotations first, then their contents.

I also think for annotations, they need to be added to the annotations.md.

[lumjjb]

EDIT: Updated to reflect introduction of annotations with link to annotations.md before metadata explanation.

Agreed on adding to annotations.md.

[mrunalp]

@vbatts ptal

[lumjjb]

@vbatts As per our conversation a couple days ago, Based on the RFC6838 and RFC6839.

A specification like nondistributable layers would work:

• application/vnd.oci.image.layer.enc.tar.v1
• application/vnd.oci.image.layer.enc.tar.gzip.v1

The other way to do this is to have one encrypted layer mediatype application/vnd.oci.image.layer.enc.v1 and specify the original underlying mediatype in an annotation. But i'm unsure if this may make handling on the runtimes a bit more difficult.

cc: @estesp @dmcgowan @crosbymichael @mrunalp @mtrmac @vrothberg

[dmcgowan]

I don't think reproducing what was done for nondistributable layers is a good idea. Runtimes end up having to special case these since they are hard to interpret. For example vnd.oci.image.layer. may indicate that an object should be treated as a layer, but then .tar.v1 indicates it is a tar. What is sandwiched in between those is some important metadata. In the nondistributable case it is just a hint of how an artifact is distributed, in the encrypted case, it is a requirement for how the tar must be processed. I think we really need to format the content types in a way that indicates how they should be processed, ensuring that modifications on those types are wrapping. Personally I like using + to indicate encoding (whether that is encryption or formatting). I think we should discuss at an upcoming OCI weekly meeting about these types and their relation to RFC6838.

[lumjjb]

Updated based on discussions from OCI weekly dev discussion to move from +enc to +encrypted. We agreed on using suffixes on mediatypes and having these defined on top of artifacts.

[lumjjb]

@mikebrow Responding to #791 (comment) in this thread instead since it is more specific to encryption details.

The main integration points for Encrypted Container Images are:

• The specification and libraries to perform encryption/decryption
• Container runtimes
• The build toolchain
• Registry
• Kubernetes Integration

Do let me know if there's something specific you are looking for and I can provide a direct link.

The specification and libraries to perform encryption/decryption

The definition of encryption indication and how encryption metadata is defined in:

• Proposal to add Encrypted Layer Mediatype #775

The implementation to perform encryption and decryption of this spec is in:

• https://github.com/containers/ocicrypt
  • func EncryptLayer(ec *config.EncryptConfig, encOrPlainLayerReader io.Reader, desc ocispec.Descriptor) (io.Reader, EncryptLayerFinalizer, error)
  • func DecryptLayer(dc *config.DecryptConfig, encLayerReader io.Reader, desc ocispec.Descriptor, unwrapOnly bool) (io.Reader, digest.Digest, error)

Container Runtimes

Implemented and upstreamed for containerd stack

• containerd (1.3+)
• https://github.com/containerd/imgcrypt

Implemented and upstreamed for RedHat stack

• cri-o 1.17+
• buildah 1.5+
• skopeo

Registry

Current docker distribution supports: https://github.com/docker/distribution. Tested with v2.7.1.

Build toolchain

The plan for this is to have docker CLI, skopeo and buildah. In the meantime, we also support encryption of images in containerd via imgcrypt ctr client.

Current implementations for RedHat stack

• buildah 1.5+
• skopeo

Current implementations for containerd stack

• ctr-enc
• To integrate into docker CLI, we are currently waiting on Proposal: containerd image integration moby/moby#38043. Tracking with Creating support for Encrypted Container Images moby/buildkit#714

Kubernetes Integration

It is important to note that kubernetes integration is not required to use encrypted container images. i.e. in cri-o you are able to specify a directory containing keys.

cc: @stefanberger, @harche

[vbatts]

@lumjjb ok, finally discussing this in the weekly call.
Since this encrypted layer discussion has begun, the https://github.com/opencontainers/artifacts has come along. To a large degree, this is another mimetype/mediaType that would be documented there, and allowed as an extension to the image-spec.

There is a lot of good work here in the PR, that should transfer to something like a README for these mime-types in the artifacts repo.
Does this make sense?

[lumjjb]

@lumjjb ok, finally discussing this in the weekly call.

Awesome, sounds good.. I will come by the next OCI call on Wednesday.

There is a lot of good work here in the PR, that should transfer to something like a README for these mime-types in the artifacts repo.
Does this make sense?

Yup! I will create an encryption.md PR to the artifacts repo, and I suppose it would be an edit pointing to that file from opencontainers/artifacts#13 ?

[AkihiroSuda]

What's current status of this?

[vbatts]

@AkihiroSuda it seems to be stalled on opencontainers/artifacts#15

[lumjjb]

As a quick update on this, I've updated the list of links in the comments above.

The encryption support is now part of upstreamed containerd, cri-o, buildah, docker registry and skopeo tools.

[npmccallum]

@vbatts I'm a bit late to the party. But this has recently come into scope for me.

It isn't clear to me from the various pieces of documentation strewn about whether the integrity protection for a given layer also provides integrity protection for all the layers beneath it.

For example, you have an image with multiple layers:

• Layer 1
• Layer 2
• Layer 3 (encrypted)

Does the integrity protection of layer 3 also include measurements of layers 2 and 1?

If not, it would be possible to modify one of the intermediate layers in order to gain backdoor access to the plaintext.

[lumjjb]

Hi @npmccallum The encryption is only targeting confidentiality, although it has some side effects of integrity on a per layer basis. This was designed to be used in conjunction with Docker Content Trust / Notary / RH Simple signing, they should be able to work hand in hand.

[npmccallum]

@lumjjb If you don't have integrity of the whole, you don't have confidentiality of the layer.

[lumjjb]

If i understand your statement correctly, I believe those signing technologies ensure the integrity of the image manifest, and the integrity of the layers are ensured as being part of the signed manifest hash. This is done as a side effect of layers being content hashes.

[npmccallum]

@lumjjb Okay. So this is working off the presumption that you have integrity protection for the whole manifest which contains measurements of all the layers. Therefore, there is integrity protection for the whole and confidentiality protection for the parts.

During deployment, do you ensure that the manifest is validly signed and all the layers match their measurements before you decrypt? Or is it possible to decrypt without validating a signature of the manifest and validating the measurement of the layers?

[stefanberger]

I suspect there is a desire to allow the layering of encrypted layers on top of unencrypted layers. It might be enough to codify:

1. Unencrypted layers `MUST NOT` be built on top of encrypted layers.

2. Encrypted layers `MAY` be built on top of unencrypted layers.

3. All encrypted layers `MUST` use the same encryption key.

4. An encrypted layer `MUST` include the measurements of all previous layers in its AEAD.

@npmccallum Would it not be sufficient to require that the last layer must be encrypted for each key that was used? At that layer the accumulation of the hashes of all previous layers, encrypted or plain, can be enforced.

[lumjjb]

Have not forgotten about this - have allocated some time next week to go through the previous discussions and synthesize them.

[lumjjb]

Below are the goals and usecases/scenarios from the initial proposal

Content encryption of OCI artifacts (Protection at rest from build -> runtime)

All artifacts have their content kept confidential from the build phase to the runtime or whenever the artifact needs to be consumed. This attempts to address the attack vectors between build and runtime. The threat model is that the producer and consumers of the image are trusted. The proposed PR is opencontainers/artifacts#15.

The discussions of this was surrounding the work going around Docker Content Trust / Notary around that time. It was the intention that DCT or other image manifest signing technologies together with the artifact design of being content addressible would be used to attest the integrity and provenance of the payload.

Initial Scope

The initial focus of the ocicrypt project was on the container images. The idea was to focus on the encryption of the content aspects. This consists of the config file and the layers.

The work was later scoped down to only layer artifacts. This was due to the implementation scope as a POC in containerd. The mechanics of introducing this into configs were much more involved than the layers, resulting in the initial scope.

Geofencing execution

Another scenario was towards providing geofencing capabilities. Encrypting content to tie it to key material which can then be securely distributed to an authorized platform. The specific motivator of this usecase was geofencing and compliance, and export control/DRM were other examples.

Keys would be used or distributed with platform attestation-backed authorization. Thus with key management, The images would only be usable if they were on nodes that are authorized. This is talked about a bit more in https://kccnceu20.sched.com/event/Zepc.

Container layer flexibility

One of the main themes for the design on layers was to try to preserve characteristics of container images that people like. This includes ability to build on top of other images, layer reuse, external image scanning.

The decision for encrypting each layer with a generated DEK, and wrapping the key was to be able to provide layer re-use. The wrapped metadata is therefore stored within the annotations and not the layer itself, as authorizing a new consumer would have changed the hash of the layer.

Having the wrapped keys as annotations (these wrapped keys could be artifacts on their own pointing to the layers), would also allow approved image scanners or trusted CI/CD tooling to perform analysis on the layers. Thus one could encrypt an image for both its runtime and a trusted party.

Another notable discussion point was that of multiple owners of different parts of the stack. The idea is one that is difficult to do with current tooling due to the way the builders construct images, but not impossible. For example, if the middleware team maintains the middleware layer and the application team is responsible for the application. An encrypted image could be created by the layering of the middleware encrypted layer, and the application could be
added on top without accessing the middleware, based on a file based contract provided between teams. The runtime could then run the container image only if it is authorized to use components of both teams.

@stefanberger

[vbatts]

Suggested change

	* **org.opencontainers.image.org.opencontainers.image.enc.keys.pkcs7** Base64 comma separated [PKCS7(RFC2315)](https://tools.ietf.org/html/rfc2315) encrypted messages that contain wrapped PrivateLayerBlockCipherOptions JSON objects for [encryption](layer.md#layer-encryption)
	* **org.opencontainers.image.enc.keys.pkcs7** Base64 comma separated [PKCS7(RFC2315)](https://tools.ietf.org/html/rfc2315) encrypted messages that contain wrapped PrivateLayerBlockCipherOptions JSON objects for [encryption](layer.md#layer-encryption)

[vbatts]

@lumjjb would like to bubble this work up. What is the current situation and your approach? has it evolved since you first opened this PR? I'm thinking this discussion could start up again.

[lumjjb]

Hey @vbatts ! Certainly , for some reason this was buried in my inbox and surfaced when i was searching for something! Would like to get back to this discussion, where shall be a good place to do this? Should we chat on the sidebar first or bring this up at one of the weekly oci calls?

[vbatts]

On 21/07/21 07:59 -0700, Brandon Lum wrote: Hey @vbatts ! Certainly , for some reason this was buried in my inbox and surfaced when i was searching for something! Would like to get back to this discussion, where shall be a good place to do this? Should we chat on the sidebar first or bring this up at one of the weekly oci calls?

Here, mailing-list ***@***.***), or the weekly.

[lumjjb]

Awesome - i should be able to come to this week's or next week's meeting.

[AkihiroSuda]

needs rebase

[optimistyzy]

May I ask a question why tar+ encrypted + zstd or tar + zstd + encrypted is not supported? Thanks.

[lumjjb]

at the time, there wasn't +zstd yet, but yes it should support. +encrypted should be treated as a suffix and thus should be handled on all mediatypes that end with +encrypted

[lumjjb]

In that case, you may want to just skip the compression since encrypted bytes have inherent randomness and high entropy , which results in compression not being effective.

…

On Wed, Sep 21, 2022, 1:45 PM Mike Brasher ***@***.***> wrote: Compressing before encrypting (tar+gzip+encrypted) limits the usability of the downloaded layer. If the layer were encrypted before compressing ( tar+encrypted+gzip), then the downloaded layer could be uncompressed and then directly mounted as a read-only tar file system over a dm-crypt target. This is crucial for confidential computing environments where layers reside in an untrusted environment and the overlay mount is performed within a trusted environment. — Reply to this email directly, view it on GitHub <#775 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXLDBVHAL6JVOVMVFPRLODV7NCTTANCNFSM4HI4F2WQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[zvonkok]

/cc @zvonkok