Add image encryption support and ctr support #3134

stefanberger · 2019-03-27T19:55:57Z

This series of patches adds support for the asymmetric and symmetric layer encryption code as well as support for ctr's images subcommand to support encryption (encrypt) and decryption (decrypt) of images as well as the retrieval of information about the layers of an image (layerinfo). The latter shows the encryption scheme used for each layer. An extensions to the documentation is also added.

stefanberger · 2019-03-27T19:56:05Z

This PR is a superset of PR #3001. It adds patches on top of the branch created for PR #3001, so the patches from PR #3001 are also visible here. The patch "Address Phil's comments to the asymmetric crypto code" is the last patch from #3001 .

codecov-io · 2019-03-28T12:28:03Z

Codecov Report

Merging #3134 into master will decrease coverage by 0.8%.
The diff coverage is 33.47%.

@@            Coverage Diff             @@
##           master    #3134      +/-   ##
==========================================
- Coverage   45.06%   44.25%   -0.81%     
==========================================
  Files         114      122       +8     
  Lines       12739    13489     +750     
==========================================
+ Hits         5741     5970     +229     
- Misses       6131     6601     +470     
- Partials      867      918      +51

Flag	Coverage Δ
#linux	`48.1% <36.49%> (-0.86%)`	⬇️
#windows	`39.72% <33.47%> (-0.65%)`	⬇️

Impacted Files	Coverage Δ
pkg/encryption/gpg.go	`0% <0%> (ø)`
pkg/encryption/reader.go	`0% <0%> (ø)`
pkg/encryption/gpgvault.go	`0% <0%> (ø)`
content/helpers.go	`19.51% <0%> (-1.92%)`	⬇️
pkg/encryption/keywrap/pgp/keywrapper_gpg.go	`35% <35%> (ø)`
pkg/encryption/keywrap/pkcs7/keywrapper_pkcs7.go	`44.61% <44.61%> (ø)`
pkg/encryption/encryption.go	`50.69% <50.69%> (ø)`
pkg/encryption/keywrap/jwe/keywrapper_jwe.go	`52.77% <52.77%> (ø)`
pkg/encryption/blockcipher/blockcipher.go	`69.44% <69.44%> (ø)`
pkg/encryption/blockcipher/blockcipher_siv.go	`79.46% <79.46%> (ø)`
... and 14 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ff1f87...dde436e. Read the comment docs.

stevvooe · 2019-04-15T17:15:49Z

docs/encryption.md

+
+# Example End User Usage
+
+We have added 3 commands in the [`ctr`](https://github.com/containerd/containerd/tree/master/cmd/ctr) client under the image module. They are `ctr image encrypt/decrypt/layerinfo`.


Enumerate the commands, rather than listing them as slashed items.

Fixed cda9d21a57c541e5b2692f1b0f8d29826fc4daec

stevvooe · 2019-04-15T17:16:40Z

docs/encryption.md

+   #                                                                    DIGEST         PLATFORM      SIZE   ENCRYPTION   RECIPIENTS
+   0   sha256:3427d6934e7749d556be6881a17265c9817abc6447df80a09c8eecc465c5bfb3      linux/amd64   2206947
+   0   sha256:d9a094b6b49fc760501d44ae96f19284e86db0a51b979756ca8a0df4a2746c79     linux/arm/v6   2146469
+   1   sha256:ef87d8b3048d8f1f7af7605328f63aab078a1433116dc15738989551184d7a87     linux/arm/v6       191   jwe,pkcs7    [jwe], [pkcs7]


Are the recipients users or are they the same as the algo?

For some encryption technologies we can display the actual recipients, such as for example OpenPGP, for others we cannot but can only show how many there are.

Actually, even displaying the exact number of recipients may not always be possible.
JWE and PKCS7 work with key (files) rather than a keyring (like OpenPGP) and don't associate these keys with email addresses like OpenPGP does via its keyring.

stevvooe · 2019-04-15T17:17:08Z

docs/encryption.md

+
+## Decrypt
+
+The following command performs an decryption of the encrypted image `docker.io/library/alpine:enc` to the image tag `docker.io/library/alpine:latest`.


This sentence doesn't match the example below.

Fixed! cda9d21a57c541e5b2692f1b0f8d29826fc4daec

stevvooe · 2019-04-15T17:19:29Z

docs/encryption.md

+// GetImageLayerInfo gets the image key Ids necessary for decrypting an image
+// We determine the KeyIds starting with  the given OCI Decriptor, recursing to lower-level descriptors
+// until we get them from the layer descriptors
+func GetImageLayerInfo(ctx context.Context, cs content.Store, desc ocispec.Descriptor, lf *encryption.LayerFilter, layerIndex int32) ([]encryption.LayerInfo, error)


Is there a PR with this? This interface doesn't make sense. First off, referencing layers by their index creates a dependency on layer ordering. Image format in the future might not have that model. Making it follow this model will make it hard to change in the future.

I removed the layerIndex now from this public function's signature.

Updated docs to reflect new change cda9d21a57c541e5b2692f1b0f8d29826fc4daec

stevvooe · 2019-04-15T17:20:19Z

images/encryption/encryption.go

+// LayerInfo holds information about an image layer
+type LayerInfo struct {
+	// The Number of this layer in the sequence; starting at 0
+	Index      uint32


Using the index to reference a layer won't be very portable if we change the image format. We need to make encryption scheme format portable.

We merely use the index of a layer to present it to the user so he can choose which layer of the stack of layers to encrypt. We did assume that the function creating the array of LayerInfos would know the stacking order of the layers.

Is there a PR with this? This interface doesn't make sense. First off, referencing layers by their index creates a dependency on layer ordering. Image format in the future might not have that model. Making it follow this model will make it hard to change in the future.

Using the index to reference a layer won't be very portable if we change the image format. We need to make encryption scheme format portable.

Two things: don't encode things as "negative numbers". This usually falls apart over time. Also, let's reference layers by their digest.

I think we see where you're coming with in terms of the API definition for encryption. In order to address this comment, one way to do it is to make it such that:

LayerFilter becomes a digest filter instead of a layer index filter

The digest is taken by looking into the descriptor manifest

The option to provide encryption for top-most layers would be done on the client side (i.e. in ctr), this would mean looking at the descriptor and finding the manifest of the top X layers and creating a layer digest filter for encryption. Reference to top most layer would be via index (-1 for topmost, -2 for 2nd top most - unless there is another idea for a layer addressing scheme which makes sense for a user trying to encrypt an image).

Does that sound about right? Let us know if this does not address the concern.

Given that, we also have some thoughts about having layer index filter be included as the API. We think that there is some merit in doing so, but we may not know certain developments in this area, so maybe the points we bring up are subject to that.

It seems like containerd API needs to know the ordering of layers, since it is part of the OCI image spec, and is required in order to unpack an image. In that sense, it is a reasonable assumption that layer indexing is made available to containerd.

This would reduce the number of calls to perform filters based on layers, and amount of data being transferred around. The digest is a good key to identify layers but requires some work beforehand to create a meaningful layer filter. If layer position is the only use case for layer filters, then it would make sense. But perhaps if media-type, size, etc. is another use case, then that would be a different valid perspective to use digest filters instead.

Let us know what you think, we are fine with either way as long as an end user is able to be given the ability to perform operations on not all layers, which is a valuable proposition in container image encryption over VM image encryption.

@stevvooe from my understanding (and with the recent changes since your initial comments), the indexing/"layer offsets" are UX-only elements which allow the command line tools to offer simple selection of a layer for encrypt/decrypt. The indexes don't become "permanent record" data within any store/metadata that persists; the actual work happens by walking OCI manifests and descriptors, so unless the OCI spec changes, I don't see any issue with indexes that end up as "helpers" for the UX interface of choosing layers.

It is possible for digests to be the selector as well, but obviously are a bit more cumbersome for the user given the lack of (usual) access to layer digests within tools like docker CLI (although ctr CLI will have a new layerinfo command via this PR though, to promote that detail for user visibility of encryption status)

Stefan or Brandon can correct me if I got anything wrong about the "persistence" of the indexing; in looking across the code I see that actual operations seem to be (properly) walking actual OCI digests/manifests, not using layer indexes as a new way to navigate images.

I understand that the indexes are client-only but it's a model that we have avoided in containerd. Layers aren't generally exposed as a first-class concept. I know that the OCI spec does declare layers, but to have any chance of moving away from layers, we need to avoid baking them into various components. I think we did a fairly good job of abstracting those concepts by decoupling snapshots and image layers. I think we can do the same here.

"layer position" is an extremely fragile way to reference layers and it is not at all necessary to solve this problem. It seems like something as simple as this would be sufficient:

func EncryptImage(desc Descriptor, shouldEncrypt func(desc Descriptor) bool) (Descriptor, error)

The above takes a desc pointing to the image and runs shouldEncrypt function on each component that says whether or not it should be encrypted. The resulting image, with encrypted components, is then returned as a descriptor. It's format agnostic and allows you to select which parts are important for encryption. The resulting content pointed to by descriptor is assembled during the process. You can use indexes or whatever you want to manage it there. None of the details are exposed to the caller.

Note that the function above might need a bit more, but hopefully that can help on avoiding exposure of internal details. I think we can work from here and find something that works.

stevvooe · 2019-04-15T17:21:29Z

images/encryption/encryption.go

+
+// LayerFilter holds criteria for which layer to select
+type LayerFilter struct {
+	// IDs of layers to touch; may be negative number to start from topmost layer


Two things: don't encode things as "negative numbers". This usually falls apart over time. Also, let's reference layers by their digest.

stevvooe · 2019-04-15T17:41:45Z

images/encryption/config/config.go

+	// for adding recipients on an already encrypted image we need the
+	// symmetric keys for the layers so we can wrap them with the recpient's
+	// public key
+	Operation int32 // currently only OperationAddRecipients is supported, if at all


We should probably represent these a different methods, rather than trying to encode an operation code. We don't really do anything like this in containerd, so this isn't a good choice for the API.

Fixed by removing. We only support adding recipients.

stevvooe · 2019-04-15T17:42:33Z

images/encryption/config/config.go

+	// symmetric keys for the layers so we can wrap them with the recpient's
+	// public key
+	Operation int32 // currently only OperationAddRecipients is supported, if at all
+	Dc        DecryptConfig


DecryptConfig for the field name.

images/encryption/config/config.go

lumjjb · 2019-04-15T20:15:31Z

Thanks for the review @stevvooe ! Looks like most of the changes are straightforward and we will start working on those. The only comment we need some clarity on is on the layer filters. @stefanberger and I had a discussion to understand the comments there and put together a summary of our discussion. I'll put that under on of the inline comments!

image_enc_test.go

images/encryption/blockcipher/blockcipher.go

images/encryption/blockcipher/blockcipher_siv_test.go

estesp · 2019-04-22T21:48:43Z

vndr is now reporting a mismatch between vendor.conf settings and actual files committed; maybe a recent commit as this used to pass? CI Log:

These files were modified:
 M vendor/golang.org/x/crypto/cast5/cast5.go
 M vendor/golang.org/x/crypto/ed25519/ed25519.go
 M vendor/golang.org/x/crypto/ed25519/internal/edwards25519/edwards25519.go
 M vendor/golang.org/x/crypto/openpgp/keys.go
 M vendor/golang.org/x/crypto/openpgp/packet/encrypted_key.go
 M vendor/golang.org/x/crypto/openpgp/packet/packet.go
 M vendor/golang.org/x/crypto/openpgp/packet/private_key.go
 M vendor/golang.org/x/crypto/openpgp/packet/public_key.go
 M vendor/golang.org/x/crypto/openpgp/packet/signature.go
 M vendor/golang.org/x/crypto/openpgp/packet/userattribute.go
 M vendor/golang.org/x/crypto/openpgp/write.go
The command "../project/script/validate/vendor" exited with 1.

theopenlab-ci · 2019-07-16T19:34:48Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 22s (non-voting)

theopenlab-ci · 2019-07-16T20:10:34Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 30s (non-voting)

theopenlab-ci · 2019-07-16T20:58:21Z

Build succeeded.

containerd-build-arm64 : FAILURE in 2m 05s (non-voting)

images/image.go

theopenlab-ci · 2019-07-16T22:13:17Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 45s (non-voting)

theopenlab-ci · 2019-07-16T22:29:06Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 51s (non-voting)

theopenlab-ci · 2019-07-16T22:45:33Z

Build succeeded.

containerd-build-arm64 : SUCCESS in 8m 10s (non-voting)

dmcgowan

Moved some code around, tested encrypt, decrypt, and layer info

LGTM

estesp · 2019-07-16T23:32:24Z

Do we want to squash this before merge? At 53 commits over a period of months, git bisect would be troublesome for a lot of the ones that needed rebase, are not compilable, etc.

dmcgowan · 2019-07-16T23:47:41Z

I don't think it is worth squashing. Mostly new code so bisect isn't important here.

lumjjb · 2019-07-17T00:11:42Z

Let me know if you'd like me to squash this and break down the changes into a single commit message.

crosbymichael · 2019-07-17T15:54:10Z

Lets squash it then get it in

Signed-off-by: Stefan Berger <stefanb@linux.vnet.ibm.com>

Signed-off-by: Stefan Berger <stefanb@linux.ibm.com> Signed-off-by: Brandon Lum <lumjjb@gmail.com>

Signed-off-by: Derek McGowan <derek@mcgstyle.net>

lumjjb · 2019-07-17T19:24:51Z

Squashed to 3 commits! Splitting up vendor imports with the crypto implementation and minor movement/changes.

estesp

LGTM; thanks for squashing @lumjjb

lumjjb · 2019-07-17T19:53:09Z

Thanks so much @dmcgowan and @estesp for following along the long journey of this gigantic PR! Am really glad to see this get merged!!! Thanks @crosbymichael , @AkihiroSuda for the reviews and comments too!

stefanberger · 2019-07-17T20:47:13Z

@dmcgowan @estesp @crosbymichael @AkihiroSuda Thanks a lot for you help!

dmcgowan · 2019-07-17T20:58:36Z

Thanks @lumjjb and @stefanberger for your patience and perseverance, this was one of the trickier ones.

stefanberger force-pushed the encryption_code_plus_ctr.pr branch 2 times, most recently from a7c096e to afac701 Compare March 28, 2019 10:46

stefanberger force-pushed the encryption_code_plus_ctr.pr branch from bbc68f5 to 87e755f Compare March 28, 2019 13:07

stefanberger force-pushed the encryption_code_plus_ctr.pr branch 3 times, most recently from 713d72f to eb08e6d Compare April 8, 2019 17:29

stevvooe reviewed Apr 15, 2019

View reviewed changes

images/encryption/config/config.go Outdated Show resolved Hide resolved

stefanberger force-pushed the encryption_code_plus_ctr.pr branch 5 times, most recently from bf01bf2 to f79f651 Compare April 16, 2019 18:06

AkihiroSuda reviewed Apr 18, 2019

View reviewed changes

image_enc_test.go Outdated Show resolved Hide resolved

AkihiroSuda reviewed Apr 18, 2019

View reviewed changes

images/encryption/blockcipher/blockcipher.go Outdated Show resolved Hide resolved

stefanberger mentioned this pull request Apr 18, 2019

Add image encryption/decryption code #3001

Closed

AkihiroSuda reviewed Apr 18, 2019

View reviewed changes

images/encryption/blockcipher/blockcipher_siv_test.go Outdated Show resolved Hide resolved

lumjjb force-pushed the encryption_code_plus_ctr.pr branch from 71a6a8d to cda9d21 Compare April 19, 2019 14:34

stefanberger force-pushed the encryption_code_plus_ctr.pr branch from cda9d21 to 1662c79 Compare April 23, 2019 00:31

lumjjb force-pushed the encryption_code_plus_ctr.pr branch from 7ed31bf to 854bcef Compare July 16, 2019 20:01

lumjjb force-pushed the encryption_code_plus_ctr.pr branch 2 times, most recently from 0c852b1 to 3fde38d Compare July 16, 2019 20:54

dmcgowan reviewed Jul 16, 2019

View reviewed changes

images/image.go Outdated Show resolved Hide resolved

dmcgowan force-pushed the encryption_code_plus_ctr.pr branch from ab79dad to 3867ddf Compare July 16, 2019 22:19

dmcgowan force-pushed the encryption_code_plus_ctr.pr branch from 3867ddf to 15a150c Compare July 16, 2019 22:36

dmcgowan approved these changes Jul 16, 2019

View reviewed changes

stefanberger and others added 3 commits July 17, 2019 15:19

Add crypto dependencies/vendoring

30c3443

Signed-off-by: Stefan Berger <stefanb@linux.vnet.ibm.com>

Implemented image encryption/decryption libraries and ctr commands

bf8804c

Signed-off-by: Stefan Berger <stefanb@linux.ibm.com> Signed-off-by: Brandon Lum <lumjjb@gmail.com>

Crypto library movement and changes to content helper interfaces

dde436e

Signed-off-by: Derek McGowan <derek@mcgstyle.net>

lumjjb force-pushed the encryption_code_plus_ctr.pr branch from 15a150c to dde436e Compare July 17, 2019 19:23

containerd deleted a comment from theopenlab-ci bot Jul 17, 2019

estesp approved these changes Jul 17, 2019

View reviewed changes

estesp merged commit c90a3d4 into containerd:master Jul 17, 2019

lumjjb mentioned this pull request Jul 18, 2019

Made fixes and optimizations to encryption GC #3430

Merged

lumjjb mentioned this pull request Oct 8, 2019

Image encryption/decryption support in skopeo containers/skopeo#732

Merged


		# Example End User Usage

		We have added 3 commands in the [`ctr`](https://github.com/containerd/containerd/tree/master/cmd/ctr) client under the image module. They are `ctr image encrypt/decrypt/layerinfo`.


		## Decrypt

		The following command performs an decryption of the encrypted image `docker.io/library/alpine:enc` to the image tag `docker.io/library/alpine:latest`.

Add image encryption support and ctr support #3134

Add image encryption support and ctr support #3134

Conversation

stefanberger commented Mar 27, 2019

stefanberger commented Mar 27, 2019

codecov-io commented Mar 28, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lumjjb Apr 19, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lumjjb commented Apr 15, 2019

estesp commented Apr 22, 2019

theopenlab-ci bot commented Jul 16, 2019

theopenlab-ci bot commented Jul 16, 2019

theopenlab-ci bot commented Jul 16, 2019

theopenlab-ci bot commented Jul 16, 2019

theopenlab-ci bot commented Jul 16, 2019

theopenlab-ci bot commented Jul 16, 2019

dmcgowan left a comment • edited Loading

Choose a reason for hiding this comment

estesp commented Jul 16, 2019

dmcgowan commented Jul 16, 2019

lumjjb commented Jul 17, 2019

crosbymichael commented Jul 17, 2019

lumjjb commented Jul 17, 2019 • edited Loading

estesp left a comment

Choose a reason for hiding this comment

lumjjb commented Jul 17, 2019

stefanberger commented Jul 17, 2019

dmcgowan commented Jul 17, 2019

codecov-io commented Mar 28, 2019 •

edited

Loading

lumjjb Apr 19, 2019 •

edited

Loading

dmcgowan left a comment •

edited

Loading

lumjjb commented Jul 17, 2019 •

edited

Loading