Proposal: Provenance step 1 - Transform images for validation and verification

[vbatts]

Background

The current image format does not allow for content addressable images nor
require metadata about the image to reference the content of the layer used by
the metadata. The identifiers for layers are randomly generated requiring
extra book keeping to map layer ids to the content being referred to. In a
highly distributed environment such as the Docker ecosystem, this book keeping
is cumbersome and complicates security.

This relates to #6805 #6959

Proposal Summary

Make images self-describing manifests containing a list of content addressable
layers, run configuration, and a signatures to identify the builder and verify
the image meets the expectations of the installer.

Image Manifest

The image manifest file will contain all the information which is needed to
pull, install, validate and run an image. It will contain a list of layers by
a content addressable id, history, run time configuration, and signatures.
This manifest is generated by the daemon. Initially this generation will happen
when an image is published, and ultimately happen anytime an image is built or
committed. Each manifest is required to be signed by the client creating the
manifest on push or build with additional signatures which can be added post
build to verify the quality of the manifest or validity of the builder. The
history will contain fully backward compatible metadata to allow old style
layer and metadata to be recreated from the manifest.

Signable manifest (or payload) refers to portions of the manifest which
will be signed by builder. The signable manifest is a JSON dictionary
containing the layers, run configuration, and history. The entire signable
manifest will signed, any changes including whitespace will require a new
signature. To aid in readability the signable manifest should be
well-formatted JSON.

Example: (totally subject to change)

{
   "name": "dmcgowan/test-image",
   "tag": "latest",
   "architecture": "amd64",
   "fsLayers": [
      {
         "blobSum": "tarsum+sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      },
      {
         "blobSum": "tarsum+sha256:cea0d2071b01b0a79aa4a05ea56ab6fdf3fafa03369d9f4eea8d46ea33c43e5f",
      },
      {
         "blobSum": "tarsum+sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      },
      {
         "blobSum": "tarsum+sha256:2a7812e636235448785062100bb9103096aa6655a8f6bb9ac9b13fe8290f66df"
      }
   ],
   "history": [
      "{\"id\":\"a9eb172552348a9a49180694790b33a1097f546456d041b6e82e4d7716ddb721\",\"parent\":\"120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16\",\"created\":\"2014-06-05T00:05:35.990887725Z\"...",
      "{\"id\":\"120e218dd395ec314e7b6249f39d2853911b3d6def6ea164ae05722649f34b16\",\"parent\":\"42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229\",\"created\":\"2014-06-05T00:05:35.692528634Z\"...",
      "{\"id\":\"42eed7f1bf2ac3f1610c5e616d2ab1ee9c7290234240388d6297bc0f32c34229\",\"parent\":\"511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158\",\"created\":\"2014-06-05T00:05:35.589531476Z\"...",
      "{\"id\":\"511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158\",\"comment\":\"Imported from -\",\"created\":\"2013-06-13T14:03:50.821769-07:00\"..."
   ],
   "schemaVersion": 1
}

Signed manifest refers to a manifest which includes the signature as well
as the signable manifest. The signed manifest could be represented as either a
JSON Web Signature (JSON serialization, see link), in which the payload is the
base64 encoded signed manifest, or an altered version of the signed manifest
JSON to include the signature as the last element of the JSON dictionary and a
record of the alternations to the original signed manifest included in the
signature. EIther format is fully verifiable and tamper-proof.

http://tools.ietf.org/html/draft-ietf-jose-json-web-signature-31#section-7.2

Example: (human readable format)

{
   "name": "dmcgowan/test-image",
   "tag": "latest",
   "architecture": "amd64",
   "blobSums": [
      {
         "blobSum": "tarsum+sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      },
      {
         "blobSum": "tarsum+sha256:cea0d2071b01b0a79aa4a05ea56ab6fdf3fafa03369d9f4eea8d46ea33c43e5f",
      },
      {
         "blobSum": "tarsum+sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      },
      {
         "blobSum": "tarsum+sha256:2a7812e636235448785062100bb9103096aa6655a8f6bb9ac9b13fe8290f66df"
      }
   ],
   "history": ["v1 compatible string encoded json for each layer"],
   "schemaVersion": 1,
   "signatures": [
      {
         "header": {
            "jwk": {
               "crv": "P-256",
               "kid": "LYRA:YAG2:QQKS:376F:QQXY:3UNK:SXH7:K6ES:Y5AU:XUN5:ZLVY:KBYL",
               "kty": "EC",
               "x": "Cu_UyxwLgHzE9rvlYSmvVdqYCXY42E9eNhBb0xNv0SQ",
               "y": "zUsjWJkeKQ5tv7S-hl1Tg71cd-CqnrtiiLxSi6N_yc8"
            },
            "alg": "ES256"
         },
         "signature": "m3bgdBXZYRQ4ssAbrgj8Kjl7GNgrKQvmCSY-00yzQosKi-8UBrIRrn3Iu5alj82B6u_jNrkGCjEx3TxrfT1rig",
         "protected": "eyJmb3JtYXRMZW5ndGgiOjYwNjMsImZvcm1hdFRhaWwiOiJDbjAiLCJ0aW1lIjoiMjAxNC0wOS0xMVQxNzoxNDozMFoifQ"
      }
   ]
}

Content Addressable Layers

Each layer of an image will be referenced by a checksum created from its
contents. This checksum will be used on push and pull to verify contents have
not been tampered and disallow the layer referred to in the manifest to be
changed after signed.

History

For auditability and assurance of the image, there will be a history section.
This history will convey the life of the the image (build steps, ancestry,
prior attestations on parent images, etc.).
It will have a generic form, and it is important to note that its content is
included in the signed payload.

Signature

Every client and daemon will contain both a public key pair which can be used
to sign manifests. So that the user on the host that publishes (or builds) the
image can sign the image manifest, without sharing their keys for all users on
the host.

Verification

Note: Verification framework will be vetted out in a separate Proposal
review, but the following is provided for a complete picture of its role.

Verification of a manifest will be done by checking the public key used to sign
the image manifest against an authorization graph linking keys to users and the
image namespaces. The authority for this graph will be a remote server which
can respond to authorization queries with signed statements which can be cached
and imported locally for future authorizations. These signed statements which
are received and cached contain a chain of trust which verify their
authenticity. A root certificate to verify this chain will ship with Docker to
allow immediate verification of these statements. Certificates are X509
certificates and verification uses an x509 chain included with the signed
statement. The signed statement will be a JSON Web Signature with the contents
a series of graph nodes and edges to be imported and the x509 chain in the
signature header.

http://tools.ietf.org/html/draft-ietf-jose-json-web-signature-31#section-4.1.6

Registry V2 API

(Note: this versioning is on the API of talking to a docker registry)

Unlike the present /v1/... which is locked to the root of the URI path, the
/v2/... is expected be the relative root of the path, for easier route
handling.

Manage manifest by tag

GET/PUT/DELETE /v2/images/<imgname>/<tagname>

List tags

GET /v2/images/<imgname>/tags

Download an image layer by content id

• Performs ACL verification
• Redirects to a temporary signed URL

GET /v2/images/<imgname>/<sumtype>/<sum>

Upload an image layer

PUT /v2/images/<imgname>/<sumtype>/<sum>

Upload an image layer

PUT /v2/images/<imgname>/<sumtype>

Compatibility

For compatibility with prior versions of docker-registries and docker daemons,
the manifest will store the json metadata used in previous versions in the
history section. This history will allow recreation of the layers in the
previous format and layout. Version 2 registries can synchronize content with
version 1 registries using this content in order to ensure content is still
accessible through the version 1 API.
There will be a couple of phases.

Phase 1

Have v2 capable registry and docker daemon that can:

• detect and push as v2 where possible
• pull from v2 where possible
• validate signature on pull

Phase 2

• escrow layer checksums on for layers on the daemon
• produce a manifest at build/commit time (not just on publish)

Phase 3

• provide a strict mode where only verified signed images can be run
• flags to validate layers

Strictly V2

In the future when signatures are enforced more strictly, it will become more
difficult to do this synchronization as version 1 will not validate signatures
and creation version 2 manifests from version 1 registries will not have a
signature of the builder.

Attribution

Folks involved in this design so far
@dmcgowan @dmp42 @jlhawn @shykes and @vbatts

[thaJeztah]

Very good proposal. Adding more trust to Docker will become more and more important.

Since I'm not an expert on signing/validation, I've got some questions (excuse me if they're silly);

• will it still be possible to run a v2 registry as a 'static' registry (i.e. static files/directories, with JSON files for meta data)
• will it be possible to configure the client / daemon to only "trust" images signed with a specific (root) certificate? I.e. Only allow employees to build from images of trusted image maintainers?

One thing that still worries me is the "chain of trust" in the automated builds. Although the maintainer can be trusted, many Dockerfiles make use of external libraries / source files that are downloaded during build and could have been tampered with. I realise this is outside the scope of this proposal, but could be something to give some thought in the future.

[dmcgowan]

@thaJeztah provenance is as alluded to in the title going to be multiple steps and phases. The chain of trust, manifest, signature, and v2 registry are the core components being baked in now that will allow for further development of features such as what you mentioned about multiple certificates and requiring different types of certification of an image. The v2 registry api should allow serving 'static' files like you mentioned, but is this referring a registry that is pull only or just a file system implementation?

The initial "chain of trust" is about verifying an image is built by someone who has permission to build images of that name. Likewise future signatures could be added in future proposal to attest to other aspects of an image. Tracking all the sources which makes up a Docker build is being thought about, and content addressable layers is the first step toward that.

[thaJeztah]

but is this referring a registry that is pull only or just a file system implementation?

Basically, a simple read-only repository hosted using apache or nginx. I've seen this was possible with the current (v1) and was wondering if this was still possible after this change.

Thanks for explaining the "roadmap" for future additions on this matter. Thank you and the other contributors for taking security and trust at heart. LVGTM 😄

[dmp42]

Basically, a simple read-only repository hosted using apache or nginx. I've seen this was possible with the current (v1)

I don't think it is possible with v1 - but we are aiming for it with v2 eventually.

[thaJeztah]

@dmp42 Just for reference; this is what I was talking about; #4607 and https://github.com/vbatts/d2r

However, let's not delve into static registries too much as it is only slightly related to this proposal and would only clutter up this discussion. 😺

[vbatts]

@thaJeztah static/read-only is expected to stay possible.

[tianon]

wait, why the embedded signatures? why not just sign the whole file, .asc style, and just pass around both the manifest and any signatures associated with it?

[tianon]

also, won't signing the whitespace too break in really fun ways? ie, space vs tab, \n vs \r\n, etc etc etc.

[tianon]

does this also mean that image signatures can only be certificate-based and not leverage other existing systems of signing and trust like GPG that are already widely deployed and used?

[dmcgowan]

@tianon yes signing whitespace does allow for breaking in fun ways. The JWS spec gives us some protection against this by encoding the payload as base64. The "Pretty" version of it is both legible (not base64 encoded) and verifiable, but at the expense of it being fragile. I don't know if it is finalized which format will be used for the output, it is important for us to have a single file which includes the signature.

[dmcgowan]

We are using the JWS format for signatures, these signatures allow for x509 chain of trust verification or verification based solely off the public key. The initial verification of the namespace based on the signature will only use the public key, just like GPG-based trust.

See http://tools.ietf.org/html/draft-ietf-jose-json-web-signature-31#section-4.1.3

[thaJeztah]

Not directly related, just a (possibly) interesting read: Announcing Keyless SSL (and the discussion on HackerNews

[tianon]

So then shouldn't the signed version be completely extraneous whitespace-free?

[tianon]

ie, {"a":"b","c":"d"} instead of { "a": "b", "c": "d" }

[tianon]

I'm just reading through this and getting the feeling that it's really quite limiting. Maybe each of those "signature" blocks could have a "type" field to make sure there's space for future growth? (like maybe some kind of GPG backend too)

[dmcgowan]

As for whitespace, we are offering a format which is whitespace safe, but it is not preferred to use since it is not legible, just as formatting JSON without whitespace is illegible. I don't see a reason to assume that a signed file should be able to have any of its bytes changed (whitespace or not) and still be expected to pass signature validation.

I agree with a type field and I think we are still trying to figure out how multiple signatures will be handled. The GPG backend I think would come in later at verification. Right now we are thinking of libtrust more as a possible backend for GPG, rather than GPG a possible backend for libtrust.

[dmcgowan]

@thaJeztah it is an interesting read and although no directly related, there seems to be some similar motivations. The approach used in this proposal is very sensitive to the idea that builders should both be identified by their public key and solely responsible for their private key. At no point does the private key need to be shared with anyone else to build and verify builds.

[vbatts]

On Sep 18, 2014 5:21 PM, "Tianon Gravi" notifications@github.com wrote:

I'm just reading through this and getting the feeling that it's really
quite limiting. Maybe each of those "signature" blocks could have a "type"
field to make sure there's space for future growth? (like maybe some kind
of GPG backend too)

A type field trip s already in the works. I'll update the proposal. The
initial one would be builder/publisher, though there will be an attestor
and likely more.

[vbatts]

Also, there will be a directory like /etc/docker/ca.d/ for landing local root CAs. This way local builds of images for an enterprise can use an internal delegation for signing images and even promotion through their environments.
For instance, the image must be signed by a trusted builder, and QA can only run the image once also signed by the QA authority, and so on, such that in a "strict" mode Production would not accidentally run an image that had not be signed by a prod authority. Make sense?

[vbatts]

@dmcgowan @shykes I realize that one of the objectives is to have the docker binary be all inclusive, but including anything like a root CA in the binary is not going to be acceptable except for demos. A production environment will manage the distribution of trusted CAs land them on disk for the daemon to use.

[proppy]

@vbatts about this part:

Content Addressable Layers

Each layer of an image will be referenced by a checksum created from its
contents. This checksum will be used on push and pull to verify contents have
not been tampered and disallow the layer referred to in the manifest to be
changed after signed.

Assuming "the content" of a layer is defined by the diff with its parent, do you plan to store attribute change (i.e: modification datetime) separately from actual content changes? So that touch'ing or re-ADD'ing a file doesn't end up duplicate it in the current layer?

(Happy to open or comment on another issue if that's outside of the scope of this proposal).

[vbatts]

On Sep 19, 2014 5:15 PM, "Johan Euphrosine" notifications@github.com
wrote:

@vbatts about this part:

Content Addressable Layers

Each layer of an image will be referenced by a checksum created from its
contents. This checksum will be used on push and pull to verify contents have
not been tampered and disallow the layer referred to in the manifest to be
changed after signed.

Assuming "the content" of a layer is defined by the diff with its parent, do you plan to store attribute change (i.e: modification datetime) separately from actual content changes? So that touch'ing or re-ADD'ing a file doesn't end up duplicate it in the current layer?

I intentionally left that a little vague. For the foreseeable future this
hash will be the TarSum of just the filesystem of the layer (not including
the json payload as is sometimes the case). This includes the contents of
the files that are diff'd from a layer's parents.

(Happy to open or comment on another issue if that's outside of the scope of this proposal).

We've been giving the tarsum calculation a lot of attention lately, so
depending on what extent you want to iron put the hashes of the layers, it
is completely relevant here. :-)

[dmcgowan]

@vbatts I think it makes sense to have a directory for root CAs. These CAs could either be checked by default, turned on/off individually, or a flag to use them all. The CA we talked about bundling I would not categorize as the same type of CA, since this root CA is used to enforce the global namespace, so it shouldn't differ between installations. Certain operations may be able to turn off enforcement or allow a work around (a local namespace), but the bundled CA should not be replaced with a different CA. How do you propose bundling the root CA if not in the binary, since the binary is the primary method of bundling and installing?

[tianon]

Oh @vbatts you gem.

I'm -1 on CAs in general, but I see how that could be attractive. As long as there's an alternative to the rat's nest that x509 is, I'll be happy with this and might actually use it. If I can say "only allow my Docker engine to run image manifests that are signed/built/acked-by XYZ person (specified via some fully unambiguous and easily verified means like a full GPG fingerprint)", that'd be amazing.

I also see an absolute need to be able to disable the default CA, even if it can't be changed in any other way.

[dmcgowan]

@tianon the default root CA in this proposal is used is to verify that the builder has access to namespace of the image. The identity of the builder will be derived from the signature, which is created by the user's private key, just like GPG signatures. The verification is done on the fingerprint of the public key. Statements signed by a namespace authority and chaining back to the root will be able to be downloaded/imported which provide identity of public keys to user and users to namespace. Without these statements chaining back to a single root CA the trust graph will be unmanageable (like of web of trust) and difficult to tie in to the existing namespace. While x509 is not ideal and was not our first choice, it is a proven mechanism for extending authority in a secure and manageable way.

[mmdriley]

It's probably better to use a SHA256 of the layer file as it's stored (the "payload" hash in Docker client parlance) rather than the TarSum.

Two reasons:
(1) TarSum is super-custom and a spec for it hasn't been written as far as I can tell. This makes it really difficult to reason the algorithm or implementations of it correct. For example, in the current TarSum implementation, what happens if I put two instances of the same filename in a .tar file? I think one of them gets ignored.

(2) The storage layer (be it S3, GCS, or local) shouldn't need special insight into files in order to hash them.

[vbatts]

Since this issue was initially opened there have been huge strides made in a
content addressable Docker image format.

• Initial merges into docker-1.3 for an interoperable image
  manifest.
• A full rewrite of the Registry for the new
  API
• Documenting this provisional image manifest
• Deprecating TarSum for push and pull

This provisional image manifest allows for a more abstracted implementation and
will act as a compatibility layer for v1 images. Though as we refine multi-arch
images, federated image layers, content addressable references and fully
qualified docker images, the work has already begun for further refining this v2
image manifest.

I am closing the step 1 issue as complete. Further work will continue in the
distribution workgroup and eventually the security/trust workgroup.

[xuedihualu]

Hi:
how can i delete images from registry2,when i curl -v -X GET registry.com:5000/v2/centos/6.6

• About to connect() to registry.com port 5000 (#0)
• Trying 10.168.89.24...
• Connected to registry.com (10.168.89.24) port 5000 (#0)

  GET /v2/centos/6.6 HTTP/1.1
  User-Agent: curl/7.29.0
  Host: registry.com:5000
  Accept: /

  < HTTP/1.1 404 Not Found
  < Content-Type: text/plain; charset=utf-8
  < Docker-Distribution-Api-Version: registry/2.0
  < Date: Tue, 13 Oct 2015 06:04:20 GMT
  < Content-Length: 19
  <
  404 page not found

[thaJeztah]

@xiekeyang i noticed you asked the same question in distribution/distribution#1091 (comment). The GitHub issue trackers are not a support forum, Could you ask your question on;

1. the forums: https://forums.docker.com/
2. the docker-user mailing list https://groups.google.com/forum/#!forum/docker-user
3. the #docker channel on Freenode.