Resolve several open issues

[pmengelbert]

Built on top of PR #204

Resolves #193 #26 #60 #68 #79 #126 #132 #183 #186 #195 #201 #202

[jonjohnsonjr]

The (and associated Blob) here no longer makes sense.

and describes the Manifest

Does it?

[pmengelbert]

I adjusted the language here, thanks for the suggestion.

[dmcgowan]

Do you have an update to push with the updated language?

[pmengelbert]

oops 🥵 . Pushed now

[dmcgowan]

Where did this wording come from related to using the digest?

The previous language was here https://github.com/opencontainers/distribution-spec/blob/f67bc11ba3a083a9c62f8fa53ad14c5bcf2116af/spec.md#cross-repository-blob-mount and is more clear. A client should NEVER use a digest from the registry without verifying it matches. Normally the client will just ignore this, I don't think it is correct to say the client should use it and verifying is expensive. Most registries use sha256 anyway, the case where it would differ is if the registry was using a different hashing algorithm.

[jonjohnsonjr]

Most registries use sha256 anyway, the case where it would differ is if the registry was using a different hashing algorithm.

It would be great to be more explicit about this in the spec. The digest canonicalization section in the original spec seems mostly to do with signed schema 1 manifests:

To provide verification of http content, any response MAY include a Docker-Content-Digest header. This will include the digest of the target entity returned in the response. For blobs, this is the entire blob content. For manifests, this is the manifest body without the signature content, also known as the JWS payload. Note that the commonly used canonicalization for digest calculation MAY be dependent on the mediatype of the content, such as with manifests.

Though the section about different algorithms is good and maybe we should copy that over:

The client MAY choose to ignore the header or MAY verify it to ensure content integrity and transport security. This is most important when fetching by a digest. To ensure security, the content SHOULD be verified against the digest used to fetch the content. At times, the returned digest MAY differ from that used to initiate a request. Such digests are considered to be from different domains, meaning they have different values for algorithm. In such a case, the client MAY choose to verify the digests in both domains or ignore the server's digest. To maintain security, the client MUST always verify the content against the digest used to fetch the content.

IMPORTANT: If a digest is used to fetch content, the client SHOULD use the same digest used to fetch the content to verify it. The header Docker-Content-Digest SHOULD NOT be trusted over the "local" digest.

We already link out to https://github.com/opencontainers/image-spec/blob/v1.0.1/descriptor.md#digests but that links to https://github.com/opencontainers/image-spec/blob/v1.0.1/canonicalization.md#canonicalization which is a 404 🤷 Looks like it should be https://github.com/opencontainers/image-spec/blob/v1.0.1/considerations.md#canonicalization (fixed at HEAD) which links to https://github.com/opencontainers/image-spec/blob/v1.0.1/considerations.md#canonicalization but I don't see anything here that would make me think "oh, I should make sure everything is using the same algorithm", especially in the context of a client-registry interaction

[pmengelbert]

Just pushed a commit that reflects these suggestions.

[dmcgowan]

I agree, there may be a little bit of gap in specification around using sha256. There are a few spots where there is an assumption made and if the registry were to use a different algorithm, it could cause some issues. When starting an upload, the client should know the desired digest (but there are cases it doesn't). If it doesn't, and the client/registry are using different algorithms, the finalization would require the registry to either recalculate the digest, calculate it using multiple algorithms, or reject. In the case of cross repository mount, the digest is likely referenced in a manifest and it is not an option to use a different digest that the registry returns.

[jonjohnsonjr]

This is also described in image-spec if you want to link to that (similar to how manifest links to apdx-2).

[pmengelbert]

Thanks again, just pushed changes addressing this and the above