Skopeo copy from docker hub gets a different hash than using Docker pull

[Kenzku]

I used skopeo copy to get image from docker:

skopeo copy \
docker://docker.io/ibmcom/icp-transformation-advisor-ui:2.3.0-test-1-amd64 \
dir:/Users/xxx/Downloads/test/transformation-advisor-operator/2.3.0-test-1-amd64

I then tried to verify it:

skopeo standalone-verify \
/Users/xxx/Downloads/test/transformation-advisor-operator/2.3.0-test-1-amd64/manifest.json \
ibmcom/icp-transformation-advisor-ui:2.3.0-test-1-amd64 \
7D9108FAF55272DCF922EE1650C2D9CFF1A2F295 \
signature-2.3.0

I got error saying that:

Error verifying signature: Signature for docker digest "sha256:e4acb2be7755fda509e3f7b01eabc5ff389fc729b5cb004f94b05d880ea2fd52" does not match

However, if I tried to pull the image with docker pull:

docker pull ibmcom/icp-transformation-advisor-ui:2.3.0-test-1-amd64

Then, use the skopeo copy from local docker:

skopeo copy --dest-tls-verify=false \
docker-daemon:docker.io/ibmcom/icp-transformation-advisor-ui:2.3.0-test-1-amd64 \
dir:/Users/xxx/Downloads/test/docker

then I can verify the images with the skopeo standalone-verify

I noticed that the layers are different between the 2 ways of pulling images.

I was wondering if I did something wrong, or is there an issue in the skopeo copy?

[mtrmac]

Thanks for your report. I don’t have access to the signature file, and I don’t know how it was created, so I’m somewhat guessing; still:

The reported issue failure is that the signature signs a manifest digest contained in the error message (e4acb2be7755…), but the image on the registry actually has a different one (I see 318233145b6…). So the signature is correctly rejected as not matching.

The signature signs a specific representation of the image (the specific way the layers have been compressed, and the resulting manifest — the specific representation of that manifest).

It seems that the signature was created to sign the docker-daemon: representation (or an equivalent docker-archive: representation); that’s very rarely the same as a representation on the registry (notably because, as you noticed, the layers are different: docker-daemon:/docker-archive: uses uncompressed layers, but layers on registries are almost always compressed; but also the docker-archive format does not have a schema2-formatted manifest at all, when copying such images to dir: or to registries a manifest is generated on the fly, and is thus conceptually dependent on the version of Skopeo.

The typical workflow to create signatures is:

• For just-built images: podman push --sign-by=…, letting the code compress the image, create a manifest, and immediately create a matching signature.
• For pre-existing images on a registry (if the source registry is trusted to contain the right version of the image!): skopeo copy --sign-by=… docker://… docker://….

It is of course possible to build this manually using skopeo standalone-sign or using some other tools to create a signature in the expected format, but the signature still must match the manifest as it exists on the registry. (And in the just-built case, that requires the registry to be trusted not to maliciously modify the image in the time between the upload and creating the signature; e.g. it could be a single-purpose registry running in a localhost-only container.)

[Kenzku]

Hej, thanks for replying. The explanation on the compressed part looks like answered my original question.

I don’t have access to the signature file, and I don’t know how it was created,

Apology for the inconvenience, I did not think you would try to reproduce at your end, but it is a public GitHub repo:

https://github.com/TransformationAdvisor/public/tree/master/ibmcom/icp-transformation-advisor-ui_2.3.0-test-1-amd64

It seems that the signature was created to sign the docker-daemon:

yes Indeed. Here is how I did it, and please correct me if they do not fall into the signing convension process:

1. I used docker build to build the image,
   but for this image, I used `operator-sdk build` (https://github.com/operator-framework/operator-sdk)

2. I copy the built image from local docker repo to a location (let's say localtion A) using:

    skopeo copy --dest-tls-verify=false ocker-daemon ...

3. I signed the image with `standalone-sign` in the location A
4. I use `skopeo copy` to copy the image from location A to Docker Hub

and I don’t know how it was created, so I’m somewhat guessing;

# copy the image from docker deamon
# shellcheck disable=SC2154
skopeo copy --dest-tls-verify=false \
docker-daemon:docker.io/${OPERATOR_IMAGE_NAME}-${PLATFORM} \
dir:$working_dir/image_signing/current_image/ || exit 1

# shellcheck disable=SC2154
# Sign the image
# ${OPERATOR_IMAGE_NAME}-${PLATFORM} is the image repo and tag e.g. ibmcom/icp-transformation-advisor-ui:2.3.0-test-1-amd64
# public_key=7D9108FAF55272DCF922EE1650C2D9CFF1A2F295 is the public key finger print.
sudo skopeo standalone-sign $working_dir/image_signing/current_image/manifest.json \
${OPERATOR_IMAGE_NAME}-${PLATFORM} \
${public_key} \
-o $working_dir/image_signing/current_image/signature || exit 1

# verify signature
# here it verifies OK
sudo skopeo standalone-verify \
$working_dir/image_signing/current_image/manifest.json \
${OPERATOR_IMAGE_NAME}-${PLATFORM} \
${public_key} \
$working_dir/image_signing/current_image/signature || exit 1

For just-built images: podman push --sign-by=…

Indeed, we want to integrated the build and sign process into the pipeline. Are you suggesting that the podman can be used to replace the skopeo?

Can you please give me an example to use the
podman in the build, sign and push image to Docker hub flow.

is thus conceptually dependent on the version of Skopeo.

I used the Skopeo 0.1.40 to sign, does it requires to the same version to verify?
What is the roadmap on the future version's backward compatibility please?

[mtrmac]

1. I used docker build to build the image,
2. I copy the built image from local docker repo to a location (let's say localtion A) using:
3. I signed the image with standalone-sign in the location A
4. I use skopeo copy to copy the image from location A to Docker Hub

That could work in principle, but it’s not quite optimal: step 2 will create an image with uncompressed layers, and the image would have to be uploaded to the Docker Hub unmodified=uncompressed.

Do you need to use standalone-sign? It’s not the default workflow, more of an escape hatch for people doing something less usual. There may well be other constraints, but as is, it seems you are manually doing work that the tools should be able to do for you.

A plain skopeo copy --sign-by=… dir:… docker:// could replace steps 3+4, and even better a skopeo copy --sign-by=… docker-daemon:… docker:// could replace 2+3+4. In both cases, the image would be compressed before signing.

skopeo copy --dest-tls-verify=false \
docker-daemon:docker.io/${OPERATOR_IMAGE_NAME}-${PLATFORM} \
dir:$working_dir/image_signing/current_image/ || exit 1

(--dest-tls-verify does nothing for dir:, but that doesn’t really matter)

# ${OPERATOR_IMAGE_NAME}-${PLATFORM} is the image repo and tag e.g. ibmcom/icp-transformation-advisor-ui:2.3.0-test-1-amd64

This works, but I’d recommend using a fully-qualified name docker.io/ibmcom/…. The semantics is the same. (Skopeo/Podman’s --sign-by normalizes it automatically.)

sudo skopeo standalone-sign $working_dir/image_signing/current_image/manifest.json \
${OPERATOR_IMAGE_NAME}-${PLATFORM} \
${public_key} \
-o $working_dir/image_signing/current_image/signature || exit 1

Here’s the most immediate problem. The signature file name is not used by the dir: format, so in step 4 skopeo copy does not recognize the image as signed, so it does not know it should not compress the image. I expect naming the file signature-1 would allow the image copied in step 4 to be verifiable. (But, see above, it’s not the optimal way to create an image.)

For just-built images: podman push --sign-by=…

Indeed, we want to integrated the build and sign process into the pipeline. Are you suggesting that the podman can be used to replace the skopeo?

It’s more of a docker replacement, but it does integrate signing, so you may not need to use skopeo at all.

Can you please give me an example to use the
podman in the build, sign and push image to Docker hub flow.

podman build …; podman push --sign-by …; other than the --sign-by argument, the rest would typically be the same as with docker.

Actually, from a quick look (no personal experience), it should be possible to use operator-sdk build --image-builder podman, which uses podman build instead of docker build; so all you would need to do after operator-sdk build would be the podman push --sign-by.

is thus conceptually dependent on the version of Skopeo.

I used the Skopeo 0.1.40 to sign, does it requires to the same version to verify?
What is the roadmap on the future version's backward compatibility please?

For the specific case of docker-archive:/docker-daemon:: There is no promise that the in-memory manifest that is created as an implementation artifact for docker-archive/docker-daemon transports will remain consistent, across no versions, not even across repeated runs of the same version of the executable. docker-archive/docker-daemon-resident images can’t be signed because they just don’t have a manifest as a persistent artifact.

(OTOH, once the image is copied out of there, e.g. the skopeo copy docker-daemon:… dir:…, a “stable” manifest exists in actual storage, and that can be signed. It is docker-daemon-resident, not docker-daemon-created.

So, your workflow that copies the image out of docker-daemon: once , and then signs it, can work fine; a hypothetical workflow that did skopeo copy docker-daemon:… dir:…, created a signature for the created files, and then did skopeo copy docker-daemon:… docker://…, would not be guaranteed to be reliable.)

[mtrmac]

Could you maybe, instead of examining effects of a specific sequence of commands, describe what you ultimately need to do, i.e. what are the inputs to the process that can’t be changed, what are the necessary outputs of the process that can’t be changed, and what are any other constraints?

[Kenzku]

Hej,

Thanks for your reply.

Speaking of my need, we produce images on amd64, power and z platforms, and I want to get the image on amd64 platform signed. This 3 images will be combined into a fat manifest, so in total 3 images plus 1 fat manifest will be pushed to Docker Hub.

When customers came, they can get the fat manifest pull down and verify the signature with our public key, public cert and chain 0 cert, if they are pull from an amd64 platform.

And if the images were copy from docker hub into other registry, the customers shall be still able to verify the signature with our public key, public cert and chain 0 cert.

As for skopeo copy --sign-by=, thanks for that suggestion and indeed it would be very simple and cut other steps away. I was following the guild line from one of our senior security architects, to use the standalone-sign, in our use cases.

A question here, how did you get the signature file, if you use skopeo copy --sign-by= please (I assumed podman will face the same issue)?

I can see the following message when I use skopeo copy --sign-by= and then when I skopeo copy the image back (without --sign-by=),

# copy to Docker hub
Writing manifest to image destination
Signing manifest
...
# copy from Docker hub
Writing manifest to image destination
Storing signatures

but I have no clue where to extract the signature file.

if I pull back with --sign-by=, I got error like:

skopeo copy \
--sign-by=7D9108FAF55272DCF922EE1650C2D9CFF1A2F295 \
docker://docker.io/ibmcom/icp-transformation-advisor-ui:2.3.0-test-2-amd64 \
dir:/root/Downloads/image_signing/transformation-advisor-operator/2.3.0

FATA[0012] Cannot determine canonical Docker reference for destination

# although I can see of the layers have been pull down to the destination 

In the issue #953 (comment), I saw you commented:

skopeo copy --all --sign-by should work for an already-created multi-arch image,

Do you meant to skopeo copy all 3 platforms to the destination "docker hub" at the same time will sign the fat manifest? if so, that is handy.

The problem we are facing is the 3 platforms images were produced on a different VM at different time, I would face difficulty to gather the 3 images at the build time.

[mtrmac]

When customers came, they can get the fat manifest pull down and verify the signature with our public key, public cert and chain 0 cert, if they are pull from an amd64 platform.

(That’s not how podman pull / CRI-O work; they only verify the signature of the architecture-specific image. You might have a custom image consumption workflow that cares about the manifest list and not the individual images, but then again, maybe not.)

A question here, how did you get the signature file, if you use skopeo copy --sign-by= please (I assumed podman will face the same issue)?

It’s either stored on the registry directly, or in a sigstore-staging location; see containers-registries.d(5).

if I pull back with --sign-by=, I got error like:

skopeo copy \
--sign-by=7D9108FAF55272DCF922EE1650C2D9CFF1A2F295 \
docker://docker.io/ibmcom/icp-transformation-advisor-ui:2.3.0-test-2-amd64 \
dir:/root/Downloads/image_signing/transformation-advisor-operator/2.3.0

FATA[0012] Cannot determine canonical Docker reference for destination

--sign-by uses the “identity” (reference) for the destination, and local directories don’t have one. But if you are just pulling images to consume them, you wouldn’t typically use --sign-by.

Yes, this makes a dir: location hard to use as a staging location for storing the results of signing; that’s where standalone-sign might come in handy, and there’s a RFE around to allow the user to specify an identity instead of inferring it.

skopeo copy --all --sign-by should work for an already-created multi-arch image,

Do you meant to skopeo copy all 3 platforms to the destination "docker hub" at the same time will sign the fat manifest? if so, that is handy.

The problem we are facing is the 3 platforms images were produced on a different VM at different time, I would face difficulty to gather the 3 images at the build time.

Reportedly podman manifest allows combining the images, and then podman manifest push --sign-by; I don’t have direct experience.

[Kenzku]

Hej,

Thanks for your response, looks like I wont be able to get the signature files from docker hub. skopeo copy --sign-by= is probably not suitable for the use case.

[mtrmac]

@Kenzku I’m not quite clear on the state of the issue. Is there anything specific we can help with?

[rhatdan]

Since we never got more information, I am closing, reopen if more info is available.