image digests cannot be specified for docker image builds

[EvanTheB]

Version of Singularity:

singularity --version
singularity version 3.2.0-rc2

Expected behavior

Specify a digest to build a specific docker image into singularity image.

https://sylabs.io/guides/3.2/user-guide/appendix.html#docker-bootstrap-agent

Actual behavior

Error message and failure.

Steps to reproduce behavior

evaben@evaben ~/c/crammer (master)> docker images debian 
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
debian              stretch             2d337f242f07        3 months ago        101MB
debian              jessie              7cd9fb1ee74f        3 months ago        129MB
debian              latest              be2868bebaba        9 months ago        101MB

evaben@evaben ~/c/crammer (master)> singularity build tmp docker-daemon://debian:latest@2d337f242f07
INFO:    Starting build...
FATAL:   While performing build: conveyor failed to get: Invalid image source: invalid reference format

evaben@evaben ~/c/crammer (master) [255]> singularity build tmp docker-daemon://debian:@2d337f242f07
INFO:    Starting build...
FATAL:   While performing build: conveyor failed to get: Invalid image source: invalid reference format

evaben@evaben ~/c/crammer (master) [255]> singularity build tmp docker-daemon://debian@2d337f242f07
INFO:    Starting build...
FATAL:   While performing build: conveyor failed to get: Invalid image source: invalid reference format

[jscook2345]

@EvanTheB

The digest you are using appears to be for the stretch tag. Did you try using that tag?

[EvanTheB]

evaben@evaben /tmp> singularity build tmp docker-daemon://debian:latest
INFO:    Starting build...
...works

evaben@evaben /tmp> singularity build tmp docker-daemon://debian:stretch
INFO:    Starting build...
...works

evaben@evaben /tmp> docker images debian
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
debian              stretch             2d337f242f07        4 months ago        101MB
debian              jessie              7cd9fb1ee74f        4 months ago        129MB
debian              latest              be2868bebaba        9 months ago        101MB

evaben@evaben /tmp> singularity build tmp docker-daemon://debian:stretch@2d337f242f07
INFO:    Starting build...
FATAL:   While performing build: conveyor failed to get: Invalid image source: invalid reference format

To me it looks like the parsing of the reference format is what is failing. Perhaps my sytax is wrong. The manual states:

From: <registry>/<namespace>/<container>:<tag>@<digest>

But provides no example of use.

docker://debian:stretch@2d337f242f07

Seems to match the format to me, but it does not work. What is the correct format?

[jscook2345]

Here is what I see in the docs as a means of example. But it appears you're using something similar there from your last comment.

https://sylabs.io/guides/3.3/user-guide/singularity_and_docker.html?highlight=docker%20daemon#locally-available-images-cached-by-docker

@ikaneshiro Can you confirm digest works as expected here?

[dtrudg]

Noting here that I believe the correct format is to get the full digest via...

docker image inspect alpine --format '{{.RepoDigests}}'
[alpine@sha256:72c42ed48c3a2db31b7dafe17d275b634664a708d901ec9fd57b1529280f01fb]

And then pull it like so.... but this is erroring...

06:04 PM $ singularity build test.sif docker-daemon://alpine@sha256:72c42ed48c3a2db31b7dafe17d275b634664a708d901ec9fd57b1529280f01fb
Build target already exists. Do you want to overwrite? [N/y] y
INFO:    Starting build...
FATAL:   While performing build: conveyor failed to get: Error initializing source oci:/home/dave/.singularity/cache/oci:f0572185c4e54b3944acd894b5b8abf3db18d9adc28d62995779a09df4bda30b: Error determining manifest MIME type for docker-daemon:alpine@sha256:72c42ed48c3a2db31b7dafe17d275b634664a708d901ec9fd57b1529280f01fb: Manifest does not match provided manifest digest sha256:72c42ed48c3a2db31b7dafe17d275b634664a708d901ec9fd57b1529280f01fb

[ikaneshiro]

@EvanTheB
Also noting that you can pull from a docker registry via a full digest like so:

$ singularity build test.sif docker://ubuntu@sha256:ca013ac5c09f9a9f6db8370c1b759a29fe997d64d6591e9a75b71748858f7da0
Build target already exists. Do you want to overwrite? [N/y] y
INFO:    Starting build...
application/vnd.docker.distribution.manifest.v2+json
application/vnd.docker.distribution.manifest.v2+json
Getting image source signatures
Skipping fetch of repeat blob sha256:35c102085707f703de2d9eaad8752d6fe1b8f02b5d2149f1d8357c9cc7fb7d0a
Skipping fetch of repeat blob sha256:251f5509d51d9e4119d4ffb70d4820f8e2d7dc72ad15df3ebd7cd755539e40fd
Skipping fetch of repeat blob sha256:8e829fe70a46e3ac4334823560e98b257234c23629f19f05460e21a453091e6d
Skipping fetch of repeat blob sha256:6001e1789921cf851f6fb2e5fe05be70f482fe9c2286f66892fe5a3bc404569c
Copying config sha256:ac8e8215f8099eeae08e66daac9bd100ae50801c63fe6c5c95fe106c821c5ab9
 2.42 KiB / 2.42 KiB [======================================================] 0s
Writing manifest to image destination
Storing signatures
application/vnd.oci.image.manifest.v1+json
application/vnd.oci.image.manifest.v1+json
2019/09/11 14:53:39  info unpack layer: sha256:35c102085707f703de2d9eaad8752d6fe1b8f02b5d2149f1d8357c9cc7fb7d0a
2019/09/11 14:53:40  info unpack layer: sha256:251f5509d51d9e4119d4ffb70d4820f8e2d7dc72ad15df3ebd7cd755539e40fd
2019/09/11 14:53:40  info unpack layer: sha256:8e829fe70a46e3ac4334823560e98b257234c23629f19f05460e21a453091e6d
2019/09/11 14:53:40  info unpack layer: sha256:6001e1789921cf851f6fb2e5fe05be70f482fe9c2286f66892fe5a3bc404569c
INFO:    Creating SIF file...
INFO:    Build complete: test.sif

@dctrud Unsure why this doesn't work for the docker daemon. Looking into why.

[carterpeel]

Hello,

This is a templated response that is being sent out to all open issues. We are working hard on 'rebuilding' the Singularity community, and a major task on the agenda is finding out what issues are still outstanding.

Please consider the following:

1. Is this issue a duplicate, or has it been fixed/implemented since being added?
2. Is the issue still relevant to the current state of Singularity's functionality?
3. Would you like to continue discussing this issue or feature request?

Thanks,
Carter