MANIFEST_UNKNOWN: The named manifest is not known to the registry

[bverkron]

Expected behavior

Attempting to compare two images. One image (281) was built with a RHEL version of docker on the host machine I'm running on and the other image (286) was built on another machine with docker-ce. They have considerably different sizes (2.6+GB vs ~700MB respectively) so I'm trying to find out why. Built from same Dockerfile. Image version :281 is the one built locally on the same machine container-diff is running on. It has not been uploaded to the remote registry as we are having problems with that (maybe network related maybe not). Regardless since the image exists locally container-diff should not be looking for it in the remote registry, which is what it appears to be doing.

container-diff diff docker-dev-local.artifactory.****.com/listener:1.1001.281 docker-dev-local.artifactory.****.com/listener:1.1001.286

Actual behavior

I get the following error. I suspect it's because container-diff is attempting to get it from the remote registry and it doesn't exist there yet (we had upload issues), only locally.

[user@kam1odbus136<>:~]$ container-diff diff docker-dev-local.artifactory.****.com/listener:1.1001.281 softserv-docker-dev-local.artifactory.****.com/listener:1.1001.286
ERRO[0000] error retrieving image docker-dev-local.artifactory.****.com/listener:1.1001.281: MANIFEST_UNKNOWN: The named manifest is not known to the registry.; map[manifest:listener/1.1001.281/manifest.json]

Information

• container-diff version: v0.17.0 built from git bea7ad9
• Operating system: RHEL 7.9

Steps to reproduce the behavior

1. ...
2. ...

[bverkron]

I tried the daemon:// designation I found in another issue and now I get this. Perhaps that's because one image was built with overlay2 and the other with devicemapper?

container-diff diff daemon://docker-dev-local.artifactory.****.com/listener:1.1001.281 daemon://docker-dev-local.artifactory.****.com/listener:1.1001.286
ERRO[0041] error retrieving image daemon://docker-dev-local.artifactory.****.com/listener:1.1001.281: retrieving image from daemon: Error response from daemon: open /var/lib/docker/overlay2/39c1428cfc7d9e2e4473c5b7e33d873432af124a32f1deb0460be89ec9201c1a/merged/etc/yum.repos.d/ubi.repo: no such file or directory

Although the image that's using overlay2 (which the error is complaining about) is the one built on this machine I'm running the commands on, which seems odd.

[bverkron]

Switched to a different image to compare (version 283 instead of 281) and now I get an out of memory error and a stack dump :/

fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xa451cf, 0x16)
        /usr/local/go/src/runtime/panic.go:1116 +0x72
runtime.sysMap(0xc104000000, 0x100000000, 0xeec0d8)
        /usr/local/go/src/runtime/mem_linux.go:169 +0xc5
runtime.(*mheap).sysAlloc(0xed7900, 0x100000000, 0xed7908, 0x80000)
        /usr/local/go/src/runtime/malloc.go:715 +0x1cd
runtime.(*mheap).grow(0xed7900, 0x80000, 0x0)
        /usr/local/go/src/runtime/mheap.go:1286 +0x11c
runtime.(*mheap).allocSpan(0xed7900, 0x80000, 0x100, 0xeec0e8, 0xfffffffffffffade)
        /usr/local/go/src/runtime/mheap.go:1124 +0x6a0
runtime.(*mheap).alloc.func1()
        /usr/local/go/src/runtime/mheap.go:871 +0x64
runtime.(*mheap).alloc(0xed7900, 0x80000, 0xc0000a0101, 0xc0000aea80)
        /usr/local/go/src/runtime/mheap.go:865 +0x81
runtime.largeAlloc(0xfffffe00, 0xc0000d0101, 0x43e2c7)
        /usr/local/go/src/runtime/malloc.go:1152 +0x92
runtime.mallocgc.func1()
        /usr/local/go/src/runtime/malloc.go:1047 +0x46
runtime.systemstack(0xc0000aa418)
        /usr/local/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
        /usr/local/go/src/runtime/proc.go:1056

goroutine 18 [running]:
runtime.systemstack_switch()
        /usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc00004d5d8 sp=0xc00004d5d0 pc=0x465340
runtime.mallocgc(0xfffffe00, 0x959680, 0x1, 0x0)
        /usr/local/go/src/runtime/malloc.go:1046 +0x895 fp=0xc00004d678 sp=0xc00004d5d8 pc=0x40cad5
runtime.makeslice(0x959680, 0xfffffe00, 0xfffffe00, 0xc00001e280)
        /usr/local/go/src/runtime/slice.go:49 +0x6c fp=0xc00004d6a8 sp=0xc00004d678 pc=0x44c50c
bytes.makeSlice(0xfffffe00, 0x0, 0x0, 0x0)
        /usr/local/go/src/bytes/buffer.go:229 +0x73 fp=0xc00004d6e8 sp=0xc00004d6a8 pc=0x4ef683
bytes.(*Buffer).grow(0xc00004d7e0, 0x200, 0x7e00)
        /usr/local/go/src/bytes/buffer.go:142 +0x15b fp=0xc00004d738 sp=0xc00004d6e8 pc=0x4eefcb
bytes.(*Buffer).ReadFrom(0xc00004d7e0, 0xb093c0, 0xc00001e2c0, 0xb093c0, 0x1, 0xb100a0)
        /usr/local/go/src/bytes/buffer.go:202 +0x48 fp=0xc00004d7a8 sp=0xc00004d738 pc=0x4ef478
io/ioutil.readAll(0xb093c0, 0xc00001e2c0, 0x200, 0x0, 0x0, 0x0, 0x0, 0x0)
        /usr/local/go/src/io/ioutil/ioutil.go:36 +0xe3 fp=0xc00004d828 sp=0xc00004d7a8 pc=0x594a03
io/ioutil.ReadAll(...)
        /usr/local/go/src/io/ioutil/ioutil.go:45
github.com/google/go-containerregistry/pkg/v1/daemon.bufferedOpener(0xb18ca0, 0xc00001e080, 0x0, 0x0, 0x0)
        /workspace/vendor/github.com/google/go-containerregistry/pkg/v1/daemon/image.go:101 +0xd0 fp=0xc00004d8a8 sp=0xc00004d828 pc=0x865530
github.com/google/go-containerregistry/pkg/v1/daemon.(*imageOpener).Open(0xc00000c0c0, 0x0, 0x0, 0x4c3c66, 0x62f6d84d)
        /workspace/vendor/github.com/google/go-containerregistry/pkg/v1/daemon/image.go:51 +0x45 fp=0xc00004d8f8 sp=0xc00004d8a8 pc=0x8651b5
github.com/google/go-containerregistry/pkg/v1/daemon.Image(0xb18ca0, 0xc00001e080, 0xc00004dac8, 0x1, 0x1, 0x0, 0x0, 0xc000068280, 0x78)
        /workspace/vendor/github.com/google/go-containerregistry/pkg/v1/daemon/image.go:132 +0xca fp=0xc00004d940 sp=0xc00004d8f8 pc=0x8657ca
github.com/GoogleContainerTools/container-diff/pkg/util.GetImage(0xc000068280, 0x78, 0xc00006e000, 0xc00006e000, 0x9a, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /workspace/pkg/util/image_utils.go:100 +0x24f fp=0xc00004dc98 sp=0xc00004d940 pc=0x87388f
github.com/GoogleContainerTools/container-diff/cmd.getImage(0x7ffe997957b1, 0x81, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /workspace/cmd/root.go:154 +0xf4 fp=0xc00004de28 sp=0xc00004dc98 pc=0x8ec5a4
github.com/GoogleContainerTools/container-diff/cmd.processImage(0x7ffe997957b1, 0x81, 0xc00020cc60, 0x0)
        /workspace/cmd/diff.go:77 +0xa3 fp=0xc00004df70 sp=0xc00004de28 pc=0x8ea973
github.com/GoogleContainerTools/container-diff/cmd.diffImages.func1(0xc0002da560, 0x7ffe997957b1, 0x81, 0xc00020cc60, 0xc0000aa410)
        /workspace/cmd/diff.go:114 +0x6b fp=0xc00004dfb8 sp=0xc00004df70 pc=0x8ee25b
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1373 +0x1 fp=0xc00004dfc0 sp=0xc00004dfb8 pc=0x4672d1
created by github.com/GoogleContainerTools/container-diff/cmd.diffImages
        /workspace/cmd/diff.go:112 +0x2e3

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc0002da568)
        /usr/local/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc0002da560)
        /usr/local/go/src/sync/waitgroup.go:130 +0x64
github.com/GoogleContainerTools/container-diff/cmd.diffImages(0x7ffe997957b1, 0x81, 0x7ffe99795833, 0x81, 0xc00027fda0, 0x1, 0x1, 0x0, 0x0)
        /workspace/cmd/diff.go:121 +0x34d
github.com/GoogleContainerTools/container-diff/cmd.glob..func4(0xeb53c0, 0xc0002192c0, 0x2, 0x2)
        /workspace/cmd/diff.go:49 +0x89
github.com/spf13/cobra.(*Command).execute(0xeb53c0, 0xc000219260, 0x2, 0x2, 0xeb53c0, 0xc000219260)
        /workspace/vendor/github.com/spf13/cobra/command.go:766 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0xeb5620, 0x1f, 0x0, 0x0)
        /workspace/vendor/github.com/spf13/cobra/command.go:852 +0x2ea
github.com/spf13/cobra.(*Command).Execute(...)
        /workspace/vendor/github.com/spf13/cobra/command.go:800
main.main()
        /workspace/main.go:33 +0x73

goroutine 6 [select]:
net/http.(*persistConn).readLoop(0xc000020120)
        /usr/local/go/src/net/http/transport.go:2158 +0x9a0
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1706 +0xc56

goroutine 7 [select]:
net/http.(*persistConn).writeLoop(0xc000020120)
        /usr/local/go/src/net/http/transport.go:2336 +0x11c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1707 +0xc7b

goroutine 21 [IO wait]:
internal/poll.runtime_pollWait(0x7f313d394e08, 0x72, 0xffffffffffffffff)
        /usr/local/go/src/runtime/netpoll.go:203 +0x55
internal/poll.(*pollDesc).wait(0xc0002ec418, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc0002ec400, 0xc0002f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b
net.(*netFD).Read(0xc0002ec400, 0xc0002f8000, 0x1000, 0x1000, 0x43736c, 0xc000090b58, 0x463e00)
        /usr/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000aa440, 0xc0002f8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/net.go:184 +0x8e
net/http.(*persistConn).Read(0xc0000a0fc0, 0xc0002f8000, 0x1000, 0x1000, 0xc000082540, 0xc000090c58, 0x405075)
        /usr/local/go/src/net/http/transport.go:1884 +0x75
bufio.(*Reader).fill(0xc00020cf60)
        /usr/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc00020cf60, 0x1, 0x0, 0x0, 0x1, 0x0, 0xc000078540)
        /usr/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*persistConn).readLoop(0xc0000a0fc0)
        /usr/local/go/src/net/http/transport.go:2037 +0x1a8
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1706 +0xc56

goroutine 22 [select]:
net/http.(*persistConn).writeLoop(0xc0000a0fc0)
        /usr/local/go/src/net/http/transport.go:2336 +0x11c
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1707 +0xc7b