docker manifest create allows duplicate arch entries

[hassenius]

Description

I've seen a few cases that if docker manifest push --purge was not completed successfully, the next docker manifest create will end up creating manifest.list.json with duplicated entries.

Having duplicated entries for the same architecture / os combination should not to my mind be a valid state, as it is impossible to predict which entry will be consumed (if there are arch specific builds between these two points)

Steps to reproduce the issue:

1. docker manifest create hk-manifest-test:invalid amd64/ubuntu:latest ppc64le/ubuntu:latest
2. docker manifest create --amend hk-manifest-test:invalid amd64/ubuntu:devel
3. docker manifest inspect hk-manifest-test:invalid

Describe the results you received:

manifest.list.json with duplicate entries for amd64 / linux (but with different digests)

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 943,
         "digest": "sha256:0b812eddc275979da443e4ab51958fb5c64c5d8d6db693cead7bc5d1c07f5386",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 943,
         "digest": "sha256:2e70e9c81838224b5311970dbf7ed16802fbfe19e7a70b3cbfa3d7522aa285b4",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 943,
         "digest": "sha256:57a6583b62e348979659d3bb3c34e6ea04a19a2a1365527add479ed69d9bfa92",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      }
   ]
}

Describe the results you expected:

Single item for amd64 / linux

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 943,
         "digest": "sha256:2e70e9c81838224b5311970dbf7ed16802fbfe19e7a70b3cbfa3d7522aa285b4",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 943,
         "digest": "sha256:57a6583b62e348979659d3bb3c34e6ea04a19a2a1365527add479ed69d9bfa92",
         "platform": {
            "architecture": "ppc64le",
            "os": "linux"
         }
      }
   ]
}

Additional information you deem important (e.g. issue happens only occasionally):

The example above is somewhat artificial, but I've seen manifest.list.jsons in the wild with duplicated entries.

Output of docker version:

 $ docker version
Client: Docker Engine - Community
 Azure integration  0.1.15
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:41:33 2020
 OS/Arch:           darwin/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:49:27 2020
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Additional environment details (AWS, VirtualBox, physical, etc.):

I've seen this behaviour in a range of environments.

[mbentley]

I've recently come across this sort of issue where I am taking an existing image that is tagged with a major.minor.bugfix and i am just wanting to write the manifest as a tag with major.minor.

I end up with two manifests for amd64/linux which doesn't make sense to me:

$ docker manifest inspect mbentley/grafana:8.2
{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2203,
         "digest": "sha256:a211f009fdfa95384b543f3b3be9aa2d384f3f7ded6f6097933547105e95478a",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 2203,
         "digest": "sha256:d700154590e348bd441641dcb261fcafd5b644450552e781fd1fbe6cb724295f",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      }
   ]
}

Maybe it is something about the flags I am using to create the manifest but I am just doing:

docker manifest create "mbentley/grafana:${MAJOR_MINOR_TAG}" --amend "grafana/grafana@${TAG_DIGEST}"
docker manifest push "mbentley/grafana:${MAJOR_MINOR_TAG}"

I suppose I could add the --purge flag to my push but that seems like it isn't reliable at all times.

The only other workaround I've seen is to make sure to also do a docker manifest rm "mbentley/grafana:${MAJOR_MINOR_TAG}" || true before creating my manifest to ensure it is creating the manifest but I believe there are some issues with that which would trigger tools like watchtower to possibly re-create a container as reported here: mbentley/docker-omada-controller#124.

I haven't taken the time to try to reproduce this but it seems like possibly re-writing a manifest, even if nothing changes for the targeted architectures or digests of their images can trigger watchtower to take action.