garbage-collect doesn't work on multi-arch docker image

[huxiaoliang]

reproduce steps:

garbage-collect works well on single arch docker image

1. start the docker registry server

docker run  -e REGISTRY_STORAGE_DELETE_ENABLED=True  -d  -p 5000:5000  -v /usr/local/registry:/var/lib/registry  --restart=always  --name registry  registry:2.6.2

2. set the docker registry as insecure mode from client side, then restart docker

[root@xhu-proxy1 test]# cat /etc/docker/daemon.json 
{ "insecure-registries":[ "xhu-proxy1:5000"] }

3. push an image to registry

docker pull tomcat:8.5
docker tag tomcat:8.5 xhu-proxy1:5000/test/tomcat-amd64:8.5
docker push  xhu-proxy1:5000/test/tomcat-amd64:8.5

4. list the image by registry v2 api using below scripts

[root@xhu-proxy1 opt]#  ./docker-registry.sh http://xhu-proxy1:5000 list
{
    "repositories": [
        "test/tomcat-amd64"
    ]
}

///////////////////////
#!/bin/bash
set -euo pipefail

CMD=$0

function usage {
    cat <<EOU

    Usage: 

    $CMD REGISTRY_BASE_URL ACTION [OPTIONS..]
    
    Actions:

    - list               list repos

    - list REPO          list tags for repo

    - delete REPO TAG    delete tag for repo

    Example:

    List all repos
        /$ $CMD https://registry.my.domain list

    List tags for one repo
        /$ $CMD https://registry.my.domain list some-repo

    Delete tag for a repo
        /$ $CMD https://registry.my.domain delete some-repo

EOU
    exit 1
}

[ $# -lt 2 ] && usage

set +e
PROTO="$(echo $1 | grep :// | sed -e's,^\(.*://\).*,\1,g')"
set -e

[ -z "$PROTO" ] && >&2 echo "ERROR: Must have protocol in registry url" && usage

# remove the protocol
REG="$(echo ${1/$PROTO/})"
shift
ACTION="$1"
shift
case "$ACTION" in
    list)
        if [ $# -eq 1 ]; then
            repo=${1}
            if [ -n "$repo" ]; then
                curl -k -s $PROTO$REG/v2/$repo/tags/list | python -m json.tool 
            fi
        else
            curl -k -s $PROTO$REG/v2/_catalog?n=4096 | python -m json.tool 
        fi

        ;;
    delete)
        repo=$1
        tag=$2
        echo $repo
        echo $tag
        digest=$(curl -k -v -s -H "Accept:application/vnd.docker.distribution.manifest.v2+json" $PROTO$REG/v2/$repo/manifests/$tag 2>&1 |grep "< Docker-Content-Digest:"|awk '{print $3}')
        digest=${digest//[$'\t\r\n']}
        echo "DIGEST: $digest"
        result=$(curl -k -s -o /dev/null -w "%{http_code}" -H "Accept:application/vnd.docker.distribution.manifest.v2+json" -X DELETE "$PROTO$REG/v2/$repo/manifests/$digest")
        if [ $result -eq 202 ]; then
            echo "Successfully deleted"
            exit 0
        else
            echo $result 
            echo "Failed to delete"
            exit 3
        fi

        ;;
esac

5. check the disk cost is 187M

[root@xhu-proxy1 registry]# pwd
//usr/local/registry
[root@xhu-proxy1 registry]# du -sh
187M    .
[root@xhu-proxy1 registry]# 

6. delete the image by registry v2 api according to doc

[root@xhu-proxy1 opt]#  ./docker-registry.sh http://xhu-proxy1:5000 delete test/tomcat-amd64 8.5
test/tomcat-amd64
8.5
DIGEST: sha256:fc5e2578efd3222266b3d4e3c7f06f32b72f74083c5d82182c2555e52c0a2f1c
Successfully deleted
[root@xhu-proxy1 opt]#

7. run garbage-collect to free the disk

[root@xhu-proxy1 opt]#  docker exec -it registry  registry garbage-collect  /etc/docker/registry/config.yml
test/tomcat-amd64

0 blobs marked, 13 blobs eligible for deletion
blob eligible for deletion: sha256:5a069ba3df4d4221755d76d905ce8a0d2eedf3edbd87dca05a6259114c7b93d4
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/5a/5a069ba3df4d4221755d76d905ce8a0d2eedf3edbd87dca05a6259114c7b93d4  go.version=go1.7.6 instance.id=fa291eb1-baf5-4aac-8b60-9278f1074a94
blob eligible for deletion: sha256:88f01b722a52db8d5c67facd263893c5199b89fbf56505dd2051434721fbd302
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/88/88f01b722a52db8d5c67facd263893c5199b89fbf56505dd2051434721fbd302  go.version=go1.7.6 instance.id=fa291eb1-baf5-4aac-8b60-9278f1074a94
blob eligible for deletion: sha256:d4b7902036fe0cefdfe9ccf0404fe13322ecbd552f132be73d3e840f95538838
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/d4/d4b7902036fe0cefdfe9ccf0404fe13322ecbd552f132be73d3e840f95538838  go.version=go1.7.6 instance.id=fa291eb1-baf5-4aac-8b60-9278f1074a94
blob eligible for deletion: sha256:d852ffd6d31fe17c0cd3a64476832681298284c7b8a4440d08298012d6b9a66b
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/d8/d852ffd6d31fe17c0cd3a64476832681298284c7b8a4440d08298012d6b9a66b  go.version=go1.7.6 instance.id=fa291eb1-baf5-4aac-8b60-9278f1074a94
blob eligible for deletion: sha256:e79bb959ec00faf01da52437df4fad4537ec669f60455a38ad583ec2b8f00498
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/e7/e79bb959ec00faf01da52437df4fad4537ec669f60455a38ad583ec2b8f00498  go.version=go1.7.6 instance.id=fa291eb1-baf5-4aac-8b60-9278f1074a94
blob eligible for deletion: sha256:11775a3d792d4fad3d8b5ef638bb5f89a70eb0c57aad99c532a73817eaaa17a6
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/11/11775a3d792d4fad3d8b5ef638bb5f89a70eb0c57aad99c532a73817eaaa17a6  go.version=go1.7.6 instance.id=fa291eb1-baf5-4aac-8b60-9278f1074a94
blob eligible for deletion: sha256:13fdd17462acb3f41d6fbef8f69c7fb29d4d6713bf80e781618dc371ec9a5269
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/13/13fdd17462acb3f41d6fbef8f69c7fb29d4d6713bf80e781618dc371ec9a5269  go.version=go1.7.6 instance.id=fa291eb1-baf5-4aac-8b60-9278f1074a94
blob eligible for deletion: sha256:2092995a1e5441ff1dfbddf2b4e0672833fef258cd773f855003ca0db3387c1a
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/20/2092995a1e5441ff1dfbddf2b4e0672833fef258cd773f855003ca0db3387c1a  go.version=go1.7.6 instance.id=fa291eb1-baf5-4aac-8b60-9278f1074a94
blob eligible for deletion: sha256:de423484a9466ec116a34ba24e068e72e45d6c875c82f7eb63375577b4b51236
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/de/de423484a9466ec116a34ba24e068e72e45d6c875c82f7eb63375577b4b51236  go.version=go1.7.6 instance.id=fa291eb1-baf5-4aac-8b60-9278f1074a94
blob eligible for deletion: sha256:fc5e2578efd3222266b3d4e3c7f06f32b72f74083c5d82182c2555e52c0a2f1c
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/fc/fc5e2578efd3222266b3d4e3c7f06f32b72f74083c5d82182c2555e52c0a2f1c  go.version=go1.7.6 instance.id=fa291eb1-baf5-4aac-8b60-9278f1074a94
blob eligible for deletion: sha256:1b2a72d4e03052566e99130108071fc4eca4942c62923e3e5cf19666a23088ef
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/1b/1b2a72d4e03052566e99130108071fc4eca4942c62923e3e5cf19666a23088ef  go.version=go1.7.6 instance.id=fa291eb1-baf5-4aac-8b60-9278f1074a94
blob eligible for deletion: sha256:c23be56a9ac10e0c4c248ce8cdf6ac08504088243c6a3878ac4681a7289c611e
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/c2/c23be56a9ac10e0c4c248ce8cdf6ac08504088243c6a3878ac4681a7289c611e  go.version=go1.7.6 instance.id=fa291eb1-baf5-4aac-8b60-9278f1074a94
blob eligible for deletion: sha256:ceaac3b844f7ebdc4f2bac1b66847dee29481bfe3f6b67683491b22383c1928d
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/ce/ceaac3b844f7ebdc4f2bac1b66847dee29481bfe3f6b67683491b22383c1928d  go.version=go1.7.6 instance.id=fa291eb1-baf5-4aac-8b60-9278f1074a94
[root@xhu-proxy1 opt]# 

8. check the disk get free now

[root@xhu-proxy1 registry]# pwd
/usr/local/registry
[root@xhu-proxy1 registry]# du -sh
52K     .
[root@xhu-proxy1 registry]#

garbage-collect doesn't work on multi arch docker image

1. clean up env and restart registry server

[root@xhu-proxy1 test]# pwd
/usr/local/registry/docker/registry/v2/repositories/test
[root@xhu-proxy1 test]# rm -rf tomcat-amd64/
[root@xhu-proxy1 opt]# docker restart registry 
registry
[root@xhu-proxy1 opt]#  ./docker-registry.sh http://xhu-proxy1:5000 list
{
    "repositories": []
}
[root@xhu-proxy1 opt]#

2. create muti-arch tomcat image by manifest-tool ,

 1) docker push  xhu-proxy1:5000/test/tomcat-amd64:8.5
 2) create muti-arch image spec
[root@xhu-proxy1 opt]# cat muti-arch.yaml 
image: xhu-proxy1:5000/test/tomcat:8.5 
manifests:
  -
    image: xhu-proxy1:5000/test/tomcat-amd64:8.5
    platform:
    architecture: amd64
    os: linux
[root@xhu-proxy1 opt]#
  3)  push muti-arch image to registy
 [root@xhu-proxy1 opt]# manifest-tool  --insecure  push from-spec muti-arch.yaml 
Digest: sha256:acfaa18c7219a4b9cbe09c44112237a6d0b882b606d01964a6ffa40512e20fa5 434
[root@xhu-proxy1 opt]# 

3. check the image in from the registry

[root@xhu-proxy1 opt]#  ./docker-registry.sh http://xhu-proxy1:5000 list test/tomcat 
{
    "name": "test/tomcat",
    "tags": [
        "8.5"
    ]
}
[root@xhu-proxy1 opt]#  ./docker-registry.sh http://xhu-proxy1:5000 list test/tomcat-amd64
{
    "name": "test/tomcat-amd64",
    "tags": [
        "8.5"
    ]
}
[root@xhu-proxy1 opt]# 

4. delete the single-arch and muti-arch image

[root@xhu-proxy1 opt]#  ./docker-registry.sh http://xhu-proxy1:5000 delete test/tomcat-amd64 8.5
test/tomcat-amd64
8.5
DIGEST: sha256:fc5e2578efd3222266b3d4e3c7f06f32b72f74083c5d82182c2555e52c0a2f1c
Successfully deleted
[root@xhu-proxy1 opt]#  ./docker-registry.sh http://xhu-proxy1:5000 delete test/tomcat 8.5
test/tomcat
8.5
DIGEST: sha256:acfaa18c7219a4b9cbe09c44112237a6d0b882b606d01964a6ffa40512e20fa5
Successfully deleted
[root@xhu-proxy1 opt]#  ./docker-registry.sh http://xhu-proxy1:5000 list test/tomcat-amd64 
{
    "name": "test/tomcat-amd64",
    "tags": null
}
[root@xhu-proxy1 opt]#  ./docker-registry.sh http://xhu-proxy1:5000 list test/tomcat
{
    "name": "test/tomcat",
    "tags": null
}
[root@xhu-proxy1 opt]#

5. run garbage-collect

[root@xhu-proxy1 opt]#  docker exec -it registry  registry garbage-collect  /etc/docker/registry/config.yml
test/tomcat
test/tomcat: marking manifest sha256:fc5e2578efd3222266b3d4e3c7f06f32b72f74083c5d82182c2555e52c0a2f1c 
test/tomcat: marking blob sha256:5a069ba3df4d4221755d76d905ce8a0d2eedf3edbd87dca05a6259114c7b93d4
test/tomcat: marking blob sha256:e79bb959ec00faf01da52437df4fad4537ec669f60455a38ad583ec2b8f00498
test/tomcat: marking blob sha256:d4b7902036fe0cefdfe9ccf0404fe13322ecbd552f132be73d3e840f95538838
test/tomcat: marking blob sha256:1b2a72d4e03052566e99130108071fc4eca4942c62923e3e5cf19666a23088ef
test/tomcat: marking blob sha256:de423484a9466ec116a34ba24e068e72e45d6c875c82f7eb63375577b4b51236
test/tomcat: marking blob sha256:ceaac3b844f7ebdc4f2bac1b66847dee29481bfe3f6b67683491b22383c1928d
test/tomcat: marking blob sha256:88f01b722a52db8d5c67facd263893c5199b89fbf56505dd2051434721fbd302
test/tomcat: marking blob sha256:c23be56a9ac10e0c4c248ce8cdf6ac08504088243c6a3878ac4681a7289c611e
test/tomcat: marking blob sha256:d852ffd6d31fe17c0cd3a64476832681298284c7b8a4440d08298012d6b9a66b
test/tomcat: marking blob sha256:11775a3d792d4fad3d8b5ef638bb5f89a70eb0c57aad99c532a73817eaaa17a6
test/tomcat: marking blob sha256:13fdd17462acb3f41d6fbef8f69c7fb29d4d6713bf80e781618dc371ec9a5269
test/tomcat: marking blob sha256:2092995a1e5441ff1dfbddf2b4e0672833fef258cd773f855003ca0db3387c1a
test/tomcat-amd64

13 blobs marked, 1 blobs eligible for deletion
blob eligible for deletion: sha256:acfaa18c7219a4b9cbe09c44112237a6d0b882b606d01964a6ffa40512e20fa5
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/ac/acfaa18c7219a4b9cbe09c44112237a6d0b882b606d01964a6ffa40512e20fa5  go.version=go1.7.6 instance.id=51115178-117b-4a62-ae10-8131837d57a1
[root@xhu-proxy1 opt]# 

6. check the disk usage, but not get freed

[root@xhu-proxy1 registry]# pwd
/usr/local/registry
[root@xhu-proxy1 registry]# du -sh
187M    .
[root@xhu-proxy1 registry]# 

[huxiaoliang]

@zetaab @m-masataka @RichardScothern any comments?

[zetaab]

I am not using multi-arch images, so might be that its not working.

[sudo-bmitch]

The script to get your digest needs to include the manifest list media type in the accept headers: "application/vnd.docker.distribution.manifest.list.v2+json". Otherwise you get the digest for only one platform and delete a single platform's manifest, and the manifest list still has pointers to the deleted manifest so GC likely ignores it.