Skip to content

'skopeo copy' gets stuck uploading images to a Docker registry on Fedora >= 40 hosts #22575

Closed
@debarshiray

Description

@debarshiray

This was observed when trying to add Fedora 40 to the list of host operating systems on which the upstream Toolbx CI is run. Eventually, it started happening on Fedora Rawhide hosts too.

The Toolbx test suite sets up a local temporary Docker registry to test some behaviour of the toolbox(1) binary.

First, it sets up a TLS key pair:

$ mkdir /var/tmp/foo
$ openssl req \
    -newkey rsa:4096 \
    -nodes -sha256 \
    -keyout "/var/tmp/foo/domain.key" \
    -addext "subjectAltName= DNS:localhost" \
    -x509 \
    -days 365 \
    -subj "/" \
    -out "/var/tmp/foo/domain.crt"
...
$ mkdir --parents ~/.config/containers/certs.d/localhost\:50000
$ cp /var/tmp/foo/domain.crt ~/.config/containers/certs.d/localhost\:50000

Second, it runs a Docker registry in a container:

$ podman run \
    --detach \
    --rm \
    --name docker-registry \
    --privileged \
    --volume /var/tmp/foo:/certs \
    -e REGISTRY_HTTP_ADDR=0.0.0.0:443 \
    -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt \
    -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key \
    -p 50000:443 \
    docker.io/library/registry:2
...

Finally, it populates the Docker registry with an image:

$ podman pull registry.fedoraproject.org/fedora:34
...
$ skopeo copy containers-storage:registry.fedoraproject.org/fedora:34 docker://localhost:50000/fedora:34
Getting image source signatures
Copying blob 883e787c00d4 done   | 
<stuck>

That last step to upload an older Fedora image to populate the local temporary Docker registry always gets stuck on hosts running Fedora 40 and newer, but doesn't on hosts running older Fedora, such as 38 and 39.

For various odd requirements and quirks, the Toolbx test suite specifically uploads an older Fedora based image to simplify the implementation. That said, I reproduced it also with the fedora:35, fedora:36, fedora:37, fedora:38 and fedora:39 images on a Fedora 40 host. The fedora:40 image didn't get stuck on a Fedora 40 host.

All my tests were done using Skopeo 1.15.0 on Fedora 41, 40, 39 and 38 hosts. The corresponding RPMs were:

skopeo-1.15.0-1.fc41.x86_64
skopeo-1.15.0-1.fc40.x86_64
skopeo-1.15.0-1.fc39.x86_64
skopeo-1.15.0-1.fc38.x86_64

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.locked - please file new issue/PRAssist humans wanting to comment on an old issue or PR with locked comments.networkNetworking related issue or featurepastapasta(1) bugs or features

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions