podman pull quay.io/devfile/universal-developer-image:ubi8-latest failing

[themr0c]

/kind bug
/area developer-images

What versions of software are you using?

Operating System: Fedora 36

Go Pkg Version: --

podman version 4.1.0

Bug Summary

Describe the bug:

I can't pull the universal-developer image on a rootless podman environment, on Fedora 36 with a BTRFS filesystem.

To Reproduce:

1. Verify podman subuid and subgid are configured:

$ cat /etc/subuid
ffloreth:200000:65536
$ cat /etc/subgid
ffloreth:200000:65536

2. Pull the image:

$ podman pull quay.io/devfile/universal-developer-image:ubi8-latest
... 
Error: writing blob: adding layer with blob "sha256:68d68a40ee2a686776d48c8d10749be3478a20322a98ee53038fa4733b27bd11": Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 0:1000640000 for /usr/local/bin/tkn): Check /etc/subuid and /etc/subgid if configured locally and run podman-system-migrate: lchown /usr/local/bin/tkn: invalid argument

3. Reproduce same behavior on a container based on ubi:

$ podman pull quay.io/mloriedo/che-docs:2370

Expected behavior

• Podman pulls the image without error.

Any logs, error output, screenshots etc? Provide the devfile that sees this bug, if applicable.

$ podman pull quay.io/devfile/universal-developer-image
... 
Error: writing blob: adding layer with blob "sha256:68d68a40ee2a686776d48c8d10749be3478a20322a98ee53038fa4733b27bd11": Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 0:1000640000 for /usr/local/bin/tkn): Check /etc/subuid and /etc/subgid if configured locally and run podman-system-migrate: lchown /usr/local/bin/tkn: invalid argument

Additional context

Any workaround?

Suggestion on how to fix the bug

[openshift-ci]

@themr0c: The label(s) area/developer-images cannot be applied, because the repository doesn't have them.

In response to this:

/kind bug
/area developer-images

What versions of software are you using?

Operating System: Fedora 36

Go Pkg Version: --

podman version 4.1.0

Bug Summary

Describe the bug:

I can't pull the universal-developer image on a rootless podman environment.

To Reproduce:

1. Verify podman subuid and subgid are configured:

$ cat /etc/subuid
ffloreth:200000:65536
$ cat /etc/subgid
ffloreth:200000:65536

2. Pull the image:

$ podman pull quay.io/devfile/universal-developer-image:ubi8-latest
... 
Error: writing blob: adding layer with blob "sha256:68d68a40ee2a686776d48c8d10749be3478a20322a98ee53038fa4733b27bd11": Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 0:1000640000 for /usr/local/bin/tkn): Check /etc/subuid and /etc/subgid if configured locally and run podman-system-migrate: lchown /usr/local/bin/tkn: invalid argument

3. Reproduce same behavior on a container based on ubi:

$ podman pull quay.io/mloriedo/che-docs:2370

Expected behavior

• Podman pulls the image without error.

Any logs, error output, screenshots etc? Provide the devfile that sees this bug, if applicable.

$ podman pull quay.io/devfile/universal-developer-image
... 
Error: writing blob: adding layer with blob "sha256:68d68a40ee2a686776d48c8d10749be3478a20322a98ee53038fa4733b27bd11": Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 0:1000640000 for /usr/local/bin/tkn): Check /etc/subuid and /etc/subgid if configured locally and run podman-system-migrate: lchown /usr/local/bin/tkn: invalid argument

Additional context

Any workaround?

Suggestion on how to fix the bug

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[themr0c]

I can't find the relevant label for developer-images related issue.

[themr0c]

Confirmed by @rkratky on another laptop, not on a BTRFS filesystem.

[l0rd]

I reproduced the problem on macOS too. Not sure that's something that we can fix in UDI though.

[kadel]

Not sure why is that but for some reason UDI requires ridiculous number of subuids/subgids (1000640000)
I'm definitely not podman expert, but for me the problem was solved increasing subuid/subgid and executing podman system migrate

$ cat /etc/subuid
tomas:100000:1000640000
$ cat /etc/subgid
tomas:100000:1000640000

Not sure if this is the right thing to do. It would be probably worth looking into the image of why it requires over a billion of uids

[michael-valdron]

/area developer-images

[ehsavoie]

@l0rd I think it is related to tektoncd-cli as the tar content files are root:root which feels strange to me.