Cannot compute composefs digest from inside of a toolbox using podman remote

[jeckersb]

Followup from bootc-dev/containers-image-proxy-rs#108 ...

I get the following:

⬢ [jeckersb@toolbx bootc]$ cargo run -- container compute-composefs-digest localhost/bootc:latest
    Finished `dev` profile [optimized + debuginfo] target(s) in 0.22s
     Running `target/debug/bootc container compute-composefs-digest 'localhost/bootc:latest'`
error: Pulling image: other error: skopeo proxy unexpectedly exited during request method Initialize: exit status: 125
Error: cannot use command "podman unshare" with the remote podman client

Which makes sense... but given the context of just reading an image from my rootless containers-storage in my home directory I don't see why that "podman unshare" is even required.

So I did a bit of digging with claude alongside, and it came up with this (attached hopefully) python script to sanity check that the proxy could be initialized. And surprisingly, it seems like it does work ok from within my toolbox:

⬢ [jeckersb@toolbx skopeo]$ ./test-proxy.py
Starting skopeo experimental-image-proxy...
Sending Initialize request...
Initialize Response: {
  "success": true,
  "value": "0.2.8",
  "pipeid": 0,
  "error_code": "",
  "error": ""
}
Sending Shutdown request...
Process exited with code: 0

=== STDERR ===
time="2025-11-20T14:49:48-05:00" level=debug msg="Executing method Initialize"
time="2025-11-20T14:49:48-05:00" level=debug msg="Using SQLite blob info cache at /var/home/jeckersb/.local/share/containers/cache/blob-info-cache-v1.sqlite"
time="2025-11-20T14:49:48-05:00" level=debug msg="Sending reply: err=<nil> value=0.2.8 pipeid=0 datafd=<nil> errfd=<nil>"
time="2025-11-20T14:49:48-05:00" level=debug msg="Executing method Shutdown"
time="2025-11-20T14:49:48-05:00" level=debug msg=terminating


=== STRACE (filtered for podman/unshare/exec) ===
215823 execve("/usr/bin/skopeo", ["skopeo", "experimental-image-proxy", "--debug"], 0x7ffead7e08d8 /* 51 vars */) = 0

So now I'm a bit confused about where exactly the error is coming from with podman unshare. Some sort of interaction with bootc itself re-execing? I don't see anything obvious in containers-image-proxy to explain it.

test-proxy.py

[cgwalters]

Basically composefs-rs is invoking this podman unshare but that won't work inside a toolbox (and is nonsensical with podman-remote in general).

I think bigger picture what we should try to do is pick up https://github.com/ublue-os/image-template/ in general and add targets to it that wrap the existing exec-bootc-in-container flow as opposed to running it from the host for now.

[jeckersb]

So reading through containers/skopeo#2563, and given our previous video call chat about that, I expected that I wouldn't be able to use skopeo to copy an image out of container-storage as a rootless user. But that seems to work just fine for me...

⬢ [jeckersb@toolbx ~]$ skopeo copy -q containers-storage:localhost/bootc-bin oci:bootc-bin-oci
WARN[0000] Reading allowed ID mappings: reading subuid mappings for user "jeckersb" and subgid mappings for group "jeckersb": no subuid ranges found for user "jeckersb" in /etc/subuid
WARN[0000] Found no UID ranges set aside for user "jeckersb" in /etc/subuid.
WARN[0000] Found no GID ranges set aside for user "jeckersb" in /etc/subgid.
⬢ [jeckersb@toolbx ~]$ ls bootc-bin-oci/
blobs  index.json  oci-layout

So maybe this is something that got fixed in the containers stack somewhere along the way??

[jeckersb]

One thing I tried is to mimic the reexec code from copy:

https://github.com/containers/skopeo/blob/107b1b1ed2180439b7142aa3c663e8453296947a/cmd/skopeo/copy.go#L119

And do that when opening an image in the proxy here:

https://github.com/containers/skopeo/blob/107b1b1ed2180439b7142aa3c663e8453296947a/cmd/skopeo/proxy.go#L225

But I'm guessing in the process of re-exec'ing it ends up dropping its half of the SOCK_SEQPACKET (probably due to O_CLOEXEC) and then the whole thing just hangs:

⬢ [jeckersb@toolbx bootc]$ CONTAINERS_IMAGE_PROXY_DEBUG=1 cargo run -- container compute-composefs-digest containers-storage:localhost/bootc
    Finished `dev` profile [optimized + debuginfo] target(s) in 0.09s
     Running `target/debug/bootc container compute-composefs-digest 'containers-storage:localhost/bootc'`
DEBU[0000] Executing method Initialize
DEBU[0000] Using SQLite blob info cache at /var/home/jeckersb/.local/share/containers/cache/blob-info-cache-v1.sqlite
DEBU[0000] Sending reply: err=<nil> value=0.2.8 pipeid=0 datafd=<nil> errfd=<nil>
DEBU[0000] Executing method OpenImage
WARN[0000] Reading allowed ID mappings: reading subuid mappings for user "jeckersb" and subgid mappings for group "jeckersb": no subuid ranges found for user "jeckersb" in /etc/subuid
WARN[0000] Found no UID ranges set aside for user "jeckersb" in /etc/subuid.
WARN[0000] Found no GID ranges set aside for user "jeckersb" in /etc/subgid.
DEBU[0000] Running [skopeo-in-a-user-namespace experimental-image-proxy --debug] with environment [redacted], UID map [{ContainerID:0 HostID:1000 Size:1}], and GID map [{ContainerID:0 HostID:1000 Size:1}]

But that feels close to what we want ultimately?

[cgwalters]

Did you see containers/skopeo#2564 that was linked from that issue?

We could probably revive it, but...

Error: cannot use command "podman unshare" with the remote podman client

That's really it, in the "truly remote" case we basically have to run a container anyways...

We could of course bypass the error by detecting that we're in a "remote podman is just local socket" path and bypass it by unsetting the environment variable too before invoking podman unshare.

(It's created an asymmetry that composefs-rs only grew that "podman unshare" logic either way, and argues for having it in containers-image-proxy-rs or skopeo)

[jeckersb]

Did you see containers/skopeo#2564 that was linked from that issue?

Nah I missed it but thanks for pointing it out!

---

Really the fact that podman is involved at all here feels wrong. I mean I get the fact that "podman unshare" just happens to be performing the necessary user namespace manipulation, but that's just an artifact of reusing something that already exists and works. Correct me if I'm wrong, but skopeo is really just wrapping containers/{image,storage} and doesn't care too much about podman. So as far as moving data around with skopeo, we don't care whether podman is remote or local (or even exists at all)... which brings up this side tangent question: does skopeo require podman at all? It looks like no. This is on a freshly provisioned f43 vm as an unprivileged user:

[jeckersb@fedora ~]$ which podman
/usr/bin/which: no podman in (/home/jeckersb/.local/bin:/home/jeckersb/bin:/usr/local/bin:/usr/bin)
[jeckersb@fedora ~]$ skopeo copy docker://registry.fedoraproject.org/fedora:43 containers-storage:registry.fedoraproject.org/fedora:43
Getting image source signatures
Copying blob ec7aeea0dda2 done   |
Copying config 8ca2fdf4ad done   |
Writing manifest to image destination
[jeckersb@fedora ~]$ skopeo copy containers-storage:registry.fedoraproject.org/fedora:43 oci:fedora:43
Getting image source signatures
Copying blob 814f65ad561f done   |
Copying config 8ca2fdf4ad done   |
Writing manifest to image destination

I can copy into and out of containers-storage without podman whatsoever.

[cgwalters]

does skopeo require podman at all?

No, they both vendor https://github.com/containers/container-libs/ as a Go library.

---

So three options:

• Revive the skopeo patch
• Change containers-image-proxy to always do the podman unshare itself (ugly since we don't want to require podman in theory; it'd be nice if skopeo exposed this, argues for doing it in skopeo)
• Just make this case a comprehensible error and work instead on including this in a template repo and/or improving docs

I lean towards the last.