when host has podman from podman-next and podman-remote is run inside a toolbox : unmarshalling error

[inknos]

Issue Description

I was trying to reproduce the issue #21974 which is valid for podman < 5 (did not bisect, take this with a grain of salt).

I realized that having podman installed from podman next leads to an error.

Steps to reproduce the issue

Steps to reproduce the issue

1. Install podman from the podman-next copr on the host
2. Follow the reproducer of podman-remote run --cidfile results in files deleted on container host #21974 until podman-remote run
3. Run the step podman-remote run --cidfile /var/tmp/cidfile fedora:latest true in the toolbox

Describe the results you received

Error: unmarshalling into &define.InspectContainerData{ID:"221aadce464435e1eed931dffe7326d8524e40139614e4c1ce0877463f064816", Created:time.Date(2024, time.May, 9, 15, 37, 24, 363282572, time
.Local), Path:"true", Args:[]string{"true"}, State:(*define.InspectContainerState)(0x4000532000), Image:"a3b2e2a9704e6b54b78f4924f5b1318dbd7aa2553736944dd966cef32da5e37a", ImageDigest:"sha25
6:a4a66434bd361d9c80cd6fd5b0ee3112a0347aed794141b372ce0c4f09afb791", ImageName:"registry.fedoraproject.org/fedora:latest", Rootfs:"", Pod:"", ResolvConfPath:"", HostnamePath:"", HostsPath:""
, StaticDir:"/home/nsella/.local/share/containers/storage/overlay-containers/221aadce464435e1eed931dffe7326d8524e40139614e4c1ce0877463f064816/userdata", OCIConfigPath:"", OCIRuntime:"crun", 
ConmonPidFile:"/run/user/1000/containers/overlay-containers/221aadce464435e1eed931dffe7326d8524e40139614e4c1ce0877463f064816/userdata/conmon.pid", PidFile:"/run/user/1000/containers/overlay-
containers/221aadce464435e1eed931dffe7326d8524e40139614e4c1ce0877463f064816/userdata/pidfile", Name:"nifty_clarke", RestartCount:0, Driver:"overlay", MountLabel:"system_u:object_r:container_
file_t:s0:c139,c898", ProcessLabel:"system_u:system_r:container_t:s0:c139,c898", AppArmorProfile:"", EffectiveCaps:[]string{"CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_FOWNER", "CAP_FSETID", "CAP_
KILL", "CAP_NET_BIND_SERVICE", "CAP_SETFCAP", "CAP_SETGID", "CAP_SETPCAP", "CAP_SETUID", "CAP_SYS_CHROOT"}, BoundingCaps:[]string{"CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_FOWNER", "CAP_FSETID",
 "CAP_KILL", "CAP_NET_BIND_SERVICE", "CAP_SETFCAP", "CAP_SETGID", "CAP_SETPCAP", "CAP_SETUID", "CAP_SYS_CHROOT"}, ExecIDs:[]string{}, GraphDriver:(*define.DriverData)(0x40006b8978), SizeRw:(
*int64)(nil), SizeRootFs:0, Mounts:[]define.InspectMount{}, Dependencies:[]string{}, NetworkSettings:(*define.InspectNetworkSettings)(0x40002bf680), Namespace:"", IsInfra:false, IsService:fa
lse, KubeExitCodePropagation:"invalid", LockNumber:0x1, Config:(*define.InspectContainerConfig)(0x4000636600), HostConfig:(*define.InspectContainerHostConfig)(0x40006c0400)}, data "{\"Id\":\
"221aadce464435e1eed931dffe7326d8524e40139614e4c1ce0877463f064816\",\"Created\":\"2024-05-09T15:37:24.363282572+02:00\",\"Path\":\"true\",\"Args\":[\"true\"],\"State\":{\"OciVersion\":\"1.2.
0\",\"Status\":\"created\",\"Running\":false,\"Paused\":false,\"Restarting\":false,\"OOMKilled\":false,\"Dead\":false,\"Pid\":0,\"ExitCode\":0,\"Error\":\"\",\"StartedAt\":\"0001-01-01T00:00
:00Z\",\"FinishedAt\":\"0001-01-01T00:00:00Z\",\"CheckpointedAt\":\"0001-01-01T00:00:00Z\",\"RestoredAt\":\"0001-01-01T00:00:00Z\"},\"Image\":\"a3b2e2a9704e6b54b78f4924f5b1318dbd7aa255373694
4dd966cef32da5e37a\",\"ImageDigest\":\"sha256:a4a66434bd361d9c80cd6fd5b0ee3112a0347aed794141b372ce0c4f09afb791\",\"ImageName\":\"registry.fedoraproject.org/fedora:latest\",\"Rootfs\":\"\",\"
Pod\":\"\",\"ResolvConfPath\":\"\",\"HostnamePath\":\"\",\"HostsPath\":\"\",\"StaticDir\":\"/home/nsella/.local/share/containers/storage/overlay-containers/221aadce464435e1eed931dffe7326d852
4e40139614e4c1ce0877463f064816/userdata\",\"OCIRuntime\":\"crun\",\"ConmonPidFile\":\"/run/user/1000/containers/overlay-containers/221aadce464435e1eed931dffe7326d8524e40139614e4c1ce0877463f0
64816/userdata/conmon.pid\",\"PidFile\":\"/run/user/1000/containers/overlay-containers/221aadce464435e1eed931dffe7326d8524e40139614e4c1ce0877463f064816/userdata/pidfile\",\"Name\":\"nifty_cl
arke\",\"RestartCount\":0,\"Driver\":\"overlay\",\"MountLabel\":\"system_u:object_r:container_file_t:s0:c139,c898\",\"ProcessLabel\":\"system_u:system_r:container_t:s0:c139,c898\",\"AppArmor
Profile\":\"\",\"EffectiveCaps\":[\"CAP_CHOWN\",\"CAP_DAC_OVERRIDE\",\"CAP_FOWNER\",\"CAP_FSETID\",\"CAP_KILL\",\"CAP_NET_BIND_SERVICE\",\"CAP_SETFCAP\",\"CAP_SETGID\",\"CAP_SETPCAP\",\"CAP_
SETUID\",\"CAP_SYS_CHROOT\"],\"BoundingCaps\":[\"CAP_CHOWN\",\"CAP_DAC_OVERRIDE\",\"CAP_FOWNER\",\"CAP_FSETID\",\"CAP_KILL\",\"CAP_NET_BIND_SERVICE\",\"CAP_SETFCAP\",\"CAP_SETGID\",\"CAP_SET
PCAP\",\"CAP_SETUID\",\"CAP_SYS_CHROOT\"],\"ExecIDs\":[],\"GraphDriver\":{\"Name\":\"overlay\",\"Data\":{\"LowerDir\":\"/home/nsella/.local/share/containers/storage/overlay/e46c7a886cbfe1e67
41a81b494cf8f025358b9e74f625b1a930960d1974e7387/diff\",\"UpperDir\":\"/home/nsella/.local/share/containers/storage/overlay/45eb3c184e2e64464f1cdc979ca8c5f4b9beca07d7cad9eb911cd889bfab54c6/di
ff\",\"WorkDir\":\"/home/nsella/.local/share/containers/storage/overlay/45eb3c184e2e64464f1cdc979ca8c5f4b9beca07d7cad9eb911cd889bfab54c6/work\"}},\"Mounts\":[],\"Dependencies\":[],\"NetworkS
ettings\":{\"EndpointID\":\"\",\"Gateway\":\"\",\"IPAddress\":\"\",\"IPPrefixLen\":0,\"IPv6Gateway\":\"\",\"GlobalIPv6Address\":\"\",\"GlobalIPv6PrefixLen\":0,\"MacAddress\":\"\",\"Bridge\":
\"\",\"SandboxID\":\"\",\"HairpinMode\":false,\"LinkLocalIPv6Address\":\"\",\"LinkLocalIPv6PrefixLen\":0,\"Ports\":{},\"SandboxKey\":\"\",\"Networks\":{\"pasta\":{\"EndpointID\":\"\",\"Gatew
ay\":\"\",\"IPAddress\":\"\",\"IPPrefixLen\":0,\"IPv6Gateway\":\"\",\"GlobalIPv6Address\":\"\",\"GlobalIPv6PrefixLen\":0,\"MacAddress\":\"\",\"NetworkID\":\"pasta\",\"DriverOpts\":null,\"IPA
MConfig\":null,\"Links\":null}}},\"Namespace\":\"\",\"IsInfra\":false,\"IsService\":false,\"KubeExitCodePropagation\":\"invalid\",\"lockNumber\":1,\"Config\":{\"Hostname\":\"221aadce4644\",\
"Domainname\":\"\",\"User\":\"\",\"AttachStdin\":false,\"AttachStdout\":false,\"AttachStderr\":false,\"Tty\":false,\"OpenStdin\":false,\"StdinOnce\":false,\"Env\":[\"FGC=f39\",\"PATH=/usr/lo
cal/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\",\"container=oci\",\"DISTTAG=f39container\"],\"Cmd\":[\"true\"],\"Image\":\"registry.fedoraproject.org/fedora:latest\",\"Volumes\":null
,\"WorkingDir\":\"/\",\"Entrypoint\":null,\"OnBuild\":null,\"Labels\":{\"license\":\"MIT\",\"name\":\"fedora\",\"vendor\":\"Fedora Project\",\"version\":\"39\"},\"Annotations\":{\"io.podman.
annotations.cid-file\":\"/var/tmp/cidfile\"},\"StopSignal\":\"SIGTERM\",\"HealthcheckOnFailureAction\":\"none\",\"CreateCommand\":[\"podman-remote\",\"run\",\"--cidfile\",\"/var/tmp/cidfile\
",\"fedora:latest\",\"true\"],\"Umask\":\"0022\",\"Timeout\":0,\"StopTimeout\":10,\"Passwd\":true,\"sdNotifyMode\":\"container\"},\"HostConfig\":{\"Binds\":[],\"CgroupManager\":\"systemd\",\
"CgroupMode\":\"private\",\"ContainerIDFile\":\"/var/tmp/cidfile\",\"LogConfig\":{\"Type\":\"journald\",\"Config\":null,\"Path\":\"\",\"Tag\":\"\",\"Size\":\"0B\"},\"NetworkMode\":\"pasta\",
\"PortBindings\":{},\"RestartPolicy\":{\"Name\":\"no\",\"MaximumRetryCount\":0},\"AutoRemove\":false,\"Annotations\":{\"io.podman.annotations.cid-file\":\"/var/tmp/cidfile\"},\"VolumeDriver\
":\"\",\"VolumesFrom\":null,\"CapAdd\":[],\"CapDrop\":[],\"Dns\":[],\"DnsOptions\":[],\"DnsSearch\":[],\"ExtraHosts\":[],\"GroupAdd\":[],\"IpcMode\":\"shareable\",\"Cgroup\":\"\",\"Cgroups\"
:\"default\",\"Links\":null,\"OomScoreAdj\":0,\"PidMode\":\"private\",\"Privileged\":false,\"PublishAllPorts\":false,\"ReadonlyRootfs\":false,\"SecurityOpt\":[],\"Tmpfs\":{},\"UTSMode\":\"pr
ivate\",\"UsernsMode\":\"\",\"ShmSize\":65536000,\"Runtime\":\"oci\",\"ConsoleSize\":[0,0],\"Isolation\":\"\",\"CpuShares\":0,\"Memory\":0,\"NanoCpus\":0,\"CgroupParent\":\"user.slice\",\"Bl
kioWeight\":0,\"BlkioWeightDevice\":null,\"BlkioDeviceReadBps\":null,\"BlkioDeviceWriteBps\":null,\"BlkioDeviceReadIOps\":null,\"BlkioDeviceWriteIOps\":null,\"CpuPeriod\":0,\"CpuQuota\":0,\"
CpuRealtimePeriod\":0,\"CpuRealtimeRuntime\":0,\"CpusetCpus\":\"\",\"CpusetMems\":\"\",\"Devices\":[],\"DiskQuota\":0,\"KernelMemory\":0,\"MemoryReservation\":0,\"MemorySwap\":0,\"MemorySwap
piness\":0,\"OomKillDisable\":false,\"PidsLimit\":2048,\"Ulimits\":[],\"CpuCount\":0,\"CpuPercent\":0,\"IOMaximumIOps\":0,\"IOMaximumBandwidth\":0,\"CgroupConf\":null}}\n": json: cannot unma
rshal string into Go struct field InspectContainerConfig.Config.StopSignal of type uint

Describe the results you expected

Command should work

podman info output

$ podman -v
podman version 5.1.0-dev-63ab9275b

$ cat /etc/fedora-release 
Fedora Asahi Remix release 39 (Thirty Nine)

$ uname -a
Linux applem1 6.8.8-400.asahi.fc39.aarch64+16k #1 SMP PREEMPT_DYNAMIC Tue Apr 30 02:30:34 UTC 2024 aarch64 GNU/Linux

### Podman in a container

Yes

### Privileged Or Rootless

Rootless

### Upstream Latest Release

Yes

### Additional environment details

_No response_

### Additional information

I am aware that I am running this on `aarch64` did not try to run it on `x86_64` or other architectures yet.

[mheon]

This looks like a client/server mismatch. We made some breaking changes to podman inspect going from 4.x to 5.0 to improve Docker compat, and stopsignal changing type was one of them. Can you make sure both client and server are 5.x versions?

[inknos]

Mmm right, I tried installing podman-remote from podman-next and the issue disappears. Should it be closed then?

I was thinking that maybe some nice stderr message of version mismatch would be more polite to the user, but only if the plan is to support different podman/podman-remote versions for long time, which I am not sure about.

[mheon]

Version mismatch message does seem like a good idea, especially for a major version mismatch - we know that 4.x and 5.x will have incompatibilities when used together

[Luap99]

I think we explicitly didn't want the version check as almost all endpoints work normal. Only inspect was changed. IMO as I commented on the PR instead of blocking outright we should rather do the work and on a v4 request return the v4 payload.