Podman Remote - play kube --network flag uses default podman network and not the networks provided in argument

[kellen-dunham]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

While using podman remote client, the play kube command's --network flag does not work. No matter what you specify here it will always use the podman network.

Steps to reproduce the issue:

1. Use podman-remote client
2. Have a remote podman socket available
3. Set podman to connect to the remote socket
4. Run podman play kube pod.yaml --network net1,net2,net3

Describe the results you received:

Pod starts as normal but uses the default podman network. If specifying multiple networks only 1 network will be used which is the default podman network.

Describe the results you expected:

Pods start with whatever network attachments defined by the --network flag.

Additional information you deem important (e.g. issue happens only occasionally):

I noticed that when inspecting the pod from remote podman the InfraConfig.Networks is null while running the same pod on the host running the podman socket it contains the list from the --network flag.

Output of podman version:

Host with podman socket:
Version: 3.0.0
API Version: 3.0.0
Go Version: go1.15.2
Built: Wed Dec 31 18:00:00 1969
OS/Arch: linux/amd64

Remote client:
Client:
Version: 3.0.1
API Version: 3.0.0
Go Version: go1.15.8
Built: Thu Jan 1 00:00:00 1970
OS/Arch: linux/amd64

Server:
Version: 3.0.0
API Version: 3.0.0
Go Version: go1.15.2
Built: Thu Jan 1 00:00:00 1970
OS/Arch: linux/amd64

Output of podman info --debug:

Host with podman socket:
host:
arch: amd64
buildahVersion: 1.19.2
cgroupManager: systemd
cgroupVersion: v1
conmon:
package: 'conmon: /usr/libexec/podman/conmon'
path: /usr/libexec/podman/conmon
version: 'conmon version 2.0.26, commit: '
cpus: 12
distribution:
distribution: ubuntu
version: "20.04"
eventLogger: journald
hostname: pc
idMappings:
gidmap: null
uidmap: null
kernel: 5.4.0-66-generic
linkmode: dynamic
memFree: 25667166208
memTotal: 33513377792
ociRuntime:
name: crun
package: 'crun: /usr/bin/crun'
path: /usr/bin/crun
version: |-
crun version 0.18.1-7931a-dirty
commit: 7931a1eab0590eff4041c1f74e2844b297c31cea
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
remoteSocket:
exists: true
path: /run/podman/podman.sock
security:
apparmorEnabled: true
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
selinuxEnabled: false
slirp4netns:
executable: ""
package: ""
version: ""
swapFree: 8589930496
swapTotal: 8589930496
uptime: 16m 25.96s
registries:
search:

• docker.io
• quay.io
• ghcr.io
  store:
  configFile: /etc/containers/storage.conf
  containerStore:
  number: 39
  paused: 0
  running: 4
  stopped: 35
  graphDriverName: overlay
  graphOptions:
  overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
  Backing Filesystem: extfs
  Native Overlay Diff: "false"
  Supports d_type: "true"
  Using metacopy: "true"
  imageStore:
  number: 106
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
  version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Wed Dec 31 18:00:00 1969
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 3.0.0

Remote client:
host:
arch: amd64
buildahVersion: 1.19.2
cgroupManager: systemd
cgroupVersion: v1
conmon:
package: 'conmon: /usr/libexec/podman/conmon'
path: /usr/libexec/podman/conmon
version: 'conmon version 2.0.26, commit: '
cpus: 12
distribution:
distribution: ubuntu
version: "20.04"
eventLogger: journald
hostname: pc
idMappings:
gidmap: null
uidmap: null
kernel: 5.4.0-66-generic
linkmode: dynamic
memFree: 25529597952
memTotal: 33513377792
ociRuntime:
name: crun
package: 'crun: /usr/bin/crun'
path: /usr/bin/crun
version: |-
crun version 0.18.1-7931a-dirty
commit: 7931a1eab0590eff4041c1f74e2844b297c31cea
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
os: linux
security:
apparmorEnabled: true
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: false
seccompEnabled: true
selinuxEnabled: false
slirp4netns:
executable: ""
package: ""
version: ""
swapFree: 8589930496
swapTotal: 8589930496
uptime: 14m 57.3s
registries:
search:

• docker.io
• quay.io
• ghcr.io
  store:
  configFile: /etc/containers/storage.conf
  containerStore:
  number: 39
  paused: 0
  running: 4
  stopped: 35
  graphDriverName: overlay
  graphOptions:
  overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphStatus:
  Backing Filesystem: extfs
  Native Overlay Diff: "false"
  Supports d_type: "true"
  Using metacopy: "true"
  imageStore:
  number: 106
  runRoot: /run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
  version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Wed Dec 31 18:00:00 1969
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 3.0.0

Package info (e.g. output of rpm -q podman or apt list podman):

Host with podman socket:
Listing... Done
podman/unknown 100:3.0.1-2 amd64 [upgradable from: 100:3.0.0-4]
podman/unknown 100:3.0.1-2 arm64
podman/unknown 100:3.0.1-2 armhf
podman/unknown 100:3.0.1-2 s390x

Remote client:
Listing... Done

(this was just the binary downloaded and the socket forwarded)

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

The remote client is a docker container with the podman socket mounted into the container. The use case is to use podman from container with the host's socket.

[rhatdan]

@ashley-cui PTAL

[rhatdan]

@kellen-dunham Interested in investigating and fix this?

[kellen-dunham]

Sure @rhatdan, I can take a look at what's going on and let you know what I find!

[kellen-dunham]

@rhatdan It looks like podman-remote is passing the wrong URL query parameter to the API. From the looks of it the networks should be passed as reference={VALUE_FROM_CMD} but they are being passed as network={VALUE_FROM_CMD}. Because of this the command values aren't being decoded properly and no CNI Networks are being assigned to the pod.

See

podman/pkg/api/handlers/libpod/play.go

Lines 22 to 30 in 60c90c3

	query := struct {
	Network string `schema:"reference"`
	TLSVerify bool `schema:"tlsVerify"`
	LogDriver string `schema:"logDriver"`
	Start bool `schema:"start"`
	}{
	TLSVerify: true,
	Start: true,
	}

for the backend API code. After looking at this file's history it looks like reference is the correct way to pass, but I defer to you. Perhaps reference is not correct and it should be network though?

In the meantime I'll try to debug the podman-remote tomorrow and give you a better idea of what is going on.

[kellen-dunham]

@rhatdan I reviewed the podman-remote call to the play kube API and it appears that specifying the CNI networks via the "network" query parameter is correct.

podman/pkg/bindings/play/play.go

Line 30 in 60c90c3

params, err := options.ToParams()

podman/pkg/bindings/generate/types_kube_options.go

Line 19 in 60c90c3

func (o *KubeOptions) ToParams() (url.Values, error) {

podman/pkg/bindings/internal/util/util.go

Line 54 in 60c90c3

func ToParams(o interface{}) (url.Values, error) {

Also the documentation for the API confirms this as well.
https://docs.podman.io/en/latest/_static/api.html#operation/libpodPlayKube

If "network" is the correct API parameter, then the fix is simply to change "reference" to "network" on line 23 of play.go I referred to last night.

[rhatdan]

Could you open a PR to fix this?

[kellen-dunham]

Sure thing - #9791