Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

podman play kube fails when a container has a containerPort set and replicas > 1 #16765

Closed
dcermak opened this issue Dec 7, 2022 · 1 comment · Fixed by #17082
Closed

podman play kube fails when a container has a containerPort set and replicas > 1 #16765

dcermak opened this issue Dec 7, 2022 · 1 comment · Fixed by #17082
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@dcermak
Copy link
Contributor

dcermak commented Dec 7, 2022

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

With podman > 4.2 podman play kube will bind the created pods to the same port & network, which causes the pods to fail to start.

Steps to reproduce the issue:

  1. Create the following hello-kubic.yaml file:
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-kubic
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-kubic
  template:
    metadata:
      labels:
        app: hello-kubic
    spec:
      containers:
      - name: hello-kubic
        image: registry.opensuse.org/kubic/hello-kubic:latest
        ports:
        - containerPort: 8080
        imagePullPolicy: Always
        env:
        # - name: MESSAGE
        #   value: I haven't specified a message yet
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
  1. Run podman play kube hello-kubic.yaml

Describe the results you received:

The pods fails to start:

Trying to pull registry.opensuse.org/kubic/hello-kubic:latest...
Getting image source signatures
Copying blob 6d68c1718a89 skipped: already exists  
Copying blob fab0d7e08c0b skipped: already exists  
Copying config bb7fd749b1 done  
Writing manifest to image destination
Storing signatures
Trying to pull registry.opensuse.org/kubic/hello-kubic:latest...
Getting image source signatures
Copying blob 6d68c1718a89 skipped: already exists  
Copying blob fab0d7e08c0b skipped: already exists  
Copying config bb7fd749b1 done  
Writing manifest to image destination
Storing signatures
[starting container 2ff27837352066a6db717e7552a5a8d17d0a7d561fa0b0a4420f370313a802af: rootlessport listen tcp 0.0.0.0:8080: bind: address already in use]
[starting container 2ff27837352066a6db717e7552a5a8d17d0a7d561fa0b0a4420f370313a802af: rootlessport listen tcp 0.0.0.0:8080: bind: address already in use starting container 1dcd7648936a517a7d83afc89de482a812e85f3c6ecbaefa5037a2265ba7311b: a dependency of container 1dcd7648936a517a7d83afc89de482a812e85f3c6ecbaefa5037a2265ba7311b failed to start: container state improper]
Trying to pull registry.opensuse.org/kubic/hello-kubic:latest...
Getting image source signatures
Copying blob 6d68c1718a89 skipped: already exists  
Copying blob fab0d7e08c0b skipped: already exists  
Copying config bb7fd749b1 done  
Writing manifest to image destination
Storing signatures
[starting container c24c2fddf85d9088fa74f4a88dc0fc7d1ec051beef4f53523a541ed3dab8a16e: rootlessport listen tcp 0.0.0.0:8080: bind: address already in use]
[starting container c24c2fddf85d9088fa74f4a88dc0fc7d1ec051beef4f53523a541ed3dab8a16e: rootlessport listen tcp 0.0.0.0:8080: bind: address already in use starting container fe3e020dc1cafbc13670803e8eb9bfdba3c6303c0e9a1cd7511d639e847b39a0: a dependency of container fe3e020dc1cafbc13670803e8eb9bfdba3c6303c0e9a1cd7511d639e847b39a0 failed to start: container state improper]
Pod:
239d23a4543d680d78970868e29197ffce9bda0e3eb326543f8b0d5ad9182fb6
Container:
ee067ca9eaf0f41c170f168e32e6d5117f80b07df4f612369d3ccda9b278e95b

Pod:
41ffd73581e4f6d3ec3933f715e6abb5cb1e48f12387b104ae4871bfe0219998
Container:
1dcd7648936a517a7d83afc89de482a812e85f3c6ecbaefa5037a2265ba7311b

starting container 2ff27837352066a6db717e7552a5a8d17d0a7d561fa0b0a4420f370313a802af: rootlessport listen tcp 0.0.0.0:8080: bind: address already in use
starting container 1dcd7648936a517a7d83afc89de482a812e85f3c6ecbaefa5037a2265ba7311b: a dependency of container 1dcd7648936a517a7d83afc89de482a812e85f3c6ecbaefa5037a2265ba7311b failed to start: container state improper

Pod:
71b268b90f76a1a9633e2c20c59fc9dd2f1f29f690ca8f99c0200e030d4fbe3a
Container:
fe3e020dc1cafbc13670803e8eb9bfdba3c6303c0e9a1cd7511d639e847b39a0

starting container c24c2fddf85d9088fa74f4a88dc0fc7d1ec051beef4f53523a541ed3dab8a16e: rootlessport listen tcp 0.0.0.0:8080: bind: address already in use
starting container fe3e020dc1cafbc13670803e8eb9bfdba3c6303c0e9a1cd7511d639e847b39a0: a dependency of container fe3e020dc1cafbc13670803e8eb9bfdba3c6303c0e9a1cd7511d639e847b39a0 failed to start: container state improper

Error: failed to start 4 containers

Describe the results you expected:

The pods should have been started

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Client:       Podman Engine
Version:      4.3.1
API Version:  4.3.1
Go Version:   go1.19.2
Built:        Fri Nov 11 16:01:27 2022
OS/Arch:      linux/amd64

Output of podman info:

host:
  arch: amd64
  buildahVersion: 1.28.0
  cgroupControllers:
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.5-1.fc37.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.5, commit: '
  cpuUtilization:
    idlePercent: 32.02
    systemPercent: 11.29
    userPercent: 56.69
  cpus: 12
  distribution:
    distribution: fedora
    version: "37"
  eventLogger: journald
  hostname: Boreas
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    - container_id: 65537
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 10000
      size: 65536
    - container_id: 65537
      host_id: 100000
      size: 65536
  kernel: 6.0.9-300.fc37.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 651243520
  memTotal: 33323687936
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.7-1.fc37.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.7
      commit: 40d996ea8a827981895ce22886a9bac367f87264
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-8.fc37.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 8585515008
  swapTotal: 8589930496
  uptime: 224h 7m 18.00s (Approximately 9.33 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - docker.io
  - quay.io
  - registry.suse.com
store:
  configFile: /home/dan/.config/containers/storage.conf
  containerStore:
    number: 8
    paused: 0
    running: 4
    stopped: 4
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/dan/.local/share/containers/storage
  graphRootAllocated: 1022488477696
  graphRootUsed: 753972285440
  graphStatus:
    Backing Filesystem: btrfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 1209
  runRoot: /run/user/1000/containers
  volumePath: /home/dan/.local/share/containers/storage/volumes
version:
  APIVersion: 4.3.1
  Built: 1668178887
  BuiltTime: Fri Nov 11 16:01:27 2022
  GitCommit: ""
  GoVersion: go1.19.2
  Os: linux
  OsArch: linux/amd64
  Version: 4.3.1

Package info (e.g. output of rpm -q podman or apt list podman or brew info podman):

podman-4.3.1-1.fc37.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

I think this is actually a regression introduced by #15946. I have reverted that commit on main and the yaml manifests can be played without issues.
@openshift-ci openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Dec 7, 2022
dcermak added a commit to dcermak/podman that referenced this issue Dec 7, 2022
@dcermak
Only bind pod ports to host when replicas == 1
b04914d 
podman play kube creates a pod per replica and tries to bind all of them to the
same host port. That however can only work when there is one replica as
otherwise multiple pods get bound to the same port on the host.

This fixes containers#16765
dcermak added a commit to dcermak/podman that referenced this issue Dec 7, 2022
@dcermak
Only bind pod ports to host when replicas == 1
5cf5167 
podman play kube creates a pod per replica and tries to bind all of them to the
same host port. That however can only work when there is one replica as
otherwise multiple pods get bound to the same port on the host.

This fixes containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Dec 7, 2022
@dcermak
Only bind pod ports to host when replicas == 1
b19f09b 
podman play kube creates a pod per replica and tries to bind all of them to the
same host port. That however can only work when there is one replica as
otherwise multiple pods get bound to the same port on the host.

This fixes containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Dec 7, 2022
@dcermak
Only bind pod ports to host when replicas == 1
f5b8563 
podman play kube creates a pod per replica and tries to bind all of them to the
same host port. That however can only work when there is one replica as
otherwise multiple pods get bound to the same port on the host.

This fixes containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Dec 7, 2022
@dcermak
Only bind pod ports to host when replicas == 1
a12d258 
podman play kube creates a pod per replica and tries to bind all of them to the
same host port. That however can only work when there is one replica as
otherwise multiple pods get bound to the same port on the host.

This fixes containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Dec 8, 2022
@dcermak
Add --expose flag to podman play kube
0b37883 
Kubernetes deployments and pods allow to specify ports for each container in the
spec. Podman was by default exposing these ports on the host, but this causes
issues when multiple replicas are involved, as then podman would bind multiple
containers to the same port.
This commit adds the new flag `--expose` to `podman play kube` which explicitly
tells podman to publish ports on the host (off by default). Additionally, if
multiple replicas are involved, then podman will override the hostPort and pick
a random port on the host to avoid clashes.

This fixes containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Dec 8, 2022
@dcermak
Add --expose flag to podman play kube
a40df04 
Kubernetes deployments and pods allow to specify ports for each container in the
spec. Podman was by default exposing these ports on the host, but this causes
issues when multiple replicas are involved, as then podman would bind multiple
containers to the same port.
This commit adds the new flag `--expose` to `podman play kube` which explicitly
tells podman to publish ports on the host (off by default). Additionally, if
multiple replicas are involved, then podman will override the hostPort and pick
a random port on the host to avoid clashes.

This fixes containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Dec 8, 2022
@dcermak
Add --expose flag to podman play kube
fd7aa23 
Kubernetes deployments and pods allow to specify ports for each container in the
spec. Podman was by default exposing these ports on the host, but this causes
issues when multiple replicas are involved, as then podman would bind multiple
containers to the same port.
This commit adds the new flag `--expose` to `podman play kube` which explicitly
tells podman to publish ports on the host (off by default). Additionally, if
multiple replicas are involved, then podman will override the hostPort and pick
a random port on the host to avoid clashes.

This fixes containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Dec 8, 2022
@dcermak
Add --expose flag to podman play kube
ddb2fdc 
Kubernetes deployments and pods allow to specify ports for each container in the
spec. Podman was by default exposing these ports on the host, but this causes
issues when multiple replicas are involved, as then podman would bind multiple
containers to the same port.
This commit adds the new flag `--expose` to `podman play kube` which explicitly
tells podman to publish ports on the host (off by default). Additionally, if
multiple replicas are involved, then podman will override the hostPort and pick
a random port on the host to avoid clashes.

This fixes containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Dec 12, 2022
@dcermak
Add --expose flag to podman play kube
80a497f 
Kubernetes deployments and pods allow to specify ports for each container in the
spec. Podman was by default exposing these ports on the host, but this causes
issues when multiple replicas are involved, as then podman would bind multiple
containers to the same port.
This commit adds the new flag `--expose` to `podman play kube` which explicitly
tells podman to publish ports on the host (off by default). Additionally, if
multiple replicas are involved, then podman will override the hostPort and pick
a random port on the host to avoid clashes.

This fixes containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Dec 12, 2022
@dcermak
Add --expose flag to podman play kube
7904cb3 
Kubernetes deployments and pods allow to specify ports for each container in the
spec. Podman was by default exposing these ports on the host, but this causes
issues when multiple replicas are involved, as then podman would bind multiple
containers to the same port.
This commit adds the new flag `--expose` to `podman play kube` which explicitly
tells podman to publish ports on the host (off by default). Additionally, if
multiple replicas are involved, then podman will override the hostPort and pick
a random port on the host to avoid clashes.

This fixes containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Dec 15, 2022
@dcermak
Pick a random port on the host for play kube deployments
5afd75b 
podman play kube started to expose the port of containers in k8s deployments on
the same port on the host with
containers#15946. However, that breaks once
replicas are involved, as they would then bind to the same port on the host.

With this commit, we change the default behavior so that podman will pick a
random port for the container instead of using `containerPort`, but only if no
`hostPort` is set.

This fixes: containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Dec 15, 2022
@dcermak
Pick a random port on the host for play kube deployments
ab04e58 
podman play kube started to expose the port of containers in k8s deployments on
the same port on the host with
containers#15946. However, that breaks once
replicas are involved, as they would then bind to the same port on the host.

With this commit, we change the default behavior so that podman will pick a
random port for the container instead of using `containerPort`, but only if no
`hostPort` is set.

This fixes: containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Dec 15, 2022
@dcermak
Pick a random port on the host for play kube deployments
6ba6ed8 
podman play kube started to expose the port of containers in k8s deployments on
the same port on the host with
containers#15946. However, that breaks once
replicas are involved, as they would then bind to the same port on the host.

With this commit, we change the default behavior so that podman will pick a
random port for the container instead of using `containerPort`, but only if no
`hostPort` is set.

This fixes: containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Dec 16, 2022
@dcermak
Pick a random port on the host for play kube deployments
2397fea 
podman play kube started to expose the port of containers in k8s deployments on
the same port on the host with
containers#15946. However, that breaks once
replicas are involved, as they would then bind to the same port on the host.

With this commit, we change the default behavior so that podman will pick a
random port for the container instead of using `containerPort`, but only if no
`hostPort` is set.

This fixes: containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Dec 20, 2022
@dcermak
Pick a random port on the host for play kube deployments
b8b9c6a 
podman play kube started to expose the port of containers in k8s deployments on
the same port on the host with
containers#15946. However, that breaks once
replicas are involved, as they would then bind to the same port on the host.

With this commit, we change the default behavior so that podman will pick a
random port for the container instead of using `containerPort`, but only if no
`hostPort` is set.

This fixes: containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Jan 2, 2023
@dcermak
Pick a random port on the host for play kube deployments
2e113b6 
podman play kube started to expose the port of containers in k8s deployments on
the same port on the host with
containers#15946. However, that breaks once
replicas are involved, as they would then bind to the same port on the host.

With this commit, we change the default behavior so that podman will pick a
random port for the container instead of using `containerPort`, but only if no
`hostPort` is set.

This fixes: containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
@github-actions
Copy link

github-actions bot commented Jan 7, 2023

A friendly reminder that this issue had no activity for 30 days.

This was referenced Jan 11, 2023
Closed
Merged
dcermak added a commit to dcermak/podman that referenced this issue Jan 12, 2023
@dcermak
Limit replica count to 1 when deploying from kubernetes YAML
c529e80 
This fixes: containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Jan 12, 2023
@dcermak
Limit replica count to 1 when deploying from kubernetes YAML
5ec7588 
This fixes: containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Jan 12, 2023
@dcermak
Limit replica count to 1 when deploying from kubernetes YAML
a78cc1d 
This fixes: containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
dcermak added a commit to dcermak/podman that referenced this issue Jan 12, 2023
@dcermak
Limit replica count to 1 when deploying from kubernetes YAML
d9bf3f1 
This fixes: containers#16765

Signed-off-by: Dan Čermák <dcermak@suse.com>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 4, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 4, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Warn about creating multiple replicas from kubernetes YAML dcermak/podman
2 participants
@rhatdan @dcermak