Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

containers-rootlessport memory hungry #10790

Closed
bear0330 opened this issue Jun 26, 2021 · 7 comments · Fixed by #11565
Closed

containers-rootlessport memory hungry #10790

bear0330 opened this issue Jun 26, 2021 · 7 comments · Fixed by #11565
Assignees
@Luap99
Labels
kind/performance locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. rootless

Comments

@bear0330
Copy link

bear0330 commented Jun 26, 2021 •
edited
Loading

/kind bug

Description

I have 6 django apps container running on CentOS 8.4 with podman 3.1.2.
I found the containers-rootlessport process and its child eat a lot of memory (not really very much but it seems high for a component of podman).

Here is my ps output:

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
..... ignored
nuwa        1080  0.0  1.0 1553588 30680 ?       Sl   Jun26   0:01 containers-rootlessport
nuwa         996  0.0  1.0 1479856 30552 ?       Sl   Jun26   0:01 containers-rootlessport
nuwa        1138  0.0  1.0 1184928 30116 ?       Sl   Jun26   0:01 containers-rootlessport-child
nuwa        1023  0.0  0.9 1479856 28548 ?       Sl   Jun26   0:01 containers-rootlessport
nuwa        1038  0.0  0.9 1184928 28432 ?       Sl   Jun26   0:01 containers-rootlessport-child
nuwa        1078  0.0  0.9 1184928 28316 ?       Sl   Jun26   0:01 containers-rootlessport-child
nuwa         993  0.0  0.9 1406124 28144 ?       Sl   Jun26   0:01 containers-rootlessport
nuwa        1280  0.0  0.9 1184928 28136 ?       Sl   Jun26   0:00 containers-rootlessport-child
root         731  0.0  0.9 307400 26820 ?        Ssl  Jun26   0:06 /usr/libexec/platform-python -s /usr/sbin/firewalld --nofork --nopid
nuwa        1004  0.0  0.9 1479856 26692 ?       Sl   Jun26   0:01 containers-rootlessport
nuwa        1047  0.0  0.9 1184928 26544 ?       Sl   Jun26   0:01 containers-rootlessport-child
..... ignored

Steps to reproduce the issue:
Run some rootless containers, use htop or ps

Output of podman version:

Version:      3.1.2
API Version:  3.1.2
Go Version:   go1.14.12
Built:        Fri Apr 23 21:43:22 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.20.1
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.28-1.el8.2.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.28, commit: '
  cpus: 2
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: journald
  hostname: test.nuwainfo.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 4.18.0-240.22.1.el8_3.x86_64
  linkmode: dynamic
  memFree: 76537856
  memTotal: 2963959808
  ociRuntime:
    name: crun
    package: crun-0.16-2.module_el8.3.0+699+d61d9c41.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.16
      commit: eb0145e5ad4d8207e84a327248af76663d4e50dd
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.module_el8.3.0+699+d61d9c41.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 0
  swapTotal: 0
  uptime: 6h 50m 45.82s (Approximately 0.25 days)
registries:
  search:
  - docker.io
store:
  configFile: /home/nuwa/.config/containers/storage.conf
  containerStore:
    number: 6
    paused: 0
    running: 5
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.5.0-1.el8.1.1.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.5
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/nuwa/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 248
  runRoot: /run/user/1001/containers
  volumePath: /home/nuwa/.local/share/containers/storage/volumes
version:
  APIVersion: 3.1.2
  Built: 1619185402
  BuiltTime: Fri Apr 23 21:43:22 2021
  GitCommit: ""
  GoVersion: go1.14.12
  OsArch: linux/amd64
  Version: 3.1.2

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.1.2-1.el8.2.1.x86_64
@openshift-ci openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Jun 26, 2021
@mheon
Copy link
Member

mheon commented Jun 27, 2021

@AkihiroSuda PTAL

@AkihiroSuda AkihiroSuda added rootless kind/performance and removed kind/bug Categorizes issue or PR as related to a bug. labels Jun 28, 2021
@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@rhatdan
Copy link
Member

rhatdan commented Aug 1, 2021

@AkihiroSuda @Luap99 @giuseppe Any thoughts on this?

@github-actions
Copy link

github-actions bot commented Sep 1, 2021

A friendly reminder that this issue had no activity for 30 days.

@Luap99
Copy link
Member

Luap99 commented Sep 2, 2021

I think one problem is that the containers-rootlessport process is using reexec inside podman. Because of it there will be a lot of unnecessary stuff loaded into memory which rootlessport never uses.

Just as a POC I created a separate rootlessport binary which only contains the rootlessport code:

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
pholzing  108520  0.0  0.0 1223400 4540 pts/0    Sl   12:54   0:00 containers-rootlessport
pholzing  108526  0.0  0.0 1075684 4152 pts/0    Sl   12:54   0:00 containers-rootlessport-child

Compare this to the version with reexec on my system:

pholzing  108914  0.0  0.1 1730508 47800 pts/0   Sl   12:56   0:00 containers-rootlessport
pholzing  108928  0.0  0.1 1804116 47972 pts/0   Sl   12:56   0:00 containers-rootlessport-child

The separate binary uses less than 1/10 of the RSS than the reexec binary, so using a separate rootlessport binary to reduce the memory footprint looks promising to me.

The disadvantage is that we would need to ship and maintain a separate binary.

@Luap99 Luap99 self-assigned this Sep 8, 2021
Luap99 added a commit to Luap99/libpod that referenced this issue Sep 14, 2021
@Luap99
rootlessport: reduce memory usage of the process
05ec686 
Don't use reexec for the rootlessport process, instaed make it a
separate binary to reduce the memory usage. The problem with reexec is
that it will import all packages that podman uses and therefore loads a
lot of stuff into the heap. The rootlessport process however only needs
the rootlesskit library.
The memory usage is a concern since the rootlessport process will spawn
two process per container which has ports forwarded. The processes stay
until the container dies. On my laptop the current reexec version uses
47800 KB RSS. The new separate binary only uses 4540 KB RSS. This is
more than a 90% improvement.

For backwards compatibility the podman binary still contains the option
for the reexec. I will remove this for podman 4.0.

The Makefile has been updated to compile the new binary and install it
to the libexec directory.

Fixes containers#10790

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Luap99 added a commit to Luap99/libpod that referenced this issue Sep 14, 2021
@Luap99
rootlessport: reduce memory usage of the process
eb8e17e 
Don't use reexec for the rootlessport process, instead make it a
separate binary to reduce the memory usage. The problem with reexec is
that it will import all packages that podman uses and therefore loads a
lot of stuff into the heap. The rootlessport process however only needs
the rootlesskit library.
The memory usage is a concern since the rootlessport process will spawn
two process per container which has ports forwarded. The processes stay
until the container dies. On my laptop the current reexec version uses
47800 KB RSS. The new separate binary only uses 4540 KB RSS. This is
more than a 90% improvement.

For backwards compatibility the podman binary still contains the option
for the reexec. I will remove this for podman 4.0.

The Makefile has been updated to compile the new binary and install it
to the libexec directory.

Fixes containers#10790

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
@Luap99 Luap99 mentioned this issue Sep 14, 2021
Merged
Luap99 added a commit to Luap99/libpod that referenced this issue Sep 14, 2021
@Luap99
rootlessport: reduce memory usage of the process
119cfbb 
Don't use reexec for the rootlessport process, instead make it a
separate binary to reduce the memory usage. The problem with reexec is
that it will import all packages that podman uses and therefore loads a
lot of stuff into the heap. The rootlessport process however only needs
the rootlesskit library.
The memory usage is a concern since the rootlessport process will spawn
two process per container which has ports forwarded. The processes stay
until the container dies. On my laptop the current reexec version uses
47800 KB RSS. The new separate binary only uses 4540 KB RSS. This is
more than a 90% improvement.

For backwards compatibility the podman binary still contains the option
for the reexec. I will remove this for podman 4.0.

The Makefile has been updated to compile the new binary and install it
to the libexec directory.

Fixes containers#10790

[NO TESTS NEEDED]

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Luap99 added a commit to Luap99/libpod that referenced this issue Sep 21, 2021
@Luap99
rootlessport: reduce memory usage of the process
e15df9b 
Don't use reexec for the rootlessport process, instead make it a
separate binary to reduce the memory usage. The problem with reexec is
that it will import all packages that podman uses and therefore loads a
lot of stuff into the heap. The rootlessport process however only needs
the rootlesskit library.
The memory usage is a concern since the rootlessport process will spawn
two process per container which has ports forwarded. The processes stay
until the container dies. On my laptop the current reexec version uses
47800 KB RSS. The new separate binary only uses 4540 KB RSS. This is
more than a 90% improvement.

For backwards compatibility the podman binary still contains the option
for the reexec. I will remove this for podman 4.0.

The Makefile has been updated to compile the new binary and install it
to the libexec directory.

Fixes containers#10790

[NO TESTS NEEDED]

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Luap99 added a commit to Luap99/libpod that referenced this issue Sep 21, 2021
@Luap99
rootlessport: reduce memory usage of the process
1aeff92 
Don't use reexec for the rootlessport process, instead make it a
separate binary to reduce the memory usage. The problem with reexec is
that it will import all packages that podman uses and therefore loads a
lot of stuff into the heap. The rootlessport process however only needs
the rootlesskit library.
The memory usage is a concern since the rootlessport process will spawn
two process per container which has ports forwarded. The processes stay
until the container dies. On my laptop the current reexec version uses
47800 KB RSS. The new separate binary only uses 4540 KB RSS. This is
more than a 90% improvement.

For backwards compatibility the podman binary still contains the option
for the reexec. I will remove this for podman 4.0.

The Makefile has been updated to compile the new binary and install it
to the libexec directory.

Fixes containers#10790

[NO TESTS NEEDED]

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Luap99 added a commit to Luap99/libpod that referenced this issue Oct 6, 2021
@Luap99
rootlessport: reduce memory usage of the process
e5034d4 
Don't use reexec for the rootlessport process, instead make it a
separate binary to reduce the memory usage. The problem with reexec is
that it will import all packages that podman uses and therefore loads a
lot of stuff into the heap. The rootlessport process however only needs
the rootlesskit library.
The memory usage is a concern since the rootlessport process will spawn
two process per container which has ports forwarded. The processes stay
until the container dies. On my laptop the current reexec version uses
47800 KB RSS. The new separate binary only uses 4540 KB RSS. This is
more than a 90% improvement.

For backwards compatibility the podman binary still contains the option
for the reexec. I will remove this for podman 4.0.

The Makefile has been updated to compile the new binary and install it
to the libexec directory.

Fixes containers#10790

[NO TESTS NEEDED]

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Luap99 added a commit to Luap99/libpod that referenced this issue Oct 6, 2021
@Luap99
rootlessport: reduce memory usage of the process
2a55a97 
Don't use reexec for the rootlessport process, instead make it a
separate binary to reduce the memory usage. The problem with reexec is
that it will import all packages that podman uses and therefore loads a
lot of stuff into the heap. The rootlessport process however only needs
the rootlesskit library.
The memory usage is a concern since the rootlessport process will spawn
two process per container which has ports forwarded. The processes stay
until the container dies. On my laptop the current reexec version uses
47800 KB RSS. The new separate binary only uses 4540 KB RSS. This is
more than a 90% improvement.

For backwards compatibility the podman binary still contains the option
for the reexec. I will remove this for podman 4.0.

The Makefile has been updated to compile the new binary and install it
to the libexec directory.

Fixes containers#10790

[NO TESTS NEEDED]

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Luap99 added a commit to Luap99/libpod that referenced this issue Oct 6, 2021
@Luap99
rootlessport: reduce memory usage of the process
ce3e659 
Don't use reexec for the rootlessport process, instead make it a
separate binary to reduce the memory usage. The problem with reexec is
that it will import all packages that podman uses and therefore loads a
lot of stuff into the heap. The rootlessport process however only needs
the rootlesskit library.
The memory usage is a concern since the rootlessport process will spawn
two process per container which has ports forwarded. The processes stay
until the container dies. On my laptop the current reexec version uses
47800 KB RSS. The new separate binary only uses 4540 KB RSS. This is
more than a 90% improvement.

For backwards compatibility the podman binary still contains the option
for the reexec. I will remove this for podman 4.0.

The Makefile has been updated to compile the new binary and install it
to the libexec directory.

Fixes containers#10790

[NO TESTS NEEDED]

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
@github-actions
Copy link

github-actions bot commented Oct 9, 2021

A friendly reminder that this issue had no activity for 30 days.

Luap99 added a commit to Luap99/libpod that referenced this issue Oct 11, 2021
@Luap99
rootlessport: reduce memory usage of the process
c51864f 
Don't use reexec for the rootlessport process, instead make it a
separate binary to reduce the memory usage. The problem with reexec is
that it will import all packages that podman uses and therefore loads a
lot of stuff into the heap. The rootlessport process however only needs
the rootlesskit library.
The memory usage is a concern since the rootlessport process will spawn
two process per container which has ports forwarded. The processes stay
until the container dies. On my laptop the current reexec version uses
47800 KB RSS. The new separate binary only uses 4540 KB RSS. This is
more than a 90% improvement.

The Makefile has been updated to compile the new binary and install it
to the libexec directory.

Fixes containers#10790

[NO TESTS NEEDED]

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Luap99 added a commit to Luap99/libpod that referenced this issue Oct 11, 2021
@Luap99
rootlessport: reduce memory usage of the process
bc4e311 
Don't use reexec for the rootlessport process, instead make it a
separate binary to reduce the memory usage. The problem with reexec is
that it will import all packages that podman uses and therefore loads a
lot of stuff into the heap. The rootlessport process however only needs
the rootlesskit library.
The memory usage is a concern since the rootlessport process will spawn
two process per container which has ports forwarded. The processes stay
until the container dies. On my laptop the current reexec version uses
47800 KB RSS. The new separate binary only uses 4540 KB RSS. This is
more than a 90% improvement.

The Makefile has been updated to compile the new binary and install it
to the libexec directory.

Fixes containers#10790

[NO TESTS NEEDED]

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Luap99 added a commit to Luap99/libpod that referenced this issue Oct 12, 2021
@Luap99
rootlessport: reduce memory usage of the process
6c24f4a 
Don't use reexec for the rootlessport process, instead make it a
separate binary to reduce the memory usage. The problem with reexec is
that it will import all packages that podman uses and therefore loads a
lot of stuff into the heap. The rootlessport process however only needs
the rootlesskit library.
The memory usage is a concern since the rootlessport process will spawn
two process per container which has ports forwarded. The processes stay
until the container dies. On my laptop the current reexec version uses
47800 KB RSS. The new separate binary only uses 4540 KB RSS. This is
more than a 90% improvement.

The Makefile has been updated to compile the new binary and install it
to the libexec directory.

Fixes containers#10790

[NO TESTS NEEDED]

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Luap99 added a commit to Luap99/libpod that referenced this issue Oct 12, 2021
@Luap99
rootlessport: reduce memory usage of the process
3ba69dc 
Don't use reexec for the rootlessport process, instead make it a
separate binary to reduce the memory usage. The problem with reexec is
that it will import all packages that podman uses and therefore loads a
lot of stuff into the heap. The rootlessport process however only needs
the rootlesskit library.
The memory usage is a concern since the rootlessport process will spawn
two process per container which has ports forwarded. The processes stay
until the container dies. On my laptop the current reexec version uses
47800 KB RSS. The new separate binary only uses 4540 KB RSS. This is
more than a 90% improvement.

The Makefile has been updated to compile the new binary and install it
to the libexec directory.

Fixes containers#10790

[NO TESTS NEEDED]

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
@yogo1212
Copy link

yogo1212 commented Mar 16, 2023 •
edited
Loading

maybe it's worth considering re-opening this:

Out of memory: Killed process 5360 (rootlessport) total-vm:2988372kB, anon-rss:1696352kB, file-rss:0kB, shmem-rss:0kB, UID:1003 pgtables:3832kB oom_score_adj:200

$ podman --version
podman version 4.2.0

Rocky Linux 9.1, Container in question is coturn.

podman run --rm --name "coturn" \
  -p '3478:3478/udp' -p '[::]:3478:3478/udp' \
  -p '5349:5349/udp' -p '[::]:5349:5349/udp' \ 
  -p '49152-65535:49152-65535/udp' -p '[::]:49152-65535:49152-65535/udp'  \
  --mount type=tmpfs,destination=/var/lib/coturn \
  docker.io/coturn/coturn

EDIT: i've tried to get a coredump before the process dies but it's incredibly difficult because my ssh-session freezes...

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Aug 29, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Aug 29, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@Luap99 Luap99

Labels
kind/performance locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. rootless
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

rootlessport: reduce memory usage of the process Luap99/libpod
6 participants
@rhatdan @yogo1212 @mheon @AkihiroSuda @bear0330 @Luap99