cephadm: Remove containers pids-limit

[asm0deuz]

cephadm: Remove containers pids-limit

The default pids-limit (docker 4096/podman 2048) prevent some
customization from working (http threads on RGW) or limits the number
of luns per iscsi target.
Replaces existing PR #42214

Fixes: https://tracker.ceph.com/issues/52898

Signed-off-by: Teoman ONAY tonay@redhat.com

[idryomov]

Superficially, LGTM. I'd like to see this tested, if not by creating 255 iSCSI LUNs, then at least by spawning a few thousand e.g. sleep processes in both tcmu-runner and rbd-target-api containers.

[asm0deuz]

@idryomov Did a sleep loop on both containers related to iscsi:

[root@ceph-1 ~]# podman ps     
CONTAINER ID  IMAGE                                                                                              COMMAND               CREATED         STATUS             PORTS       NAMES
6dbaa3a177fb  quay.ceph.io/ceph-ci/ceph:master                                                                   -n mon.ceph-1 -f ...  47 minutes ago  Up 47 minutes ago              ceph-21c63de8-753e-11ec-bc0d-
5254008362ff-mon-ceph-1                             
13afea0d7d14  quay.ceph.io/ceph-ci/ceph:master                                                                   -n mgr.ceph-1.hpa...  47 minutes ago  Up 47 minutes ago              ceph-21c63de8-753e-11ec-bc0d-
5254008362ff-mgr-ceph-1-hparpn                                                                           
4bae29fc0d96  quay.ceph.io/ceph-ci/ceph@sha256:9d4f37582847bafd702d2e848bc9e785f1561a1d51823968de9f62df9709403f  -n mgr.ceph-1.gnt...  46 minutes ago  Up 46 minutes ago              ceph-21c63de8-753e-11ec-bc0d-
5254008362ff-mgr-ceph-1-gntrmx                                                                           
b87b0549db61  quay.ceph.io/ceph-ci/ceph@sha256:9d4f37582847bafd702d2e848bc9e785f1561a1d51823968de9f62df9709403f  -n client.crash.c...  46 minutes ago  Up 46 minutes ago              ceph-21c63de8-753e-11ec-bc0d-
5254008362ff-crash-ceph-1 
af7bede1fa30  quay.ceph.io/ceph-ci/ceph@sha256:9d4f37582847bafd702d2e848bc9e785f1561a1d51823968de9f62df9709403f  -n osd.0 -f --set...  43 minutes ago  Up 43 minutes ago              ceph-21c63de8-753e-11ec-bc0d-
5254008362ff-osd-0                                                                                       
f309c8e229ac  quay.ceph.io/ceph-ci/ceph@sha256:9d4f37582847bafd702d2e848bc9e785f1561a1d51823968de9f62df9709403f                        42 minutes ago  Up 42 minutes ago              ceph-21c63de8-753e-11ec-bc0d-
5254008362ff-iscsi-iscsi-ceph-1-ejymjq-tcmu                                                              
470bf3e9c065  quay.ceph.io/ceph-ci/ceph@sha256:9d4f37582847bafd702d2e848bc9e785f1561a1d51823968de9f62df9709403f                        42 minutes ago  Up 42 minutes ago              ceph-21c63de8-753e-11ec-bc0d-
5254008362ff-iscsi-iscsi-ceph-1-ejymjq                                                                   
[root@ceph-1 ~]# podman exec -it 470bf3e9c065 /bin/bash
(reverse-i-search)`3500': for i in {0..3500}; do sleep 180 & done
.
.
.
[3494] 18682
[3495] 18683
[3496] 18684
[3497] 18685
[3498] 18686
[3499] 18687
[3500] 18688
[3501] 18689
[root@ceph-1 /]# ps -ef | grep sleep | wc -l
3502
[root@ceph-1 ~]# podman exec -it f309c8e229ac /bin/bash
[root@ceph-1 /]# for i in {0..3500}; do sleep 180& done
.

[3496] 3524
[3497] 3525
[3498] 3526
[3499] 3527
[3500] 3528
[3501] 3529
[root@ceph-1 /]# ps -ef | grep sleep | wc -l
3502

[idryomov]

Why not 10000 processes (or at least something bigger than 4096)? Are you sure that the default value for podman is actually 2048 and not 4096? This is a quote from podman-run man page:

Tune the container’s pids limit. Set to -1 to have unlimited pids for the container.
The default is 4096 on systems that support “pids” cgroup controller.

[asm0deuz]

@idryomov

Just did it for 20K processes:

[23500] 23537
[23501] 23538
[23502] 23539
[23503] 23540
[root@ceph-1 /]# ps -ef | grep sleep | wc -l
20003
[root@ceph-1 /]# 

Regarding that limit of 4096 or 2048, I just checked podman documentation and it says now that the limit is 4096:

--pids-limit=limit
Tune the container’s pids limit. Set to -1 to have unlimited pids for the container. The default is 4096 on systems that support “pids” cgroup controller.

When I did the same change in ceph-ansible it was 2048 in the meantime they did the change in podman to match docker value:

commit 118cf1fc634ffc63b908d6b082ffc3a53553a6af
Author: Daniel J Walsh dwalsh@redhat.com
Date: Sat Sep 14 06:21:10 2019 -0400

Setup a reasonable default for pids-limit 4096

CRI-O defaults to 1024 for the maximum pids in a container.  Podman
should have a similar limit. Once we have a containers.conf, we can
set the limit in this file, and have it easily customizable.

Currently the documentation says that -1 sets pids-limit=max, but -1 fails.
This patch allows -1, but also indicates that 0 also sets the max pids limit.

Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>

[idryomov]

Great, thanks! Looks like you need to adjust TestIscsi test case.

[sebastian-philipp]

as you already figured out the shell commands to run: would be great to have this in teuthology as well. Do you want to add a new workunit task that does it?

https://github.com/ceph/ceph/tree/master/qa/suites/orch/cephadm/workunits

Maybe something like this could work:

roles:
- - host.a
  - osd.0
  - osd.1
  - osd.2
  - mon.a
  - mgr.a
  - client.0
tasks:
- install:
- cephadm:
- cephadm.shell:
    host.a:
      - ceph orch apply iscsi iscsi
- workunit:
    clients:
      client.0:
        - cephadm/test_iscsi_pid_limit.sh

??

[idryomov]

jenkins test make check

[idryomov]

I'm not sure why, but with docker on Ubuntu Focal this issue doesn't seem to reproduce in teuthology. I went with restricting this test to podman on CentOS Stream 8 (where the problem was reported and reproduces) and tweaked the workunit script a bit as the switch between docker and podman is no longer needed.

I also added an assert on CONT_COUNT after accidentally running into a case where the test would succeed without actually running anything due to ISCSI_CONT_IDS being empty.

[idryomov]

https://pulpito.ceph.com/dis-2022-03-22_20:43:37-orch:cephadm-wip-dis-testing-distro-default-smithi/
https://pulpito.ceph.com/dis-2022-03-22_20:44:06-orch:cephadm-wip-dis-testing-1-distro-default-smithi/

Without the fix:

2022-03-22T21:06:36.102 INFO:tasks.workunit.client.0.smithi136.stderr:+ for i in ${ISCSI_CONT_IDS}
2022-03-22T21:06:36.102 INFO:tasks.workunit.client.0.smithi136.stderr:+ sudo podman exec ae56b8245120 /bin/sh -c 'for j in {0..20000}; do sleep 30 & done'
2022-03-22T21:06:36.704 INFO:tasks.workunit.client.0.smithi136.stderr:/bin/sh: fork: retry: Resource temporarily unavailable
2022-03-22T21:06:37.705 INFO:tasks.workunit.client.0.smithi136.stderr:/bin/sh: fork: retry: Resource temporarily unavailable
2022-03-22T21:06:39.705 INFO:tasks.workunit.client.0.smithi136.stderr:/bin/sh: fork: retry: Resource temporarily unavailable
2022-03-22T21:06:43.706 INFO:tasks.workunit.client.0.smithi136.stderr:/bin/sh: fork: retry: Resource temporarily unavailable
2022-03-22T21:06:51.706 INFO:tasks.workunit.client.0.smithi136.stderr:/bin/sh: fork: Resource temporarily unavailable
2022-03-22T21:06:51.891 DEBUG:teuthology.orchestra.run:got remote process result: 254

With the fix:

2022-03-22T21:06:30.808 INFO:tasks.workunit.client.0.smithi040.stderr:+ for i in ${ISCSI_CONT_IDS}
2022-03-22T21:06:30.808 INFO:tasks.workunit.client.0.smithi040.stderr:+ sudo podman exec c315377940f9 /bin/sh -c 'for j in {0..20000}; do sleep 30 & done'
2022-03-22T21:06:38.887 INFO:tasks.workunit.client.0.smithi040.stderr:+ for i in ${ISCSI_CONT_IDS}
2022-03-22T21:06:38.888 INFO:tasks.workunit.client.0.smithi040.stderr:+ sudo podman exec 072965d17e08 /bin/sh -c 'for j in {0..20000}; do sleep 30 & done'
2022-03-22T21:06:46.968 INFO:tasks.workunit.client.0.smithi040.stderr:+ for i in ${ISCSI_CONT_IDS}
2022-03-22T21:06:46.968 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo podman exec c315377940f9 /bin/sh -c 'ps -ef | grep -c sleep'
2022-03-22T21:06:48.032 INFO:tasks.workunit.client.0.smithi040.stderr:+ SLEEP_COUNT=20003
2022-03-22T21:06:48.032 INFO:tasks.workunit.client.0.smithi040.stderr:+ test 20003 -gt 20000
2022-03-22T21:06:48.032 INFO:tasks.workunit.client.0.smithi040.stderr:+ for i in ${ISCSI_CONT_IDS}
2022-03-22T21:06:48.032 INFO:tasks.workunit.client.0.smithi040.stderr:++ sudo podman exec 072965d17e08 /bin/sh -c 'ps -ef | grep -c sleep'
2022-03-22T21:06:49.089 INFO:tasks.workunit.client.0.smithi040.stderr:+ SLEEP_COUNT=20003
2022-03-22T21:06:49.089 INFO:tasks.workunit.client.0.smithi040.stderr:+ test 20003 -gt 20000
2022-03-22T21:06:49.090 INFO:tasks.workunit.client.0.smithi040.stdout:OK