PAM responder delays with to many requests

[jbd]

Hello everybody,

tldr, it seems that the hardcoded backlog value (10) used in the listen call of src/responder/common/responder_common.c can introduce delays.

https://github.com/SSSD/sssd/blob/2.6.3/src/responder/common/responder_common.c#L887

We've got servers with 2x48 cores in them running sssd, with an ldap provider. We are using the slurm job scheduler to let users run their jobs on the cluster.

From time to time, we've got timeout issue from the slurm daemon on the compute node which is trying to retrieve the user environnement by (literally) running /bin/su - <username> -c /usr/bin/env.

https://github.com/SchedMD/slurm/blob/slurm-21-08-6-1/src/common/env.c#L1978

This can take time for multiple reasons, but we were still able to reproduce the timeout problem without executing login scripts (and deactivating pam_lastlog which can also introduce a one second delay as it try to lock /var/log/lastlog).

For example, here is a reproducer (without the su --login option, pam_lastlog deactivated):

# for i in {1..64}; do time /bin/su jbdenis -c /usr/bin/env & done |& grep real; wait
real	0m0.178s
real	0m0.184s
[...]   # a few dozens
real	0m0.178s
real	0m0.174s
real	0m0.185s
real	0m0.178s
real	0m1.154s
real	0m1.154s
real	0m1.148s
real	0m1.156s
real	0m1.147s
real	0m1.146s
real	0m1.157s
real	0m1.157s
real	0m1.146s
real	0m1.150s
real	0m1.150s
real	0m1.149s
real	0m1.154s
real	0m1.152s
real	0m1.152s
real	0m1.149s
real	0m1.151s
real	0m1.154s
real	0m1.147s
real	0m1.148s

You've got tow timeout groups: d <1s and 1s < d < 2s. This is a synthetic reproducer but that's exactly what can occured on a compute node when a logt of jobs

We "patched" the listen call from src/responder/common/responder_common.c using a listen function wrapper in an .so file (it was simpler in our context) using 128 as backlog parameter and checked that we only had 99% <1s answers after the change. (We used this recipe and patchelf --add-needed instead of LD_PRELOAD: https://access.redhat.com/solutions/3314151).

What do you think about it ?

Thank you for your help.

Jean-Baptiste

[alexey-tikhonov]

Hi,

thanks for your test. I wouldn't object backlog size increase.

I'd suggest to open a PR. Not sure if it's worth making size configurable...

[alexey-tikhonov]

Hi @sumit-bose,

what would you say about deducing listen() backlog size from fd_limit responder conf setting?
For example, fd_limit / 10?

[sumit-bose]

Hi,

this would work as well. But it looks like PR #6036 already increased the size to 128 to a value which is causing no issues for the reporter and we just forgot to close this ticket here. So I would wait until we come across an use-case where a larger or more flexible backlog size is needed.

bye,
Sumit

[alexey-tikhonov]

Ok, thank you.

So: fixed via 91e8c4f