lua treats ENOENT as success in getpwall/getgrall

[avtikhon]

Tarantool version:
master

OS version:
CentOS 8

Bug description:
lua treats ENOENT as success in getpwall/getgrall
Issue described at systemd/systemd#9585
Previously suggested fix at 7732dad
Error log shows:
builtin/pwd.lua:169: getpwall failed [errno 2]: No such file or directory

Steps to reproduce:
./test-run.py app-tap/pwd.test.lua

Optional (but very desirable):

• coredump
• backtrace
• netstat

[Totktonada]

Yep, systemd/systemd#9585 was closed in systemd v240, while CentOS 8 offers v239 (at least for now). /etc/nsswitch.conf contains systemd entries for passwd and group:

(CentOS 8) $ grep '^passwd\|^group' /etc/nsswitch.conf
passwd:      sss files systemd
group:       sss files systemd

CentOS 7 has no systemd entries here:

(CentOS 7) $ grep '^passwd\|^group' /etc/nsswitch.conf
passwd:     files sss
group:      files sss

That is why we don't see exaclty this problem on CentOS 7.

[Totktonada]

pwd.getpwall() hangs now on CentOS 7 (with some specific configuration) and due to commit f5d8331 it leads to tarantool hang at startup. However after several minutes it gives control back.

I'll update this comment with more info, asked it from the user.

[Totktonada]

Jira: TNT-13.

[ligurio]

can't reproduce

systemd-libs-239-18.el8_1.5.x86_64
systemd-pam-239-18.el8_1.5.x86_64
systemd-239-18.el8_1.5.x86_64
systemd-udev-239-18.el8_1.5.x86_64

$ ../../test/test-run.py --builddir=/home/centos/tarantool/build --vardir=/home/centos/tarantool/build/test/var app-tap/pwd.test.lua
<snipped>
======================================================================================
WORKR TEST                                            PARAMS          RESULT
---------------------------------------------------------------------------------
[001] app-tap/pwd.test.lua                                            [ pass ]
---------------------------------------------------------------------------------
$ cat /etc/redhat-release                                                                                         
CentOS Linux release 8.1.1911 (Core)

on ci passed too - https://gitlab.com/tarantool/tarantool/pipelines/148600403

@avtikhon, please help to reproduce

[avtikhon]

I've tried all suggestions mentioned in our chats, tried to rollback all commits step by step to src/lua/pwd.lua, even tried different changes in the file on CentOS 8 and Fedora 29, and failed to reproduce the issue. The only possible way that I've found is to change the /etc/nsswitch.conf file in wrong way to reproduce the ENOENT in getgrall function:

passwd:      sss files ldap
group:       sss files systemd

Anyway this error mentioned in getgrent as:

       ENOENT ... For NSS backends in glibc this indicates the backend is not correctly configured.

And seems that the root issue wasn't reproduced.

Trying to run on dev1 host a lot of runs of the app-tap/pwd.test.lua test, got only issue with command:

cd /export/avtikhon/src/test && while time ./test-run.py --builddir ../../bld --long -j 160 `for r in {1..200} ; do echo app-tap/pwd.test.lua ; done` ; do date ; done

that once in 400 test run the test hangs for a half of a minute and then continues.

Also the error message is not user friendly:

local pwgr_errstr = "get%s failed [errno %d]: %s"
produces:
[001] LuajitError: builtin/pwd.lua:101: getgetgrall failed [errno 2]: No such file or directory
while it has to be:
[001] LuajitError: builtin/pwd.lua:101: get getgrall failed [errno 2]: No such file or directory

Seems

[Totktonada]

I unable to reproduce it too. Tried in a CentOS 8 docker container and also on a virtual machine with CentOS 8.

I found a chat log with Alexander related to this problem. It was on a work-in-progress patch to support CentOS 8 and there were suspectful errors about libcurl.so (see below). So maybe this pwd test problem was infuenced by this.

[001] app-tap/pwd.test.lua                                            [ fail ]
[001] Test failed! Output from reject file app-tap/pwd.reject:
[001] TAP version 13
[001] 1..6
[001] ok - checking user by id
[001] ok - checking user by name
[001] ok - checking group by id
[001] ok - checking group by name
[001] 
[001] Last 15 lines of Tarantool Log file [Instance "app_server"][/mnt/test/var/001_app-tap/pwd.test.lua.tarantool.log]:
[001] tarantool: /lib64/libcurl.so.4: no version information available (required by tarantool)
[001] builtin/pwd.lua:169: getpwall failed [errno 2]: No such file or directory
[Main process] Got failed test; gently terminate all workers...
[001] Worker "001_app-tap" got failed test; stopping the server...
---------------------------------------------------------------------------------
Statistics:
* fail: 1
Failed tasks:
- [app-tap/pwd.test.lua, null]

@avtikhon Let's close the issue with removing pwd.skipcond file.

Filed #5034 to fix the startup freeze.