test_shutil fails when gid name is missing

[DineshWijekoon]

BPO	26067
Nosy	@ezio-melotti, @DinoV, @serhiy-storchaka, @MatzeB, @iritkatriel
PRs	bpo-26067: Do not fail test_shutil / chown when gid/uid cannot be resolved #19032

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-11-20.15:13:08.982>
created_at = <Date 2016-01-09.21:10:35.713>
labels = ['extension-modules', 'type-bug']
title = 'test_shutil fails when gid name is missing'
updated_at = <Date 2020-11-20.15:13:08.981>
user = 'https://bugs.python.org/DineshWijekoon'

bugs.python.org fields:

activity = <Date 2020-11-20.15:13:08.981>
actor = 'iritkatriel'
assignee = 'none'
closed = True
closed_date = <Date 2020-11-20.15:13:08.982>
closer = 'iritkatriel'
components = ['Extension Modules']
creation = <Date 2016-01-09.21:10:35.713>
creator = 'Dinesh Wijekoon'
dependencies = []
files = []
hgrepos = []
issue_num = 26067
keywords = ['patch']
message_count = 6.0
messages = ['257858', '257859', '258026', '364356', '364445', '378373']
nosy_count = 7.0
nosy_names = ['ezio.melotti', 'dino.viehland', 'python-dev', 'serhiy.storchaka', 'Dinesh Wijekoon', 'Matthias Braun', 'iritkatriel']
pr_nums = ['19032']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue26067'
versions = ['Python 3.6']

[DineshWijekoon]

./python.exe -m test -j3 -v test_shutil

The above test fails when id name is missing. The fail message is
group = grp.getgrgid(gid)[0]
KeyError: 'getgrgid(): gid not found: 203135016'

Following is the results from console "id" command, when the failure happens.

uid=1336551206(athukora) gid=203135016 groups=203135016,402(com.apple.sharepoint.group.2),12(everyone),62(netaccounts),80(admin),401(com.apple.sharepoint.group.1),33(_appstore),98(_lpadmin),100(_lpoperator),204(_developer),398(com.apple.access_screensharing),399(com.apple.access_ssh)

And again when I change user to sudo the "id" command returns the following and test get passed.
uid=0(root) gid=0(wheel) groups=0(wheel),1(daemon),2(kmem),3(sys),4(tty),5(operator),8(procview),9(procmod),12(everyone),20(staff),29(certusers),61(localaccounts),80(admin),401(com.apple.sharepoint.group.1),402(com.apple.sharepoint.group.2),33(_appstore),98(_lpadmin),100(_lpoperator),204(_developer),398(com.apple.access_screensharing),399(com.apple.access_ssh)

PS:

We tried to debug this a bit and found the bug is possibly in Modules/grpmodule.c, may be at method grp_getgrall_impl. Adding "PyString_AsString(v);" before if command " PyList_Append(d, v) != 0" will fix the issue in test. But we have no idea how its working with this change.

[DineshWijekoon]

This reported bug occurred in Mac OS.

[ezio-melotti]

We looked at this at the PyLadies Sprint in Helsinki.
When the group has an id but not a name (see "gid=203135016" in the "id" output -- the name is missing), its entry is missing from the list returned by grp.getgrall().

If this behaviour is expected and desired, then the test should be fixed to handle cases where the name is missing and the KeyError is raised.
Otherwise, the entry could still be included with "None" as name.

[MatzeB]

I have the same issue on my work macbook which has some form of network login configured. The primary group id of my user account there cannot be found in the group database and cannot be resolved to a name.

I'm not sure why that is but I am pretty sure there is no obligation that a gid must be resolvable to a name. I submitted a PR to just skip the last part of the test_shutil/chown if the uid/gid cannot be resolved to a name.

[DinoV]

New changeset 5226894 by Matthias Braun in branch 'master':
bpo-26067: Do not fail test_shutil / chown when gid/uid cannot be resolved (bpo-19032)
5226894

[iritkatriel]

This seems complete, can it be closed?