os.getgrouplist() can fail on macOS if the user has more than 17 groups

[corona10]

BPO	40014
Nosy	@ronaldoussoren, @vstinner, @ned-deily, @zooba, @corona10, @miss-islington, @shihai1991
PRs	bpo-40014: Skip os.getgrouplist related test if OSError is raised. #19075 bpo-40014: test.pythoninfo catchs os.getgrouplist() error #19117 bpo-40014: Fix os.getgrouplist() failure on macOS with many groups #19118 [3.8] bpo-40014: Fix os.getgrouplist() on macOS (GH-19118) #19123 [3.7] bpo-40014: Fix os.getgrouplist() on macOS (GH-19118) #19124 bpo-40014: Fix os.getgrouplist() #19126 [3.8] bpo-40014: Fix os.getgrouplist() (GH-19126) #19143 [3.7] bpo-40014: Fix os.getgrouplist() (GH-19126) #19144

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-03-24.17:51:27.264>
created_at = <Date 2020-03-19.16:04:54.967>
labels = ['OS-mac', '3.7', '3.8', '3.9', 'type-feature', 'tests']
title = 'os.getgrouplist() can fail on macOS if the user has more than 17 groups'
updated_at = <Date 2020-03-24.17:51:27.264>
user = 'https://github.com/corona10'

bugs.python.org fields:

activity = <Date 2020-03-24.17:51:27.264>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-03-24.17:51:27.264>
closer = 'vstinner'
components = ['macOS', 'Tests']
creation = <Date 2020-03-19.16:04:54.967>
creator = 'corona10'
dependencies = []
files = []
hgrepos = []
issue_num = 40014
keywords = ['patch']
message_count = 17.0
messages = ['364607', '364608', '364610', '364615', '364862', '364864', '364870', '364880', '364884', '364885', '364888', '364889', '364904', '364950', '364955', '364956', '364957']
nosy_count = 7.0
nosy_names = ['ronaldoussoren', 'vstinner', 'ned.deily', 'steve.dower', 'corona10', 'miss-islington', 'shihai1991']
pr_nums = ['19075', '19117', '19118', '19123', '19124', '19126', '19143', '19144']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue40014'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[corona10]

example: https://github.com/python/cpython/pull/19073/checks?check_run_id=519539592

I suggest to not to add information for os.getgrouplist
if the OSError is raised.

[corona10]

or adding <OSError> might be great just like getpwuid

[vstinner]

These errors are really strange.

Steve Dower: are you aware of anything recent change on the macOS job of Azure Pipelines / GH Actions.

pythoninfo:
---
os.uname: posix.uname_result(sysname='Darwin', nodename='Mac-1422.local', release='19.3.0', version='Darwin Kernel Version 19.3.0: Thu Jan 9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64', machine='x86_64')

platform.architecture: 64bit

platform.platform: macOS-10.15.3-x86_64-i386-64bit
---

example: https://github.com/python/cpython/pull/19073/checks?check_run_id=519539592

./python.exe -m test.pythoninfo
ERROR: collect_pwd() failed
Traceback (most recent call last):
  File "/Users/runner/runners/2.165.2/work/cpython/cpython/Lib/test/pythoninfo.py", line 761, in collect_info
    collect_func(info_add)
  File "/Users/runner/runners/2.165.2/work/cpython/cpython/Lib/test/pythoninfo.py", line 336, in collect_pwd
    groups = os.getgrouplist(entry.pw_name, entry.pw_gid)
OSError: [Errno 0] Error

Collection failed: exit with error

PR 19075

test_posix.test_getgrouplist() failed with:

======================================================================
ERROR: test_getgrouplist (test.test_posix.PosixTester)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/runner/runners/2.165.2/work/cpython/cpython/Lib/test/test_posix.py", line 1028, in test_getgrouplist
    self.assertIn(group, posix.getgrouplist(user, group))
OSError: [Errno 25] Inappropriate ioctl for device

[ned-deily]

This error has to do with the number of groups a particular user is associated with. We’ve squashed similar bugs in the past but it looks like something has changed again in recent releases of macOS. It’s not Azure specific.

[vstinner]

tl; dr os.getgrouplist() is limited to 17 groups and fails with a random OSError if the request user has more groups. PR 19118 fix the issue.

--

On macOS-10.15.1-x86_64-i386-64bit (python -m platform), os.getgrouplist() fails with my user name and group identifier:

>>> os.getgrouplist('haypo', 20)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 0] Error

Full example:

macbook:master haypo$ ./python.exe 
Python 3.9.0a4+ (heads/master:da2914d, Mar 20 2020, 09:45:36) 
[Clang 11.0.0 (clang-1100.0.33.8)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, pwd
>>> entry=pwd.getpwuid(os.getuid())

>>> os.getgrouplist(entry.pw_name, entry.pw_gid)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
InterruptedError: [Errno 4] Interrupted system call

>>> os.getgrouplist(entry.pw_name, entry.pw_gid)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [Errno 0] Error

>>> entry
pwd.struct_passwd(pw_name='haypo', pw_passwd='********', pw_uid=502, pw_gid=20, pw_gecos='Victor Stinner', pw_dir='/Users/haypo', pw_shell='/bin/bash')

My user has the following groups:

macbook:master haypo$ id
uid=502(haypo) gid=20(staff) groups=20(staff),702(com.apple.sharepoint.group.2),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),701(com.apple.sharepoint.group.1),33(_appstore),100(_lpoperator),204(_developer),250(_analyticsusers),395(com.apple.access_ftp),398(com.apple.access_screensharing),399(com.apple.access_ssh),400(com.apple.access_remote_ae),703(com.apple.sharepoint.group.3)

My user has 18 groups.

MAX_GROUPS=16, Python uses 1 + MAX_GROUPS.

getgrouplist() manual page says:

RETURN VALUES
The getgrouplist() function returns 0 on success. If the size of the group list is too small to hold
all the user's groups, getgrouplist() returns -1 to indicate failure. In this case, the group array
will be filled with as many groups as will fit.

In short, getgrouplist() doesn't set errno on failure.

[vstinner]

Python 3.7 and 3.8 are also affected on macOS 10.15 (Catalina).

[ned-deily]

Thanks for the PR, Victor, it does address the problem. I went back and explicitly tested on macOS 10.14 and 10.13 by creating some more groups and adding the user to them and the same failure occurred there so this is not just a 10.15 issue. Most likely this hasn't been an issue before is that 10.15 creates more groups for system services than the earlier versions did. So the PR should be updated to remove the 10.15 (and 10.5!) references.

[vstinner]

Most likely this hasn't been an issue before is that 10.15 creates more groups for system services than the earlier versions did.

Aha, interesting. I updated my PR to remove mentions of the macOS version number. I just wrote "on macOS". Yeah, I wouldn't be surprised that the function always failed.

--

I'm not sure that the Linux implementation is correct. It rely on errno, whereas Linux manual page of getgrouplist() doesn't say anything about errno being set on error.

Moreover, on Linux, getgrouplist() can also fail with -1 if the group list is too small. In that case, ngroups is updated to the number of groups and so can be used to enlarge the list!

[vstinner]

test.pythoninfo of the macOS job on the PR:

os.getgrouplist: 20, 0, 12, 61, 79, 80, 81, 98, 264, 399, 701, 33, 100, 204, 250, 395, 398, 400
os.getgroups: 20, 0, 12, 61, 79, 80, 81, 98, 264, 399, 701, 33, 100, 204, 250, 395, 398, 400

So yeah, there are 18 groups ;-)

[vstinner]

New changeset 8ec7370 by Victor Stinner in branch 'master':
bpo-40014: Fix os.getgrouplist() on macOS (GH-19118)
8ec7370

[miss-islington]

New changeset 21bee0b by Miss Islington (bot) in branch '3.8':
bpo-40014: Fix os.getgrouplist() on macOS (GH-19118)
21bee0b

[miss-islington]

New changeset 1cdc61c by Miss Islington (bot) in branch '3.7':
bpo-40014: Fix os.getgrouplist() on macOS (GH-19118)
1cdc61c

[vstinner]

Moreover, on Linux, getgrouplist() can also fail with -1 if the group list is too small. In that case, ngroups is updated to the number of groups and so can be used to enlarge the list!

I tested by manually initializing ngroups to a value way lower than MAX_GROUPS (65536 on my Fedora 31): os.getgrouplist() raises OSError() with a random errno, getgrouplist() doesn't set errno on failure.

I wrote PR 19126 to fix the issue on all platforms. The glibc implementation is detected and used: use updated ngroups value (only if it's larger).

[vstinner]

New changeset f5c7cab by Victor Stinner in branch 'master':
bpo-40014: Fix os.getgrouplist() (GH-19126)
f5c7cab

[miss-islington]

New changeset 5753fc6 by Miss Islington (bot) in branch '3.7':
bpo-40014: Fix os.getgrouplist() (GH-19126)
5753fc6

[miss-islington]

New changeset af6fd1f by Miss Islington (bot) in branch '3.8':
bpo-40014: Fix os.getgrouplist() (GH-19126)
af6fd1f

[vstinner]

Ned: so Linux was also impacted, but Linux only has an issue with more than 65536 groups :-D macOS MAX_GROUPS is now only 16 (Python uses MAX_GROUPS+1)! Maybe MAX_GROUPS was reduced recently. I'm not sure.

Anyway, os.getgrouplist() does no longer depend on MAX_GROUPS hardcoded limit, but grow the group list dynamically. It should now work with any number of groups an all platforms (which provide the getgrouplist() function ;-)) on Python 3.7, 3.8 and master branches.

Thanks Dong-hee Na for the bug report and your PR.