-
-
Notifications
You must be signed in to change notification settings - Fork 31.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test_importlib: test_multiprocessing_pool_circular_import() fails with "Too many open files" error on os.pipe() #87758
Comments
x86-64 macOS 3.x: Build triggered by the commit 88d9983 which changes PyImport_Import(). The buildbot is running on macOS 10.15.7 (Darwin Kernel Version 19.6.0) with a limit of 256 file descriptors. The latest successful build is build 968, whereas the the RLIMIT_NOFILE resource soft limit was also set to 256: test.pythoninfo:
FAIL: test_multiprocessing_pool_circular_import (test.test_importlib.test_threaded_import.ThreadedImportTests) Traceback (most recent call last):
File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/test_threaded_import.py", line 258, in test_multiprocessing_pool_circular_import
script_helper.assert_python_ok(fn)
File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
return _assert_python(True, *args, **env_vars)
File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py", line 145, in _assert_python
res.fail(cmd_line)
File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/support/script_helper.py", line 72, in fail
raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/Users/buildbot/buildarea/3.x.billenstein-macos/build/python.exe', '-X', 'faulthandler', '-I', '/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py'] stdout: --- stderr: Traceback (most recent call last):
File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Pool
File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/pool.py", line 196, in __init__
File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/context.py", line 113, in SimpleQueue
File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/queues.py", line 341, in __init__
self._reader, self._writer = connection.Pipe(duplex=False)
File "/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/connection.py", line 532, in Pipe
fd1, fd2 = os.pipe()
OSError: [Errno 24] Too many open files
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 110 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-56i6i4ap': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-0au5otkl': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-vcv0xwbi': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-vxfb4ks9': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-5e2_0z1f': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-6vsgax4k': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-mq51g4b_': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-eik0n2aq': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-g7oeb4aw': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-tiabsvgr': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-ykag01b2': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.x.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-nl2kdidn': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e)) |
test_importlib only fails *randomly* on x86-64 macOS 3.x. For example, the build 970 is a success (but orange): "Tests result: FAILURE then SUCCESS". test_importlib failed with too many open files, and then passed. |
FWIW, I don't recall ever seeing a "too many open files" failure for test_importlib on any of my macOS systems. |
I have the same error when running the tests on my machine (Same macOS version). |
x86-64 macOS 3.10: ====================================================================== Traceback (most recent call last):
File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/test_threaded_import.py", line 258, in test_multiprocessing_pool_circular_import
script_helper.assert_python_ok(fn)
File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py", line 160, in assert_python_ok
return _assert_python(True, *args, **env_vars)
File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py", line 145, in _assert_python
res.fail(cmd_line)
File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/support/script_helper.py", line 72, in fail
raise AssertionError("Process return code is %d\n"
AssertionError: Process return code is 1
command line: ['/Users/buildbot/buildarea/3.10.billenstein-macos/build/python.exe', '-X', 'faulthandler', '-I', '/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py'] stdout: --- stderr: Traceback (most recent call last):
File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/test/test_importlib/partial/pool_in_threads.py", line 9, in t
with multiprocessing.Pool(1):
File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/context.py", line 119, in Pool
return Pool(processes, initializer, initargs, maxtasksperchild,
File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py", line 212, in __init__
self._repopulate_pool()
File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py", line 303, in _repopulate_pool
return self._repopulate_pool_static(self._ctx, self.Process,
File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/pool.py", line 326, in _repopulate_pool_static
w.start()
File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/process.py", line 121, in start
File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/context.py", line 284, in _Popen
return Popen(process_obj)
File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_spawn_posix.py", line 32, in __init__
File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_fork.py", line 19, in __init__
self._launch(process_obj)
File "/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/popen_spawn_posix.py", line 53, in _launch
OSError: [Errno 24] Too many open files
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 120 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-klnfwvgv': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e))
/Users/buildbot/buildarea/3.10.billenstein-macos/build/Lib/multiprocessing/resource_tracker.py:237: UserWarning: resource_tracker: '/mp-phgypx9f': [Errno 2] No such file or directory
warnings.warn('resource_tracker: %r: %s' % (name, e)) |
On this machine, RLIMIT_NOFILE soft limit is 256. test.pythoninfo: resource.RLIMIT_NOFILE: (256, 9223372036854775807) All limits: resource.RLIMIT_AS: (9223372036854775807, 9223372036854775807) |
Try setting the following: $ sudo sysctl -w kern.maxfiles=65536
$ sudo sysctl -w kern.maxfilesperproc=65536
$ sudo ulimit -n 65536 |
Before you do it, check the defaults with: $ launchctl limit maxfiles I'm curious. |
FYI the test is still failing with the same error on the billenstein-macos builder worker, x86-64 macOS 3.10: "OSError: [Errno 24] Too many open files" |
macOS often has an unreasonably low default max number of file descriptors. This is easy to raise from user code, it's a soft limit. is out to try that from regrtest itself. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: