"Too many open files" when running ./mach test-wpt

[atbrakhi]

Running ./mach test-wpt gets stuck for some time and then throws the error below.

Tested on macOS

➜  servo git:(master) ./mach test-wpt                                                   
Running 34226 tests in web-platform-tests

  [3826/34226] No tests running.
Error running mach:

    ['test-wpt']
    
The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

OSError: [Errno 24] Too many open files

  File "/Users/atbrakhi/Documents/igalia/servo/python/servo/testing_commands.py", line 363, in test_wpt
    ret = self.run_test_list_or_dispatch(kwargs["test_list"], "wpt", self._test_wpt, **kwargs)
  File "/Users/atbrakhi/Documents/igalia/servo/python/servo/testing_commands.py", line 392, in run_test_list_or_dispatch
    return correct_function(**kwargs)
  File "/Users/atbrakhi/Documents/igalia/servo/python/servo/testing_commands.py", line 387, in _test_wpt
    return wpt.run.run_tests(**kwargs)
  File "/Users/atbrakhi/Documents/igalia/servo/python/wpt/run.py", line 137, in run_tests
    wptrunner.run_tests(**kwargs)
  File "/Users/atbrakhi/Documents/igalia/servo/tests/wpt/web-platform-tests/tools/wptrunner/wptrunner/wptrunner.py", line 455, in run_tests
    iter_success = run_test_iteration(test_status, test_loader, test_source_kwargs,
  File "/Users/atbrakhi/Documents/igalia/servo/tests/wpt/web-platform-tests/tools/wptrunner/wptrunner/wptrunner.py", line 268, in run_test_iteration
    manager_group.run(tests_to_run)
  File "/Users/atbrakhi/Documents/igalia/servo/tests/wpt/web-platform-tests/tools/wptrunner/wptrunner/testrunner.py", line 949, in run
    manager = TestRunnerManager(self.suite_name,
  File "/Users/atbrakhi/Documents/igalia/servo/tests/wpt/web-platform-tests/tools/wptrunner/wptrunner/testrunner.py", line 328, in __init__
    self.command_queue = mp.Queue()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 103, in Queue
    return Queue(maxsize, ctx=self.get_context())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/queues.py", line 42, in __init__
    self._reader, self._writer = connection.Pipe(duplex=False)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/connection.py", line 527, in Pipe
    fd1, fd2 = os.pipe()

[sagudev]

Again Too many open files? Can you run ./mach test-wpt --log-raw /tmp/servo.log so we can get full log to see which test is causing fail.

[atbrakhi]

Again Too many open files? Can you run ./mach test-wpt --log-raw /tmp/servo.log so we can get full log to see which test is causing fail.

@sagudev https://gist.github.com/atbrakhi/050850079b0a39bbb571b69c8673b5b7

[sagudev]

Again Too many open files? Can you run ./mach test-wpt --log-raw /tmp/servo.log so we can get full log to see which test is causing fail.

@sagudev https://gist.github.com/atbrakhi/050850079b0a39bbb571b69c8673b5b7

Did you also get error when running this command? Because I cannot find any test that would be even run?

[atbrakhi]

Again Too many open files? Can you run ./mach test-wpt --log-raw /tmp/servo.log so we can get full log to see which test is causing fail.

@sagudev https://gist.github.com/atbrakhi/050850079b0a39bbb571b69c8673b5b7

Did you also get error when running this command? Because I cannot find any test that would be even run?

yes, I did.

Running 34226 tests in web-platform-tests

  [3826/34226] No tests running.
Error running mach:

    ['test-wpt', '--log-raw', '/tmp/servo.log']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

OSError: [Errno 24] Too many open files

  File "/Users/atbrakhi/Documents/igalia/servo/python/servo/testing_commands.py", line 363, in test_wpt
    ret = self.run_test_list_or_dispatch(kwargs["test_list"], "wpt", self._test_wpt, **kwargs)
  File "/Users/atbrakhi/Documents/igalia/servo/python/servo/testing_commands.py", line 392, in run_test_list_or_dispatch
    return correct_function(**kwargs)
  File "/Users/atbrakhi/Documents/igalia/servo/python/servo/testing_commands.py", line 387, in _test_wpt
    return wpt.run.run_tests(**kwargs)
  File "/Users/atbrakhi/Documents/igalia/servo/python/wpt/run.py", line 137, in run_tests
    wptrunner.run_tests(**kwargs)
  File "/Users/atbrakhi/Documents/igalia/servo/tests/wpt/web-platform-tests/tools/wptrunner/wptrunner/wptrunner.py", line 455, in run_tests
    iter_success = run_test_iteration(test_status, test_loader, test_source_kwargs,
  File "/Users/atbrakhi/Documents/igalia/servo/tests/wpt/web-platform-tests/tools/wptrunner/wptrunner/wptrunner.py", line 268, in run_test_iteration
    manager_group.run(tests_to_run)
  File "/Users/atbrakhi/Documents/igalia/servo/tests/wpt/web-platform-tests/tools/wptrunner/wptrunner/testrunner.py", line 949, in run
    manager = TestRunnerManager(self.suite_name,
  File "/Users/atbrakhi/Documents/igalia/servo/tests/wpt/web-platform-tests/tools/wptrunner/wptrunner/testrunner.py", line 316, in __init__
    self.child_stop_flag = mp.Event()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 93, in Event
    return Event(ctx=self.get_context())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/synchronize.py", line 325, in __init__
    self._flag = ctx.Semaphore(0)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 83, in Semaphore
    return Semaphore(value, ctx=self.get_context())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/synchronize.py", line 126, in __init__
    SemLock.__init__(self, SEMAPHORE, value, SEM_VALUE_MAX, ctx=ctx)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(

[sagudev]

Hm, the weirdest part is that it s coming from python not servo. Maybe adding --processes 1 to limit concurrent jobs could help.

[mrobinson]

@atbrakhi does running with --processes 1 result in the same error?

[atbrakhi]

@atbrakhi does running with --processes 1 result in the same error?

no. I don't get any error with --processes 1

[sagudev]

We really need to change IPC to start using less sockets. #23905 again.