ValueError: need at most 63 handles, got a sequence of length 65

[alexandrul]

Black runs just fine with a single file or a folder without subfolders.
However, when specifying a folder with subfolders I'm getting the errors below.

Since I don't have another server with the same Windows version, I can't exclude a Windows installation issue, even if other python scripts and tools are working fine.

Python 3.7

Exception in thread QueueManagerThread:
Traceback (most recent call last):
  File "c:\python37\lib\threading.py", line 917, in _bootstrap_inner
    self.run()
  File "c:\python37\lib\threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "c:\python37\lib\concurrent\futures\process.py", line 354, in _queue_management_worker
    ready = wait(readers + worker_sentinels)
  File "c:\python37\lib\multiprocessing\connection.py", line 868, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "c:\python37\lib\multiprocessing\connection.py", line 800, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 65

Python 3.6

Exception in thread Thread-1:
Traceback (most recent call last):
  File "c:\python36\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "c:\python36\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "c:\python36\lib\concurrent\futures\process.py", line 270, in _queue_management_worker
    ready = wait([reader] + sentinels)
  File "c:\python36\lib\multiprocessing\connection.py", line 859, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "c:\python36\lib\multiprocessing\connection.py", line 791, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 64

Operating system: Windows Server 2012 R2 (x64)
Python version: 3.6.6, 3.7.0, 3.7.1rc2 (x64)
Black version: 18.6b4, 18.9b0
Does also happen on master: Yes

[alexandrul]

While running the tests I'm getting into a similar behavior: it seems to freeze after printing some differences from tests\data\string_quotes.py

For the record, there are 65 python processes with --multiprocessing-fork in their command line.

[gwdekker]

A colleague of mine did some digging, and found that this seems to be related to an old open python issue here. Sounds like Black limiting itself to max 60 threads on Windows could be a solution? Could not find any option to set this from the user side when using black, any advice is appreciated.

[zsol]

Sounds easy enough, anyone willing to put up a pr so we can test it out?

[rbtcollins]

Steve Dower has offered to review patches to Python itself to fix this; I'd be happy to argue for backporting them existing releases, if someone would like to fix this at source, rather than band-aiding it.

(edited, it was late is my only excuse). In short: multiprocessing is using a thread per process to track the processes, which is quite wasteful on both unix and windows where nonblocking primitives exist, but nevertheless, thats what it is doing. black would need to limit itself to 60 processes to avoid the limit in question as a result; which is probably fine - one would need a hecka lot of code to run out of patience at that concurrency, at least for now.

[Jasha10]

Still running into the same problem:

(myenv) C:\Users\Jasha\git_repos\myenv_tick_scraper>black.exe .
Exception in thread QueueManagerThread:
Traceback (most recent call last):
  File "C:\Users\Jasha\.conda\envs\myenv\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "C:\Users\Jasha\.conda\envs\myenv\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Jasha\.conda\envs\myenv\lib\concurrent\futures\process.py", line 362, in _queue_management_worker
    ready = mp.connection.wait(readers + worker_sentinels)
  File "C:\Users\Jasha\.conda\envs\myenv\lib\multiprocessing\connection.py", line 878, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "C:\Users\Jasha\.conda\envs\myenv\lib\multiprocessing\connection.py", line 810, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 63

This is black version 19.10b0 and python 3.8, installed using Anaconda.

[mitoihs]

I'm also still encountering this issue with black==19.10b0 and Python 3.7.4 on Windows machine.

>python -m black --check --fast .
Exception in thread QueueManagerThread:
Traceback (most recent call last):
  File "C:\Program Files\Python37\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Program Files\Python37\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Python37\lib\concurrent\futures\process.py", line 361, in _queue_management_worker
    ready = wait(readers + worker_sentinels)
  File "C:\Program Files\Python37\lib\multiprocessing\connection.py", line 868, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "C:\Program Files\Python37\lib\multiprocessing\connection.py", line 800, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 63

[kevinkleinQC]

I'm encountering this issue with black 19.10b0 and python 3.8.3 on Windows Server 2012.

$ black dbcheck
Exception in thread QueueManagerThread:
Traceback (most recent call last):
  File "D:\Personal\Kevin_Klein\envs\dbcheck\lib\threading.py", line 932, in _bootstrap_inner
    self.run()
  File "D:\Personal\Kevin_Klein\envs\dbcheck\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "D:\Personal\Kevin_Klein\envs\dbcheck\lib\concurrent\futures\process.py", line 362, in _queue_management_worker
    ready = mp.connection.wait(readers + worker_sentinels)
  File "D:\Personal\Kevin_Klein\envs\dbcheck\lib\multiprocessing\connection.py", line 879, in wait
    ready_handles = _exhaustive_wait(waithandle_to_obj.keys(), timeout)
  File "D:\Personal\Kevin_Klein\envs\dbcheck\lib\multiprocessing\connection.py", line 811, in _exhaustive_wait
    res = _winapi.WaitForMultipleObjects(L, False, timeout)
ValueError: need at most 63 handles, got a sequence of length 63