Emscripten test worker sometimes fails witth "BlockingIOError: write could not complete without blocking"

[tiran]

Bug report

Sometimes test worker processes on wasm32-emscripten build bot fails with BlockingIOError: write could not complete without blocking. Example https://buildbot.python.org/all/#/builders/1044/builds/27/steps/10/logs/stdio

0:37:30 load avg: 4.42 [422/436/3] test_pathlib crashed (Exit code 1)
... 
result=Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node/build/Lib/test/regrtest.py", line 47, in <module>
    _main()
    ^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node/build/Lib/test/regrtest.py", line 43, in _main
    main()
    ^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node/build/Lib/test/libregrtest/main.py", line 754, in main
    Regrtest().main(tests=tests, **kwargs)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node/build/Lib/test/libregrtest/main.py", line 692, in main
    self._main(tests, kwargs)
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node/build/Lib/test/libregrtest/main.py", line 713, in _main
    run_tests_worker(self.ns, self.worker_test_name)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/buildbot/bcannon-wasm/3.x.bcannon-wasm.emscripten-node/build/Lib/test/libregrtest/runtest_mp.py", line 102, in run_tests_worker
    print(json.dumps(result, cls=EncodeTestResult), flush=True)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
BlockingIOError: [Errno 6] write could not complete without blocking

The error seems to occur at the end of a test job when the worker process returns its status to the control process. The control process runs CPython natively (x86_64-pc-linux-gnu in case of the build bot). The worker processes is wasm32-emscripten CPython with NodeJS as runner. I guess that every now and then, the test worker process produces output faster than the control process can consume from the pipe that connects stdout of the worker and read pipe of the control process. The runtime does not like blocking IO and returns EAGAIN.

>>> import sys, errno
>>> sys.platform
'emscripten'
>>> errno.errorcode[6]
'EAGAIN'

[tiran]

The fcntl module is currently not available on Emscripten. I hacked the module manually and was able to get flags from the stdout fd. Emscripten does not report that the fd is set to non-blocking mode.

>>> import os, fcntl
>>> fcntl.fcntl(1, fcntl.F_GETFL)
577
>>> fcntl.fcntl(1, fcntl.F_GETFL) & os.O_NONBLOCK
0

[tiran]

I applied a hack on the buildbot worker host. Buildbot now starts NodeJS with nice -n5. That seems to be good enough to make the tests pass.

[serhiy-storchaka]

Why is it set to non-blocking mode? Can it be set to blocking mode? Can it be replaced with an fd set to blocking mode? Can TextIOWrapper and BufferedWriter be made to handle this internally? Can we use an alternate channel for communication between processes?

[tiran]

It is set to non-blocking mode because different rules apply to browser-like environments. It is not possible to change the property. In fact there is no way to even detect that the pipe fd is in non-blocking mode. I can only deduce that it is in non-blocking mode because tests sometimes fail when the system is under high load and the other side cannot read from the pipe fast enough.

Emscripten tries hard to mimic a POSIX-like OS on top of a JavaScript and WebAssembly engine. Ultimately it is bound by restriction of the runtime environment.

To handle the special case in Python's io module we would first have to detect the special case. That is not possible on Emscripten yet. The fcntl module is not available because I disabled ioctl as it leads to crashes. The fcntl(F_GETFD) syscall does not return O_NONBLOCK and fstat does not work on pipes and standard streams.

We could only use a temporary file for communication. Socket's don't work as expected. IPC and socketpair are not available.