Multiprocessing crashes on macOS when a page makes an HTTP request

[ratanasov]

ALL software version info

Panel 1.3 or 1.2.3
Python 3.10.12
macOS 14.0

Description of expected behavior and the observed behavior

Panel multiprocessing crashes on macOS when a page makes an HTTP request to a third-party service (e.g. OAuth). This seems to be a known issue with (Python) multiprocessing on macOS and handling it probably requires changes in the multiprocessing code.

Complete, minimal, self-contained example code that reproduces the issue

import panel as pn
import requests

def app():
    print("error?")
    requests.get("https://panel.holoviz.org")
    print("ok")

    return "ok"

pn.serve(app, num_procs=2)

Stack traceback and/or browser JavaScript console output

Launching server at http://localhost:63064
Launching server at http://localhost:63064
error?
objc[12572]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[12572]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
child 1 (pid 12572) killed by signal 6, restarting

... the above / below error repeated many times ...

error?
objc[12869]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[12869]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
child 0 (pid 12869) killed by signal 6, restarting
Launching server at http://localhost:63064
error?
objc[12872]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[12872]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
child 1 (pid 12872) killed by signal 6, restarting
Traceback (most recent call last):
  File "/Users/rado/p/hpred/panel_apps/scratch/objc_fork_err.py", line 27, in <module>
    pn.serve(app, num_procs=2)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/hpred/lib/python3.10/site-packages/panel/io/server.py", line 922, in serve
    return get_server(panels, **kwargs)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/hpred/lib/python3.10/site-packages/panel/io/server.py", line 1184, in get_server
    server = Server(apps, port=port, **opts)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/hpred/lib/python3.10/site-packages/panel/io/server.py", line 376, in __init__
    super().__init__(*args, **kwargs)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/hpred/lib/python3.10/site-packages/bokeh/server/server.py", line 453, in __init__
    http_server.start(opts.num_procs)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/hpred/lib/python3.10/site-packages/tornado/tcpserver.py", line 301, in start
    process.fork_processes(num_processes, max_restarts)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/hpred/lib/python3.10/site-packages/tornado/process.py", line 167, in fork_processes
    raise RuntimeError("Too many child restarts, giving up")
RuntimeError: Too many child restarts, giving up
error?
objc[12875]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called.
objc[12875]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.

Workarounds

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES makes the app work, but is not recommended, since it disables an ObjC runtime check which is potentially preventing even nastier bugs (e.g. here).

multiprocessing.set_start_method("spawn") and multiprocessing.set_start_method("forkserver") didn't fix the issue.

[philippjfr]

I don't think num_procs can work when running in pn.serve unfortunately. There may be ways to making this possible but I think a few things happen that mean that forking the process after panel is fully imported isn't possible.

[philippjfr]

Sorry I guess I didn't read your issue in sufficient detail. I suspect there's nothing we can really do about this in Panel, the fix would likely have to be either in Tornado or in Bokeh.

[ratanasov]

I don't think num_procs can work when running in pn.serve unfortunately. There may be ways to making this possible but I think a few things happen that mean that forking the process after panel is fully imported isn't possible.

Does this mean that multiprocessing is not officially supported with panel.serve (and that's why it's also not documented)?

For a simple demo app it seemed like the basics work ok (and we stumbled upon this issue, when we tried it in our more realistic app).

[philippjfr]

It's a good question, I'd say it's definitely in a gray area. Using --num-procs from the commandline is definitely supported but I don't have a good answer for pn.serve, especially given the issues you're reporting above. Can you reproduce an issue when you do:

import panel as pn
import requests

requests.get("https://panel.holoviz.org")

panel serve app.py --num-procs 2

[ratanasov]

Yes, the issue is still reproducible, as soon as the app gets its first HTTP request (or immediately with --show). Removing the (unnecessary) panel import also doesn't help.

[ratanasov]

I suspect there's nothing we can really do about this in Panel, the fix would likely have to be either in Tornado or in Bokeh.

After some digging it seems that the root cause is in Tornado, which uses os.fork, instead of the multiprocessing module, which switched the default on macOS to the spawn start method exactly because of this issue.

I'll try to file an issue with Tornado later this week.

[ratanasov]

After some digging it seems that the root cause is in Tornado, which uses os.fork, instead of the multiprocessing module, which switched the default on macOS to the spawn start method exactly because of this issue.

I'll try to file an issue with Tornado later this week.

@philippjfr , I could not reliably reproduce this with a pure Tornado example and I'm out of time, so I'll have to leave following up with Tornado to you, if you would like to have this fixed upstream.

The following example, always crashes during the first request, when run from Pycharm and never crashes when run from the Terminal. I could not figure out the difference. I already made sure that OBJC_DISABLE_INITIALIZE_FORK_SAFETY is not set in both cases.

import asyncio

import requests
import tornado
from tornado.httpserver import HTTPServer
from tornado.netutil import bind_sockets
from tornado.web import Application
from tornado.web import RequestHandler

class RootHandler(RequestHandler):
    def get(self):
        html = requests.get("https://www.tornadoweb.org/").content
        self.write(html)

app = Application([("/", RootHandler)])

# from here on it's almost a verbatim copy of pattern 2. from https://www.tornadoweb.org/en/stable/httpserver.html
# (except passing the app to HTTPServer and the empty lines)
sockets = bind_sockets(8888)
tornado.process.fork_processes(0)

async def post_fork_main():
    server = HTTPServer(app)
    server.add_sockets(sockets)
    await asyncio.Event().wait()

asyncio.run(post_fork_main())

[cdeil]

I tried with latest MacOS Sonoma 14.4 (23E214) and latest Python etc pp

python                    3.12.2          hdf0ec26_0_cpython    conda-forge
tornado                   6.4             py312he37b823_0    conda-forge
bokeh                     3.4.0                    pypi_0    pypi
panel                     1.4.0rc4                 pypi_0    pypi
jupyter_bokeh             4.0.1                      py_0    bokeh

None of the examples above crash, not from the MacOS terminal or the VSCode or PyCharm one.

@philippjfr - Is there anything that could be improved in Tornado / Bokeh / Panel for multi-processing? Is it still useful to keep this ticket open as reminder?