-
-
Notifications
You must be signed in to change notification settings - Fork 6.4k
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FastAPI gets terminated when child multiprocessing process terminated #1487
Comments
Hi @jongwon-yi, |
This behavior doesn't related to request headers or payloads. You can simply reproduce this by
|
@Kludex May I work on this? |
@victorphoenix3 I'm not in charge of anything hahaha If I were you, I'd wait for someone else to confirm the bug (you can confirm by yourself), then if it's really a problem, you can work on it. There are no PRs related to this issue. :) |
@tiangolo @Kludex I did not find this issue on my system. Terminating the child process did not terminate the parent. Can you please re-confirm?
|
Oh, that's amazing, thanks a lot for taking the time to debug and try to reproduce it @victorphoenix3 ! 👏 🙇 That helps a lot! 🍰 I tried it locally and wasn't able to reproduce it either. @jongwon-yi please check with @victorphoenix3's example. |
@victorphoenix3 I tried your code and there is no issue. (Because the subprocess already terminated..?) But after I adding "sleep 30 seconds", and the issue comes. |
With the sleep I was able to reproduce it as well. jfyk |
@Kludex can you tell me what am doing wrong? still haven't been able to reproduce it.
and it terminates without shutting fastapi
|
You're doing the same thing as us, but it works just fine for you. I'll paste here my configs and python packages/version later. |
Hi, I have discovered this situation and came to the next conclusions:
The second and third conclusions is not true, the real problem was founded and described below. But it still possible to solve this problem (without changing FastAPI or uvicorn) - you can change @app.post('/task/run')
def task_run():
multiprocessing.set_start_method('spawn')
proc = multiprocessing.Process(
target=task,
args=(10,))
proc.start()
return proc.pid It's work for me (python3.7, macOS 10.15.5) |
When I tried @Mixser's solution, the second time
|
Hi, @johnthagen. The |
To avoid import multiprocessing
...
app = FastAPI()
@app.on_event("startup")
def startup_event() -> None:
multiprocessing.set_start_method("spawn") |
@johnthagen Your comment is not clear - please clarify, did it help? |
@Mixser I edited my original message above. |
Ran into this too and although @johnthagen's fix worked, it made my child processes swallow their logs. I tried to find a solution that didn't involve OS signals at all and came up with this:
from asyncio import gather, get_running_loop, run
from multiprocessing import JoinableQueue, Process
from my_code.some_infinite_executor import SomeInfiniteExecutor
STOP_FLAG = 'STOP'
def start_process(flag_queue: JoinableQueue) -> None:
some_process = Process(
target=start_process_target,
args=(flag_queue, )
)
some_process.start()
def start_process_target(flag_queue: JoinableQueue) -> None:
run(execute(flag_queue))
async def execute(flag_queue: JoinableQueue) -> None:
executor = SomeInfiniteExecutor()
await gather(
executor.execute(),
wait_for_stop(executor, flag_queue)
)
async def wait_for_stop(executor: SomeInfiniteExecutor, flag_queue: JoinableQueue) -> None:
event_loop = get_running_loop()
await event_loop.run_in_executor(None, stop_queue.get)
await executor.stop()
stop_queue.task_done()
def stop_process(some_process: Process, flag_queue: JoinableQueue) -> None:
flag_queue.put(STOP_FLAG}
some_process.join() The trick is to share a Queue between the parent process and its child. The child has to wait for a stop flag in the queue from the parent and stop itself gracefully (like by cancelling its tasks, etc.), the parent just has to join the child process after sending this flag. API functions can simply call |
I tried to implement multiprocessing in fastAPI using the fork and the spawn method (as suggested in #1487 (comment)). While the usage of the fork method makes the API shut down after one call, the spawn method makes the API extremely slow. Does anybody else experience this issue and have a solution for that? Thanks a lot! |
Hi, I've found a new approach how to avoiding this behavior. But at first, let's figure out what's going on here. We have a master process ( Next, we are creating a new process in our HTTP handler -- this new process will inherit this opened socket (opened file descriptor), and we are waiting for signals from this socket. When we are sending a signal to the child, it goes to this opened socket. But our parent process listens to this socket too, so it receives a signal to terminate and shut down the application. How to avoid -- we need to return the default behavior of signal handlers for child process and don't use the inherited fd from the parent; We can achieve this by calling There is a PoC from time import sleep
from fastapi import FastAPI
import os, signal
import psutil
import multiprocessing
app = FastAPI()
def task(pid: int):
signal.set_wakeup_fd(-1)
signal.signal(signal.SIGTERM, signal.SIG_DFL)
signal.signal(signal.SIGINT, signal.SIG_DFL)
print(f"{pid} {os.getpid()}")
while True:
sleep(1)
@app.get('/task/run')
async def task_run():
pid = os.getpid()
proc = multiprocessing.Process(
target=task,
args=(pid, ))
proc.start()
return proc.pid
@app.get('/task/abort')
def task_abort(pid: int):
proc = psutil.Process(pid)
proc.terminate()
return 0
|
@tiangolo I think you can close this, because this is not related to the FastApi or uvicorn -- this is a specific behaviour of signal handling in asyncio. |
I added |
Thanks for the discussion everyone! I think @Mixser tips would do it, right? If that solves the problem you can close the issue @jongwon-yi. ☕ |
Assuming the original need was handled, this will be automatically closed now. But feel free to add more comments or create new issues or PRs. |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Describe the bug
Make a multiprocessing Process and start it.
Right after terminate the process, fastapi itself(parent) terminated.
To Reproduce
Start command: /usr/local/bin/uvicorn worker.stts_api:app --host 127.0.0.1 --port 8445
Expected behavior
Parent process should not be terminated after child terminated.
Environment
Additional context
I tried same code with Flask with gunicorn, it never terminated.
The text was updated successfully, but these errors were encountered: