Skip to content

Document multiprocessing deadlock with fork + OpenMP and add runtime warning#279

Merged
letmaik merged 5 commits intomainfrom
copilot/fix-deadlock-multiprocessing
Jan 28, 2026
Merged

Document multiprocessing deadlock with fork + OpenMP and add runtime warning#279
letmaik merged 5 commits intomainfrom
copilot/fix-deadlock-multiprocessing

Conversation

Copy link
Contributor

Copilot AI commented Jan 27, 2026

Problem

rawpy deadlocks on the second call in child processes created via multiprocessing on Linux. This occurs because OpenMP's thread pool mutexes remain locked after fork(), causing subsequent rawpy calls to hang:

import multiprocessing as mp
import rawpy

def child_process():
    with rawpy.imread("image.nef") as raw:
        raw.postprocess()  # Works
    with rawpy.imread("image.nef") as raw:
        raw.postprocess()  # Deadlocks

mp.Process(target=child_process).start()  # Hangs indefinitely

This is a known OpenMP + fork() limitation, not a rawpy bug.

Changes

  • README.md: Added FAQ entry explaining the issue and solution with code example
  • rawpy/init.py: Added runtime detection that warns when rawpy is used in forked child processes with OpenMP enabled
  • test/test_multiprocessing.py: Added test verifying spawn method works correctly

Solution

Use spawn or forkserver start method instead of fork:

import multiprocessing as mp
import rawpy

if __name__ == '__main__':
    mp.set_start_method('spawn')  # Must be first mp call
    
    with mp.Pool(processes=4) as pool:
        results = pool.map(process_raw, ['img1.nef', 'img2.nef'])

The spawn method creates fresh Python processes, avoiding the OpenMP thread state issue entirely.

Original prompt

This section details on the original issue you should resolve

<issue_title>Deadlock when using fork in multiprocessing in Linux</issue_title>
<issue_description>Hello, I am experiencing an issue where I need to run these

with rawpy.imread(RAW_FILE) as raw:
        rgb = raw.postprocess()

in separate child process, but when I do that it deadlocks and I have to terminate the process forcefully

import rawpy
import multiprocessing
import time

RAW_FILE = "test.arw"  # put any RAW file here


def load_raw():
    print("reading")
    with rawpy.imread(RAW_FILE) as raw:
        rgb = raw.postprocess()
    print("finished")


def child_process():
    # Call inside child
    load_raw()
    # Second call
    load_raw()  # deadlock


if __name__ == "__main__":
    print("Start")
    load_raw()  # works

    # Start child process
    p = multiprocessing.Process(target=child_process)
    p.start()

    # Let it run for 10 seconds
    for _ in range(10):
        if not p.is_alive():
            break
        print("Child is still running")
        time.sleep(2)

    p.terminate()
    p.join()
    print("end")

When you run the above code, you will see this - not finishing ever in the forked process:

Start
reading
finished
Child is still running
reading
Child is still running
end

Which I read it like - when it goes to the child the function stops, so my question is - does this issue refer to libraw itself or can be fixed in rawpy?
</issue_description>

<agent_instructions>try to reproduce this issue from the rawpy pypi release, since building from source is not trivial. after you reproduced it, search online if we use the right libraw library variant (there are two). determine if the issue is in rawpy or libraw</agent_instructions>

Comments on the Issue (you are @copilot in this section)


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 3 commits January 27, 2026 23:13
Co-authored-by: letmaik <530988+letmaik@users.noreply.github.com>
Co-authored-by: letmaik <530988+letmaik@users.noreply.github.com>
…an up test

Co-authored-by: letmaik <530988+letmaik@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix deadlock issue in multiprocessing with rawpy Document multiprocessing deadlock with fork + OpenMP and add runtime warning Jan 27, 2026
Copilot AI requested a review from letmaik January 27, 2026 23:20
@letmaik letmaik marked this pull request as ready for review January 27, 2026 23:49
@letmaik letmaik merged commit 005a54e into main Jan 28, 2026
110 checks passed
@letmaik letmaik deleted the copilot/fix-deadlock-multiprocessing branch January 28, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deadlock when using fork in multiprocessing in Linux

2 participants