Labtech inconsistently hangs on Windows when using `"spawn"` backend via notebook

[nathanjmcdougall]

I've tried using an if __name__ == "__main__": block.

It starts the tqdm progress bar and displays no errors, but just hangs:

"thread" works fine.

The frustrating thing is that it's consistent. I haven't yet managed to reproduce it consistently. This sometimes fails (on both my and a colleague's machine):

import labtech


@labtech.task
class ExampleTask:
    """Task to calculate fragility for a given PGA value."""

    x: float

    def run(self) -> float:
        """Calculate fragility for the given parameters."""
        return 0


tasks = [ExampleTask(x=0.2)]

# Run the tasks in parallel
if __name__ == "__main__":
    results = labtech.Lab(storage=None, runner_backend="spawn").run_tasks(tasks)

[ben-denham]

Thanks for the report, I've tried this myself with labtech 0.7.3 on Windows 10 with Python 3.13 and I can't replicate either by:

• Running as a script (uv run main.py)
• Running from a Jupyter cell (with the task class defined in and imported from a separate module)

Could you please provide a few more details to help isolate the issue:

1. Windows version
2. Python version
3. How are you running Labtech when this happens? E.g. with uv run main.py or from a Jupyter cell in VS Code?
   • If running from Jupyter or an interactive Python session, are you defining your task types in a separate module? As per: https://ben-denham.github.io/labtech/cookbook/#why-do-i-see-the-following-error-attributeerror-cant-get-attribute-your_task_class-on-module-__main__-built-in
4. If you interrupt Python while it is hanging (i.e. Ctrl-C or Interrupt Kernel), can you please provide a stack trace so that I can see which line is executing when it starts to hang.
   • Alternatively, you could get the process ID from task manager and run: py-spy dump --pid 12345 to get the call stack (https://github.com/benfred/py-spy?tab=readme-ov-file#dump)

A few other notes:

• I've noticed stdout/stderr aren't being captured from the spawned subprocesses in 0.7.3. I'll need to investigate this further, but it might be masking output printed from subprocesses.
• I'm planning to introduce a runner that is more similar to that in 0.6.1by using concurrent.futures to manage a pool of pre-spawned subprocesses instead of spawning a new subprocess for each task. I will likely make that the default runner for at least "spawn", and I expect that may resolve many of these issues. Still, it will be useful to isolate what is going on here.

[ben-denham]

Another troubleshooting idea: Can you replicate this issue with max_workers=1 or other small numbers of workers?

[ben-denham]

This issue is likely be resolved by #55 because it switches back to using a concurrent.futures pool of processes for the spawn runner backend (which also avoids the overhead of spawning a separate subprocess for each task, without any increased memory sharing in the case of the spawn start method).

It would still be interesting to understand more about what caused this issue in version 0.7.3 for future reference.

[ben-denham]

Process pooling in 0.8.0 appears to have resolved this intermittent issue.