Pyomo deadlock

[simonhackler]

Summary

When running pyomo with solve we sometimes hit this error:

raise DeveloperError("Deadlock closing capture_output")\npyomo.common.errors.DeveloperError: Internal Pyomo implementation error:\n 'Deadlock closing capture_output'\n Please report this to the Pyomo Developers."

    solver.options["log_file"] = log_path
    solver.options["output_flag"] = True
    solver.options["log_to_console"] = False
    result = solver.solve(model, tee=False)

Steps to reproduce the issue

I tried but can't reproduce the error. It happens only sometimes

Error Message

raise DeveloperError("Deadlock closing capture_output")\npyomo.common.errors.DeveloperError: Internal Pyomo implementation error:\n 'Deadlock closing capture_output'\n Please report this to the Pyomo Developers."

Information on your system

Pyomo version: 6.10
Python version: 3.13
Operating system: ubuntu
How Pyomo was installed (PyPI, conda, source): pypi (uv)
Solver (if applicable): highs

Additional information

[jsiirola]

Which solver interface are you using? highs or appsi_highs? Please submit a complete minimal example that reproduces the behavior, even if it only reproduces it intermittently (but please give us an idea as to how often we should expect to see it).

[simonhackler]

We are using highs without appsi: solver = pyo.SolverFactory("highs"), but we had the same issue with appsi high.
I will try to create a reproduction example but it is flakey in our environment and not dependent on the input data.
So far I have not been able to reproduce it.

[jsiirola]

Thanks. The exception is because we are unable to acquire() a threading.Lock. I expect it to be very intermittent, and is likely to be due to how you are orchestrating things. Some questions:

• are you using threading or multiprocessing?
• is this a single solve, or you running some form of analysis loop?
• is this on the console, through an IDE, or in a "remote" environment (either in a container or on a remote server)?
• are you seeing any logged output, and particular a "WARNING: Significant delay observed waiting to join reader threads, possible output stream deadlock"?
• is this happening when the interpreter should otherwise be shutting down (e.g., the end of the script, or the processes was signaled or Ctrl-C'ed)?

One thing you can try is editing pyomo/common/tee.py and applying the following patch:

--- a/pyomo/common/tee.py
+++ b/pyomo/common/tee.py
@@ -345,7 +345,7 @@ class capture_output:
         return FAIL
 
     def __enter__(self):
-        if not capture_output.startup_shutdown.acquire(timeout=_poll_timeout_deadlock):
+        if not capture_output.startup_shutdown.acquire(timeout=2*_poll_timeout_deadlock):
             # This situation *shouldn't* happen.  If it does, it is
             # unlikely that the user can fix it (or even debug it).
             # Instead they should report it back to us.
@@ -365,7 +365,7 @@ class capture_output:
             capture_output.startup_shutdown.release()
 
     def __exit__(self, et, ev, tb):
-        if not capture_output.startup_shutdown.acquire(timeout=_poll_timeout_deadlock):
+        if not capture_output.startup_shutdown.acquire(timeout=2*_poll_timeout_deadlock):
             # See comments & breadcrumbs in __enter__() above.
             raise DeveloperError("Deadlock closing capture_output")
         try:

I doubt that it will make the exception go away completely, but it may move it to the actual root cause.

[jsiirola]

One more: are you seeing a long (100 second) pause before that exception is being raised, or is it happening immediately on solver exit? (You are only supposed to see it when the acquire() times out ... but if you are seeing it immediately, it might be that the interpreter is shutting down).