-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable multi-threaded execution of in MonteCarloSimulation launched from Python #17363
Comments
I liked the original intent of the thread - to explicitly fail if attempting to mix C++ multithreading with Python. For concurrent execution in Python involving CPython API extensions, I think I will probably be unable to dedicate any real time to this in the near future myself. Am not entirely sure how to delegate. |
Yes, I tend to agree with this.
For my own part, I don't see any particular urgency here. I believe the resolution would be to create an illustration / example / tutorial showing how to set up concurrent simulations using |
Two thoughts:
|
(2) Sure. If we add a new function This should catch >90% of the user errors. (I'm not confident that we can plug all of the callback holes. The Monte Carlo documentation should directly explain to Python users that they cannot have any Python code in their Diagram in any way.) The main challenge will be with the Another approach would be to add a "bool use_serial_callbacks" flag to Monte Carlo itself, so that it internally governs the make vs output callbacks -- it might be able to keep the thread schedule more full if we give it direct control. It might also end up being useful for C++ code that can't easily make a thread safe output calculator. (1) I think pickling the functor that creates the diagram, instead of the diagram itself, should be a fairly easy and plausible work-around for many cases. It also makes it possible to scale the concurrency beyond a single machine. The functor can have bound arguments (functools.partial) for any extra configuration data; it doesn't always need to be a zero-argument function. It will scale in case the user needs to add a Python System (e.g., looping in their controller). It is a minor hurdle to carve out the factory functor to pickle it, but it has the upside that you don't hit a brick wall anytime you need to go beyond pydrake's C++-bound code. |
(2) For the case of |
FYI we have this comment now: drake/bindings/pydrake/systems/analysis_py.cc Lines 419 to 422 in fda9e48
Adding the |
This is a follow-up to the conversation started in this Slack thread: https://drakedevelopers.slack.com/archives/C2CHRT98E/p1653658311307059
The monte-carlo simulation tools have become a useful model for multi-process executions in Drake. We currently disallow multi-threaded execution for MonteCarloSimulation at the binding layer:
drake/bindings/pydrake/systems/analysis_py.cc
Lines 280 to 282 in 4cfbb2e
According to @calderpg-tri ... running the current python tests for monte-carlo with multi-processing enabled will cause the system to hang. My current understanding of the problem is that the c++ code needs deal with the Python GIL. (see, for instance, pybind/pybind11#1087). The big question is whether we can handle this at the level of the bindings, or whether every potential c++ caller must deal with managing the lock.
Enabling this for the monte-carlo tools will pave the way for many other relevant use cases (such as trajectory optimization and DrakeGym) where users would benefit substantially from multi-process executions but also have the ability to pass Python callbacks into the C++ execution.
+@EricCousineau-TRI as pydrake owner.
The text was updated successfully, but these errors were encountered: