[Debugger] Adding debupgy as the ray debugger #42311
Conversation
|
The files do matches the diff based on my review, please verify end to end working before merging. |
brycehuang30
requested review from
richardliaw,
ericl and
edoakes
as code owners
brycehuang30
changed the title
|
I manually tested that the task can be paused and it awaits connection from debugger client. Tested tasks running on worker nodes and head node. I also tested post-mortem debugging is working Manually verified it's e2e working |
python/setup.py
Outdated
| @@ -342,6 +342,7 @@ def get_packages(self): | |||
| "aiosignal", | |||
| "frozenlist", | |||
| "requests", | |||
| "debugpy", | |||
There was a problem hiding this comment.
We probably shouldn't install this by default; can we instead do a soft-import and raise a warning/exception if this is not installed?
There was a problem hiding this comment.
sure! will do
|
I updated the PR to soft import Belowing are the manual tests after adding the soft import for debugpy breakpoint with Post mortem without post mortem with |
python/ray/util/debugpy.py
Outdated
| log.error( | ||
| f"Module '{module}' cannot be loaded. " | ||
| f"Ray Debugger will not work without '{module}'. " | ||
| f"Install this module using 'pip install {module}' " | ||
| ) |
There was a problem hiding this comment.
I tried throwing an exception when import error occurred, but it doesn't work well. The exception will be captured by Ray and drop the program into post-mortem mode, which will try to import debugpy again and throw another import error again... And end up with multiple levels of exception all saying there is an import error. So I choose to just log the error message and skip debugger logic if the module is not available.
python/ray/util/debugpy.py
Outdated
| def _try_import(module): | ||
| try: | ||
| return importlib.import_module(module) | ||
| except ModuleNotFoundError: | ||
| log.error( | ||
| f"Module '{module}' cannot be loaded. " | ||
| f"Ray Debugger will not work without '{module}'. " | ||
| f"Install this module using 'pip install {module}' " |
There was a problem hiding this comment.
make this specific, look for debugpy >= 1.8.0, and then raise warning if not found (and suggest to user to install debugpy==1.8.0)
python/ray/util/debugpy.py
Outdated
| except ModuleNotFoundError: | ||
| log.error( | ||
| f"Module '{module}' cannot be loaded. " | ||
| f"Ray Debugger will not work without '{module}'. " |
There was a problem hiding this comment.
Ray Debugpy Debugger
- enabled debugpy as the ray debugger for breakpoint and post_mortem debugging - added flag RAY_DEBUG=1 to enable debugpy. If RAY_DEBUG is not set and RAY_PDB is set, then rpdb will be used. - used state api to save worker debugging port.
- enabled debugpy as the ray debugger for breakpoint and post_mortem debugging - added flag RAY_DEBUG=1 to enable debugpy. If RAY_DEBUG is not set and RAY_PDB is set, then rpdb will be used. - used state api to save worker debugging port.
- enabled debugpy as the ray debugger for breakpoint and post_mortem debugging - added flag RAY_DEBUG=1 to enable debugpy. If RAY_DEBUG is not set and RAY_PDB is set, then rpdb will be used. - used state api to save worker debugging port.
Why are these changes needed?
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.I manually tested that the task can be paused and it awaits connection from debugger client. Tested tasks running on worker nodes and head node. I also tested post-mortem debugging is working.