Skip to content

[util][rpdb] Fix IndexError in legacy post-mortem debugging#61479

Merged
edoakes merged 3 commits into
ray-project:masterfrom
thakoreh:fix/rpdb-frame-index-error
May 13, 2026
Merged

[util][rpdb] Fix IndexError in legacy post-mortem debugging#61479
edoakes merged 3 commits into
ray-project:masterfrom
thakoreh:fix/rpdb-frame-index-error

Conversation

@thakoreh
Copy link
Copy Markdown
Contributor

@thakoreh thakoreh commented Mar 4, 2026

Summary

Fixes #61470

When using RAY_DEBUG=legacy with RAY_DEBUG_POST_MORTEM=1, the _post_mortem() function calls _connect_ray_pdb() which tries to access inspect.getouterframes(...)[2]. However, the frame depth may not always be sufficient, causing IndexError: list index out of range.

Changes

  • Added a fallback in _connect_ray_pdb() to use the last available frame when there aren't enough frames in the call stack
  • The fix: frames[2] if len(frames) > 2 else frames[-1]

Testing

The fix ensures that:

  1. Normal set_trace() paths continue to work (frame depth is predictable)
  2. Post-mortem debugging paths no longer crash with IndexError

Checklist

  • I've signed off every commit (git commit -s) (N/A: single commit)
  • I've updated tests (N/A: defensive fix for edge case)
  • I've linked this PR to the relevant issue

Related Issues

Fixes #61470

@thakoreh thakoreh requested a review from a team as a code owner March 4, 2026 07:08
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request addresses a potential IndexError in the legacy post-mortem debugging functionality. The issue arises when _connect_ray_pdb attempts to access a specific frame index that may not exist if the call stack is not deep enough. The change introduces a check for the number of available frames and falls back to the last available frame if the stack depth is insufficient. The fix is correct and robustly handles the described edge case. I have no further comments.

@ray-gardener ray-gardener Bot added core Issues that should be addressed in Ray Core observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling community-contribution Contributed by the community labels Mar 4, 2026
When using RAY_DEBUG=legacy with RAY_DEBUG_POST_MORTEM=1, the
_post_mortem() function calls _connect_ray_pdb() which tries to
access inspect.getouterframes(...)[2]. However, the frame depth
may not always be sufficient, causing IndexError.

Fix: Add a fallback to use the last available frame when there
aren't enough frames in the call stack.

Fixes ray-project#61470

Signed-off-by: Hiren <hiren.thakore58@gmail.com>
@thakoreh thakoreh force-pushed the fix/rpdb-frame-index-error branch from eceeeeb to c4a3d61 Compare March 5, 2026 16:15
@eval-dev
Copy link
Copy Markdown

eval-dev commented Mar 9, 2026

hi @justinvyu , can you help take a look at this one line fix? Thanks

@github-actions
Copy link
Copy Markdown

This pull request has been automatically marked as stale because it has not had
any activity for 14 days. It will be closed in another 14 days if no further activity occurs.
Thank you for your contributions.

You can always ask for help on our discussion forum or Ray's public slack channel.

If you'd like to keep this open, just leave any comment, and the stale label will be removed.

@github-actions github-actions Bot added the stale The issue is stale. It will be closed within 7 days unless there are further conversation label Mar 24, 2026
@Kunchd Kunchd added train Ray Train Related Issue and removed core Issues that should be addressed in Ray Core labels Mar 24, 2026
@github-actions github-actions Bot added unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it. and removed stale The issue is stale. It will be closed within 7 days unless there are further conversation labels Mar 25, 2026
Copy link
Copy Markdown
Member

@pseudo-rnd-thoughts pseudo-rnd-thoughts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR, LGTM

@pseudo-rnd-thoughts pseudo-rnd-thoughts added the go add ONLY when ready to merge, run all tests label May 12, 2026
@edoakes edoakes merged commit 4315a0c into ray-project:master May 13, 2026
5 of 6 checks passed
am-kinetica pushed a commit to kineticadb/ray that referenced this pull request May 14, 2026
…ect#61479)

## Summary

Fixes ray-project#61470

When using `RAY_DEBUG=legacy` with `RAY_DEBUG_POST_MORTEM=1`, the
`_post_mortem()` function calls `_connect_ray_pdb()` which tries to
access `inspect.getouterframes(...)[2]`. However, the frame depth may
not always be sufficient, causing `IndexError: list index out of range`.

## Changes

- Added a fallback in `_connect_ray_pdb()` to use the last available
frame when there aren't enough frames in the call stack
- The fix: `frames[2] if len(frames) > 2 else frames[-1]`

## Testing

The fix ensures that:
1. Normal set_trace() paths continue to work (frame depth is
predictable)
2. Post-mortem debugging paths no longer crash with IndexError

## Checklist

- [x] I've signed off every commit (git commit -s) (N/A: single commit)
- [x] I've updated tests (N/A: defensive fix for edge case)
- [x] I've linked this PR to the relevant issue

## Related Issues

Fixes ray-project#61470

Signed-off-by: Hiren <hiren.thakore58@gmail.com>
Co-authored-by: Mark Towers <mark.m.towers@gmail.com>
Signed-off-by: anindyam1969 <amukherjee@kinetica.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-contribution Contributed by the community go add ONLY when ready to merge, run all tests observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling train Ray Train Related Issue unstale A PR that has been marked unstale. It will not get marked stale again if this label is on it.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[util][rpdb] Legacy Post Mortem Debugging got IndexError from inspect.getouterframes

5 participants