Skip to content

Conversation

ngimel
Copy link
Collaborator

@ngimel ngimel commented Mar 15, 2024

Initializing profiler while holding gil can lead to deadlocks, as it makes some presumably synchronizing cuda calls

@ngimel ngimel requested review from albanD and soulitzer as code owners March 15, 2024 03:29
@pytorch-bot
Copy link

pytorch-bot bot commented Mar 15, 2024

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/121949

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit 9c0b405 with merge base efbeefb (image):

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@janeyx99 janeyx99 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Mar 15, 2024
@ngimel
Copy link
Collaborator Author

ngimel commented Mar 15, 2024

cc @aaronenyeshi. Even if some cupti calls that kineto makes are not thread safe, they should be guarded by a lock in kineto, not by gil.

@aaronenyeshi aaronenyeshi added release notes: profiler release notes category topic: not user facing topic category ciflow/trunk Trigger trunk jobs on your pull request labels Mar 15, 2024
Copy link
Member

@aaronenyeshi aaronenyeshi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this diff, I will import it to see if any internal CI depends on the gil.

@facebook-github-bot
Copy link
Contributor

@aaronenyeshi has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@pytorchbot merge -f 'Landed internally'

(Initiating merge automatically since Phabricator Diff has merged, using force because this PR might not pass merge_rules.json but landed internally)

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@github-actions github-actions bot deleted the ngimel/prof_gil branch April 19, 2024 01:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged open source release notes: profiler release notes category topic: not user facing topic category triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants