-
Notifications
You must be signed in to change notification settings - Fork 22.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
torch.compile
+ torch.no_grad
not working for Mask R-CNN
#97340
Comments
This looks like a PT2 issue, so it may be better to post it (or cross-post it) as a PyTorch issue. |
The code will seg fault if device is "cpu":
Seg fault happens to maskrcnn, retinanet, fasterrcnn and also plain resnet50, but does not seg fault in debug mode:
Other notes:
|
We should fix whatever is causing it to choke under standard configuration, but I'm adding dynamic shapes label as I don't expect MaskRCNN to work end to end without dynamic shapes. Note that maskrcnn is in torchbench but it is currently disabled as not working. EDIT: When I run the working thing in a loop you hit
See also #93457 |
@ezyang I'm trying to figure out why but FYI |
Repro is no longer failing. Closing as completed. |
🐛 Describe the bug
Greetings,
I have been trying to get
torch.compile
to work with a Mask R-CNN model, but have not been able to do so in combination with gradient disabling. An error also occurs if I usetorch.inference_mode
instead oftorch.no_grad
. I've seen it mentioned thattorch.inference_mode
does not work withtorch.compile
(here), but not for gradient disabling in general. The following snippet should reproduce the error:The output:
I've tried the same test using a different model (
torchvision.models.resnet50
), and inthis case the error does not happen. So it could perhaps be specific to the Mask R-CNN
model.
Versions
cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh
The text was updated successfully, but these errors were encountered: