Pytorch2 compile (#1505) #1516

N-Friederich · 2023-03-20T18:36:23Z

Extension accordingly #1505

Add pytorch.compile(...) to Yolo.init(...) incl. version check
Use Compile with default options. More information here

@AyushExel @glenn-jocher Maybe there is a better place. We'll have to discuss that. Currently, I see a significant speedup of 1.5 on Apple M2 Pro with MPS. Sorry, I am still in the process of understanding the code and the structure of the code modules.

`🤖 Generated by Copilot at e984a48`

Summary

🚀🛠️📦

This pull request adds an option to compile the YOLO model with the experimental PyTorch 2.0 feature, which improves the performance but prevents exporting. It also updates the tests, benchmarks, and exporter to disable this feature by default, to ensure compatibility and stability.

We're sailing with the YOLO crew, we've got a job to do
We need to export our models fast, but PyTorch 2 won't do
So we'll pass a flag to skip the compile, and save ourselves some trouble
Heave away, me hearties, heave away on the double

Walkthrough

Add compile_model argument to YOLO constructor and check PyTorch version and compile model with experimental feature if True (link, link, link)
Raise ValueError in YOLO.export method if compile_model is True (link)
Pass compile_model=False to YOLO constructor in export function in ultralytics/yolo/engine/exporter.py and benchmark function in ultralytics/yolo/utils/benchmarks.py (link, link, link)
Pass compile_model argument to YOLO constructor based on mode argument in entrypoint function in ultralytics/yolo/cfg/__init__.py (link)
Modify test functions for exporting model to different formats to pass compile_model=False to YOLO constructor in tests/test_python.py (link, link, link, link)
Import check_version function from ultralytics/yolo/utils/checks.py to ultralytics/yolo/engine/model.py and ultralytics/yolo/engine/trainer.py (link, link)
Import torch to ultralytics/yolo/engine/model.py and remove unused import from ultralytics/yolo/cfg/__init__.py (link, link)

🛠️ PR Summary

_{Made with ❤️ by Ultralytics Actions}

📊 Key Changes

Added compile_model flag to allow the compilation of PyTorch models with new Torch 2.0 features.
Adjusted model property in the trainer.py to use the compiled model if the compile_model flag is true and the PyTorch version is 2.0 or above.
Updated various parts of the codebase to support this new compilation feature including autobackend.py, predictor.py, and default.yaml.

🎯 Purpose & Impact

The goal of these changes is to utilize the performance benefits provided by model compilation in PyTorch 2.0, potentially reducing inference time and improving efficiency.
Users working with the updated Ultralytics repository can now expect faster model execution if using PyTorch 2.0, given they enable the new compile_model option.

🌟 Summary

Leverage PyTorch 2.0's model compilation for enhanced performance in Ultralytics' AI models. 🚀

for more information, see https://pre-commit.ci

glenn-jocher · 2023-03-21T11:15:50Z

@N-Friederich got it. It looks like most CI is passing except for exports.

glenn-jocher · 2023-03-21T11:16:24Z

Also note Python 1.7 is not supported by torch 2.0, which is why 1.7 CI is passing.

N-Friederich · 2023-03-21T11:37:26Z

Also note Python 1.7 is not supported by torch 2.0, which is why 1.7 CI is passing.

@glenn-jocher Good point. I'll take a look.

N-Friederich · 2023-03-21T11:39:05Z

Also note Python 1.7 is not supported by torch 2.0, which is why 1.7 CI is passing.

@glenn-jocher Ok, but then this is a pytorch recommendation. Maybe we need to build a test case explicitly for Pytorch 2?

glenn-jocher · 2023-03-21T12:22:20Z

No don't worry, CI is fine, since 1.13.1 is applied automatically for Python 1.7 which will end of life in a few months, all over CI's use torch 2.0 and are passing in main branch now.

The problem is the compiled models don't seem to export correctly. Maybe we should only compile when an inference command is run, i.e. predict, train, val?

N-Friederich · 2023-03-21T12:40:51Z

@glenn-jocher I think a torch._dynamo.reset() do the job in the export method. I will test this later.

N-Friederich · 2023-03-21T14:23:03Z

@glenn-jocher I think I misunderstood the method. I am currently considering an alternative.

N-Friederich · 2023-03-21T16:52:20Z

@glenn-jocher I think the problem is difficult to solve. Because what if you call train(...) in python and then export(...) on the same model. if this is compiled, you can no longer export it.

N-Friederich · 2023-03-21T16:55:04Z

@glenn-jocher I'll look at it again in detail tomorrow.

glenn-jocher · 2023-03-22T11:46:08Z

@glenn-jocher I think the problem is difficult to solve. Because what if you call train(...) in python and then export(...) on the same model. if this is compiled, you can no longer export it.

Yes might be a little complicated, but I think the first step would be to only compile on inference methods (predict, val, train), and then check in export if compiled: warning with explanation and exit.

glenn-jocher · 2023-03-22T11:51:34Z

@N-Friederich I reviewed https://pytorch.org/tutorials/intermediate/torch_compile_tutorial.html, it seems that compile ops are best used for long term actions like training and val on large datasets. Their demo code for example shows that it takes several seconds to compile a model, so it may not be suitable for simply running inference on an image.

Maybe we should opt-in with an argument to let the user decide, i.e. compile=False?

@AyushExel @Laughing-q what do you guys think of the new torch 2.0 compile examples in the link above?

CI fix

glenn-jocher · 2023-03-22T13:02:54Z

@N-Friederich I tried to compile on train, but getting compile errors. Seems like it may not be ready yet to apply by default, probably want an opt-in argument here.

N-Friederich · 2023-03-22T13:09:02Z

@glenn-jocher I think the problem is difficult to solve. Because what if you call train(...) in python and then export(...) on the same model. if this is compiled, you can no longer export it.

Yes might be a little complicated, but I think the first step would be to only compile on inference methods (predict, val, train), and then check in export if compiled: warning with explanation and exit.

@glenn-jocher I have never received an error. Tested it for a Titan RTX and MPS (+CPU).Can you send me your error?

N-Friederich · 2023-03-22T13:11:19Z

@N-Friederich I tried to compile on train, but getting compile errors. Seems like it may not be ready yet to apply by default, probably want an opt-in argument here.

@glenn-jocher That was my idea too. But I think it's a bit "ugly". That's why I wanted to build something better. But I think that is now the only option.

AyushExel · 2023-03-22T13:32:33Z

Yeah I agree with having compile=True opt in

N-Friederich · 2023-03-22T15:38:28Z

@glenn-jocher @AyushExel 1) The question is how we deal with benchmark. Because pytorch models should be supported. 2.) If export is used, only a warning and skipping, or an error? In case of an error, the tests would have to be adapted accordingly.

- Set compile=False for all export operations

…pile # Conflicts: # ultralytics/yolo/engine/model.py

N-Friederich · 2023-03-23T10:32:56Z

@glenn-jocher @AyushExel I have implemented it for now without any additional flag and only turned it off for export. Do you think we also need a flag?
By the way, the tutorials also need to be adjusted in the Jupyter. I have already adjusted the tests.

glenn-jocher · 2023-03-23T12:13:19Z

@N-Friederich got it! Please fix the CI errors that are appearing.

N-Friederich · 2023-03-23T13:39:16Z

@glenn-jocher This is interesting. I tested it locally before my commit and got no errors. I'll take a look at it.

N-Friederich · 2023-03-24T11:33:39Z

@glenn-jocher The problem is that the model is saved as a pickel. This is generally not recommended and leads to the error with compiled models. I have now changed the saving as recommended here. But their are some issues with weight loading.

glenn-jocher · 2023-03-24T14:19:20Z

@N-Friederich I haven't looked too deeply into the changes but I see you've added a manual argument on model load, which we want to avoid. We infer a model type in AutoBackend and then naturally don't run compile commands if type is not PyTorch.

YOLO() should be able to load and handle all export formats without any additional arguments required.

N-Friederich · 2023-04-04T12:07:29Z

@glenn-jocher The problem currently is rather the serialization. This is currently not possible directly from a compiled model. I.e. there must be considered an alternative. But I am not as deep in the code as you are.

(Source: https://pytorch.org/get-started/pytorch-2.0/)

N-Friederich · 2023-04-04T12:10:04Z

@N-Friederich I haven't looked too deeply into the changes but I see you've added a manual argument on model load, which we want to avoid. We infer a model type in AutoBackend and then naturally don't run compile commands if type is not PyTorch.

YOLO() should be able to load and handle all export formats without any additional arguments required.

@glenn-jocher I just changed something in the YOLO.init(...), is that what you meant by "model load"?

glenn-jocher · 2023-04-04T12:35:18Z

@N-Friederich yes, I meant the YOLO() initialization. You added a manual argument for model load, which we should try to avoid. Instead, we should infer the model type in AutoBackend to determine whether or not to run compile commands. Additionally, YOLO() should be able to load and handle all export formats without needing additional arguments.

N-Friederich · 2023-04-04T12:57:39Z

@N-Friederich yes, I meant the YOLO() initialization. You added a manual argument for model load, which we should try to avoid. Instead, we should infer the model type in AutoBackend to determine whether or not to run compile commands. Additionally, YOLO() should be able to load and handle all export formats without needing additional arguments.

@glenn-jocher Ok, I'll take a look. Thanks for the info.

glenn-jocher · 2023-04-04T16:53:20Z

@N-Friederich sure, I'd be happy to explain ideas and concepts without using code examples. Let me know what you need help with!

N-Friederich · 2023-04-11T06:53:03Z

@glenn-jocher From my perspective, there are several ways to resolve the problem. One could (similar to EMA) save a separate model for the compiled model and transfer the weights to the non-compiled model before saving. This is the best option from my perspective because as of now you can't serialize a compiled model directly (except half via the state_dict).

…py to ultralytics.yolo.engine.trainer.py

# Conflicts: # ultralytics/yolo/cfg/__init__.py

N-Friederich · 2023-04-13T15:35:17Z

@glenn-jocher @AyushExel I think I have found a relatively good way. Have implemented only for train for now, should we do it for the Val and Pred as well? I would say, if then we should compile the model before. Then the compile time is not included.
And srry that it took so long, I think I understand the code better now (The changes from YOLOv5 to YOLOv8 was bigger than I thought) :)

glenn-jocher · 2023-04-13T18:10:04Z

@N-Friederich That's great to hear that you have found a solution to the problem of serializing compiled models. It's important to note that the solution should be scaleable and applicable for all stages of the pipeline, which means that it should be implemented for Val and Pred as well. If the model needs to be compiled before the serialization, then we should include the compile time in the overall processing time as well. It's understandable that it took a while to familiarize yourself with the changes from YOLOv5 to YOLOv8, but taking the time to understand the code is necessary to ensure the quality of the contributions.

…ainer.py

nodeav · 2023-06-12T17:21:15Z

@N-Friederich @glenn-jocher Hey! is there any way to push this forward?
I'd be more than happy to chip in and lend a hand if it makes a difference.

N-Friederich · 2023-06-21T19:05:00Z

Hi @nodeav ,
thanks for your message! I had not tried since then. Unfortunately, it's not that easy, especially regarding the model export it's really tricky. I think the best way would be to do it with a decorator, but then some exports will fail.
Do you have ideas @glenn-jocher and @glenn-jocher ? Think this is a great-added value for YOLOv8.
Thx and best,
Nils

nodeav · 2023-06-22T15:26:12Z

I think if it's only done for training purposes (for now?), it might be simpler? then export the non-compiled version, because pytorch doesn't support it yet AFAIK

pderrenger · 2023-11-16T04:19:23Z

@nodeav absolutely, focusing on implementing this specifically for training purposes initially would simplify the process. After training, the non-compiled version can be exported, considering that PyTorch doesn't currently fully support exporting compiled models. This iterative approach maintains simplicity and can be expanded upon as the ecosystem evolves.

N-Friederich and others added 3 commits March 20, 2023 19:08

Add torch.compile(...) to Yolo.__init__(...)

351ce77

Pytorch.compile(...) only if version > 2.0

e713f55

[pre-commit.ci] auto fixes from pre-commit.com hooks

cc3fa9c

for more information, see https://pre-commit.ci

glenn-jocher changed the base branch from main to updates March 21, 2023 11:24

glenn-jocher changed the base branch from updates to main March 21, 2023 11:24

Merge branch 'main' into Pytorch2_Compile

f9bd39d

glenn-jocher added 3 commits March 22, 2023 13:23

Update trainer.py

20a601e

Update model.py

771d73e

Update trainer.py

a1a2dea

CI fix

N-Friederich added 3 commits March 23, 2023 11:27

Add "compile_model" arg to Yolo.__init__(...)

e64cd47

- Set compile=False for all export operations

Merge remote-tracking branch 'own/Pytorch2_Compile' into Pytorch2_Com…

df7735e

…pile # Conflicts: # ultralytics/yolo/engine/model.py

Pre-commit formatting

e984a48

N-Friederich added 2 commits April 13, 2023 17:17

Move torch.compile implementation from ultralytics.yolo.engine.model.…

b7b6f15

…py to ultralytics.yolo.engine.trainer.py

Merge branch 'main' into Pytorch2_Compile

bbfad36

# Conflicts: # ultralytics/yolo/cfg/__init__.py

Add torch.compile to AutoBackend and fix issue with model suage in tr…

836ca54

…ainer.py

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pytorch2 compile (#1505) #1516

Pytorch2 compile (#1505) #1516

N-Friederich commented Mar 20, 2023 •

edited by UltralyticsAssistant

glenn-jocher commented Mar 21, 2023

glenn-jocher commented Mar 21, 2023

N-Friederich commented Mar 21, 2023 •

edited

N-Friederich commented Mar 21, 2023 •

edited

glenn-jocher commented Mar 21, 2023

N-Friederich commented Mar 21, 2023

N-Friederich commented Mar 21, 2023

N-Friederich commented Mar 21, 2023

N-Friederich commented Mar 21, 2023

glenn-jocher commented Mar 22, 2023

glenn-jocher commented Mar 22, 2023

glenn-jocher commented Mar 22, 2023

N-Friederich commented Mar 22, 2023 •

edited

N-Friederich commented Mar 22, 2023 •

edited

AyushExel commented Mar 22, 2023

N-Friederich commented Mar 22, 2023

N-Friederich commented Mar 23, 2023 •

edited

glenn-jocher commented Mar 23, 2023

N-Friederich commented Mar 23, 2023

N-Friederich commented Mar 24, 2023 •

edited

glenn-jocher commented Mar 24, 2023

N-Friederich commented Apr 4, 2023

N-Friederich commented Apr 4, 2023

glenn-jocher commented Apr 4, 2023

N-Friederich commented Apr 4, 2023

glenn-jocher commented Apr 4, 2023

N-Friederich commented Apr 11, 2023

N-Friederich commented Apr 13, 2023

glenn-jocher commented Apr 13, 2023

nodeav commented Jun 12, 2023

N-Friederich commented Jun 21, 2023

nodeav commented Jun 22, 2023

pderrenger commented Nov 16, 2023

Pytorch2 compile (#1505) #1516

Are you sure you want to change the base?

Pytorch2 compile (#1505) #1516

Conversation

N-Friederich commented Mar 20, 2023 • edited by UltralyticsAssistant

🤖 Generated by Copilot at e984a48

Summary

Walkthrough

🛠️ PR Summary

📊 Key Changes

🎯 Purpose & Impact

🌟 Summary

glenn-jocher commented Mar 21, 2023

glenn-jocher commented Mar 21, 2023

N-Friederich commented Mar 21, 2023 • edited

N-Friederich commented Mar 21, 2023 • edited

glenn-jocher commented Mar 21, 2023

N-Friederich commented Mar 21, 2023

N-Friederich commented Mar 21, 2023

N-Friederich commented Mar 21, 2023

N-Friederich commented Mar 21, 2023

glenn-jocher commented Mar 22, 2023

glenn-jocher commented Mar 22, 2023

glenn-jocher commented Mar 22, 2023

N-Friederich commented Mar 22, 2023 • edited

N-Friederich commented Mar 22, 2023 • edited

AyushExel commented Mar 22, 2023

N-Friederich commented Mar 22, 2023

N-Friederich commented Mar 23, 2023 • edited

glenn-jocher commented Mar 23, 2023

N-Friederich commented Mar 23, 2023

N-Friederich commented Mar 24, 2023 • edited

glenn-jocher commented Mar 24, 2023

N-Friederich commented Apr 4, 2023

N-Friederich commented Apr 4, 2023

glenn-jocher commented Apr 4, 2023

N-Friederich commented Apr 4, 2023

glenn-jocher commented Apr 4, 2023

N-Friederich commented Apr 11, 2023

N-Friederich commented Apr 13, 2023

glenn-jocher commented Apr 13, 2023

nodeav commented Jun 12, 2023

N-Friederich commented Jun 21, 2023

nodeav commented Jun 22, 2023

pderrenger commented Nov 16, 2023

N-Friederich commented Mar 20, 2023 •

edited by UltralyticsAssistant

`🤖 Generated by Copilot at e984a48`

N-Friederich commented Mar 21, 2023 •

edited

N-Friederich commented Mar 21, 2023 •

edited

N-Friederich commented Mar 22, 2023 •

edited

N-Friederich commented Mar 22, 2023 •

edited

N-Friederich commented Mar 23, 2023 •

edited

N-Friederich commented Mar 24, 2023 •

edited