Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable training on XPU devices in OTX2.0 #3094

Merged
merged 67 commits into from
Mar 24, 2024
Merged

Enable training on XPU devices in OTX2.0 #3094

merged 67 commits into from
Mar 24, 2024

Conversation

kprokofi
Copy link
Contributor

Summary

How to test

Checklist

  • I have added unit tests to cover my changes.​
  • I have added integration tests to cover my changes.​
  • I have added e2e tests for validation.
  • I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
  • I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
  • I have linked related issues.

License

  • I submit my code changes under the same Apache License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below).
# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

kprokofi and others added 29 commits January 14, 2024 23:05
@kprokofi
add raising an error when metric is None
0abdf10
@kprokofi
Merge branch 'v2' of https://github.com/openvinotoolkit/training_exte…
8756968 
…nsions into v2
@kprokofi
Merge branch 'v2' of https://github.com/openvinotoolkit/training_exte…
e171e9d 
…nsions into v2
@kprokofi
Merge branch 'v2' of https://github.com/openvinotoolkit/training_exte…
4e6e21e 
…nsions into v2
@kprokofi
Merge branch 'v2' of https://github.com/openvinotoolkit/training_exte…
c0abe24 
…nsions into v2
@kprokofi
Merge branch 'v2' of https://github.com/openvinotoolkit/training_exte…
1868961 
…nsions into v2
@kprokofi
Merge branch 'v2' of https://github.com/openvinotoolkit/training_exte…
d33e66e 
…nsions into v2
@kprokofi
Merge branch 'v2' of https://github.com/openvinotoolkit/training_exte…
35e925f 
…nsions into v2
@kprokofi
Merge branch 'v2' of https://github.com/openvinotoolkit/training_exte…
3de253f 
…nsions into v2
@kprokofi
Merge branch 'v2' of https://github.com/openvinotoolkit/training_exte…
3f0ce95 
…nsions into v2
@kprokofi
Merge branch 'v2' of https://github.com/openvinotoolkit/training_exte…
bddffa6 
…nsions into v2
@kprokofi
Merge branch 'v2' of https://github.com/openvinotoolkit/training_exte…
8b69f62 
…nsions into v2
@kprokofi
Merge branch 'v2' of https://github.com/openvinotoolkit/training_exte…
7efa031 
…nsions into v2
@kprokofi
Merge branch 'v2' of https://github.com/openvinotoolkit/training_exte…
1f8b9ce 
…nsions into v2
@kprokofi
Merge branch 'v2' of https://github.com/openvinotoolkit/training_exte…
6e0f0b6 
…nsions into v2
@kprokofi
Merge branch 'v2' of https://github.com/openvinotoolkit/training_exte…
4619997 
…nsions into v2
@kprokofi
added accelerators
96104fc
@kprokofi
fix packages
d09392c
@chuneuny-emily
fix assigning model
8c59e04
@kprokofi
debug on MAX
57b0f33
@kprokofi
change precision
1471fbb
@kprokofi
update MixedPrecisionXPUPlugin
0a4a69a
@kprokofi
debug
d77b84b
@kprokofi
merge
37601e2
@kprokofi
Merge branch 'kp/xpu_otx2.0' of https://github.com/openvinotoolkit/tr…
272a534 
…aining_extensions into kp/xpu_otx2.0
@kprokofi
added monkey patching
79ec108
@kprokofi
minor
1558295
@kprokofi
minor
d71492a
@kprokofi
Merge branch 'kp/xpu_otx2.0' of https://github.com/openvinotoolkit/tr…
1e7f005 
…aining_extensions into kp/xpu_otx2.0
@github-actions github-actions bot added the DEPENDENCY Any changes in any dependencies (new dep or its version) should be produced via Change Request on PM label Mar 13, 2024
@harimkang harimkang added this to the 2.0.0 milestone Mar 21, 2024
harimkang
harimkang previously approved these changes Mar 21, 2024
View reviewed changes
@kprokofi
Copy link
Contributor Author

kprokofi commented Mar 22, 2024 •
edited

I updated strategy: 98e0b69 removing torch.xpu.optimize for Semantic Segmentation since it was reported by IPEX that they have a bug there in that case and to be able to train segmentation model we should remove optimization.
I hope it will be fixed in the next IPEX releases

@kprokofi
Copy link
Contributor Author

kprokofi commented Mar 22, 2024 •
edited

@harimkang , seems like a problem with installing mmdet occurs during set up env for unit tests. Could you take a look?

@harimkang
Copy link
Contributor

@harimkang , seems like a problem with installing mmdet occurs during set up env for unit tests. Could you take a look?

There was a brief issue with the CI network. I rerun the test and the install is fine.

@kprokofi kprokofi enabled auto-merge (squash) March 24, 2024 22:45
@kprokofi kprokofi merged commit 9c746da into releases/2.0.0 Mar 24, 2024
16 checks passed
@kprokofi kprokofi deleted the kp/xpu_otx2.0 branch March 24, 2024 23:30
kprokofi added a commit that referenced this pull request Apr 9, 2024
@kprokofi @chuneuny-emily
Enable training on XPU devices in OTX2.0 (#3094)
0ffcdc5 
* add raising an error when metric is None

* added accelerators

* fix packages

* fix assigning model

* debug on MAX

* change precision

* update MixedPrecisionXPUPlugin

* debug

* added monkey patching

* minor

* minor

* added patch for mmengine

* fix OD and IS

* benchmark debug

* change device

* quick fix for instance seg

* fix pre-commit

* fix pre-commit

* clean the code

* added additional flag for mmcv

* added unit tests

* fixed unit test

* fix linter

* added unit tests and replied comments

* fix pre-commit

* minor fix

* added documentation

* fix unit test

* add workaround for semantic segmentation

* remove RoiAlignTest due to unstability

* minor

* remove strategy back

* try to patch SingleDeviceStrategy

* added auto xpu configuration

* patch strategy

* small fix

* reply to comments

* move patching xpu packages to accelerator

* fix test_xpu test

* remove do-not-install-mmcv

* fix pre-commit

* remove torch.xpu.optimize for segmentation

---------

Co-authored-by: Emily <emily.chun@intel.com>
@kprokofi kprokofi mentioned this pull request Apr 9, 2024
Merged
8 tasks
kprokofi added a commit that referenced this pull request Apr 16, 2024
@kprokofi @chuneuny-emily @eunwoosh
Move XPU/Opt related PRs from releases2.0 to develop (#3295)
255be7a 
* Enable training on XPU devices in OTX2.0 (#3094)

* add raising an error when metric is None

* added accelerators

* fix packages

* fix assigning model

* debug on MAX

* change precision

* update MixedPrecisionXPUPlugin

* debug

* added monkey patching

* minor

* minor

* added patch for mmengine

* fix OD and IS

* benchmark debug

* change device

* quick fix for instance seg

* fix pre-commit

* fix pre-commit

* clean the code

* added additional flag for mmcv

* added unit tests

* fixed unit test

* fix linter

* added unit tests and replied comments

* fix pre-commit

* minor fix

* added documentation

* fix unit test

* add workaround for semantic segmentation

* remove RoiAlignTest due to unstability

* minor

* remove strategy back

* try to patch SingleDeviceStrategy

* added auto xpu configuration

* patch strategy

* small fix

* reply to comments

* move patching xpu packages to accelerator

* fix test_xpu test

* remove do-not-install-mmcv

* fix pre-commit

* remove torch.xpu.optimize for segmentation

---------

Co-authored-by: Emily <emily.chun@intel.com>

* Add exporter/demo unit tests (#3218)

* added unit tests. Need to clean up

* move tests

* fix pre-commit

* return demo back

* minor

* delete unnecessery comments

* fix unit test

* fix pre-commit

* fix pre-commit 2

* fix test_postprocess_openvino_model

* fix unit tests

* test_precommit

* Fix a bug that engine.test doesn't work with XPU (#3293)

* fix bug

* align with pre-commit

---------

Co-authored-by: Emily <emily.chun@intel.com>

* fix merge conflicts for pre-commit

* fix precommit 2

* fix unit test

* fix pre-commit

* fix export tests

* fix pre-commit

* fix tox

* fix pre-commit

---------

Co-authored-by: Emily <emily.chun@intel.com>
Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vinnamkim vinnamkim vinnamkim left review comments

@eugene123tw eugene123tw eugene123tw approved these changes

@eunwoosh eunwoosh eunwoosh approved these changes

@jaegukhyun jaegukhyun jaegukhyun approved these changes

@harimkang harimkang harimkang left review comments

@samet-akcay samet-akcay Awaiting requested review from samet-akcay samet-akcay is a code owner

@sungmanc sungmanc Awaiting requested review from sungmanc

@chuneuny-emily chuneuny-emily Awaiting requested review from chuneuny-emily chuneuny-emily is a code owner

@sovrasov sovrasov Awaiting requested review from sovrasov sovrasov is a code owner

Assignees
No one assigned
Labels
DOC Improvements or additions to documentation OTX 2.0 For OTX v2.0 TEST Any changes in tests
Projects
None yet
Milestone
2.0.0
Development

Successfully merging this pull request may close these issues.

None yet
7 participants
@kprokofi @harimkang @eugene123tw @vinnamkim @eunwoosh @jaegukhyun @chuneuny-emily