Add mmcls.VisionTransformer backbone support #1908

harimkang · 2023-03-20T04:44:25Z

Summary

New Backbone Support

mmcls.VisionTransformer
mmcls.T2T_ViT
mmcls.Conformer

mmcls.VisionTransformer support

otx find with mmcls transformer backbones
otx build with mmcls transformer backbones
otx train with mmcls transformer backbones

Things to know:
The current implementation uses dump_feature and dump_saliency_map as False in infer when using VisionTransformerClsHead due to the different output structure of transformer.

How to test

(otx) otx find --task classification --backbone mmcls
(otx) otx build --task classification
(otx) cd otx-workspace-CLASSIFICATION
# Edit data.yaml
(otx) otx build --backbone mmcls.VisionTransformer
(otx) otx train

Checklist

I submit my changes into the develop branch
I have added description of my changes into CHANGELOG
I have updated the documentation accordingly
I have added tests to cover my changes
I have linked related issues

License

I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
I have updated the license header for each file (see an example below)

# Copyright (C) 2023 Intel Corporation
#
# SPDX-License-Identifier: MIT

goodsong81

Thank you for the quick fix.

sungmanc

Shouldn't we check whether these backbones are successfully represented at otx find or not?

otx/algorithms/classification/configs/configuration.yaml

harimkang · 2023-03-20T05:24:30Z

Shouldn't we check whether these backbones are successfully represented at otx find or not?

The existing branch cannot find mmcls.VisionTransformer in otx find, but it will be added in this PR.

However, I still need to verify that this works properly, so there may be additional changes in this PR.

harimkang · 2023-03-20T05:27:08Z

@JihwanEom @sungchul2 This will not work with these settings at this time, please let us know what additional settings need. I want to make the command below work.

(otx) otx build --task classification
(otx) cd otx-workspace-CLASSIFICATION
# Edit data.yaml
(otx) otx build --backbone mmcls.VisionTransformer
(otx) otx train

codecov-commenter · 2023-03-20T05:48:47Z

Codecov Report

❗ No coverage uploaded for pull request base (releases/1.1.0@e3aa433). Click here to learn what that means.
Patch has no changes to coverable lines.

Additional details and impacted files

@@                Coverage Diff                @@
##             releases/1.1.0    #1908   +/-   ##
=================================================
  Coverage                  ?   80.66%           
=================================================
  Files                     ?      489           
  Lines                     ?    33403           
  Branches                  ?        0           
=================================================
  Hits                      ?    26946           
  Misses                    ?     6457           
  Partials                  ?        0

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

goodsong81

Not to merge.

otx/mpa/cls/inferrer.py

goodsong81

LGTM! Welcome new backbones!

sungchul2

Thanks for your hard work. I left some questions and let me know the answers.

otx/algorithms/classification/configs/configuration.yaml

otx/cli/builder/builder.py

otx/mpa/cls/inferrer.py

harimkang · 2023-03-21T02:32:10Z

Thanks for your hard work. I left some questions and let me know the answers.

@goodsong81 I modified the max_value of batch size in configuration.yaml, will this affect Geti?

@sungchul2 Hooks (FeatureVectorHook, ReciproCAMHook) may need to be added for 'otx explain' functionality, but I'm not familiar with those hooks, so I've disabled it for now. This is something that needs to be discussed, but for now training and exporting are fine. @goodsong81 Should we support the explain of the Transformer model in this PR?

* Bump up version to 1.1.0rc1 * Updated daily workflow (#1905) - remove if statement to allow running on any branch by manually * [FIX] Wrong test temp directory path (#1902) * Fix wrong test temp directory * Update tests/unit/algorithms/action/adapters/mmaction/utils/test_action_config_utils.py * Update test_action_config_utils.py * Update PR template (#1914) * Update . location. * Fix OTX1.1 -> Geti1.4 integration issues (#1910) * Add dill to requirements/api.txt * Return None instead of raising NotImplementedError in IMedia2DEntity.path * Move utils HPO uses into HPO directory (#1912) move utils into hpo dir * [FIX] Add stability to explain detection (#1901) Add stability to explain detection * [Fix] e2e tests FQ references (#1918) Fix FQ references * Move mpa.deploy to otx.algorithms.common (#1903) * Move deploy modules to otx * test: fix mmdeploy api replacement error * Fix pre-commit issues * Update otx/mpa/det/exporter.py * Update otx/mpa/exporter_mixin.py * Update otx/mpa/seg/exporter.py * Update otx/mpa/cls/exporter.py * Update otx/algorithms/common/adapters/mmdeploy/utils/mmdeploy.py * Update otx/algorithms/common/adapters/mmdeploy/utils/operations_domain.py * Add mmcls.VisionTransformer backbone support (#1908) * Add mmcls transformer backbones * Fix VisionTransformeroutput check * Add changes * Disable recording forward hooks in inferrer * Remove unused import * Move semantic-segmentation related codes to otx adapters (#1911) * Added security.md Add the security related notification * Update security.md fix prettier issue * Add custom exception class for CLI (#1919) * Add custom exception class for CLI * Fixed TCs for config_manager * Correct License to Apache * Remove temp hot key values (#1916) * Refactor OTX classification phase 1: move modules from MPA to OTX (#1893) * Revert TrainType typo (#1928) * Fix Conflict * Fix conflict * Fix unit-tests * Fix cli tests * Fix cli tests * Fix type * Move all hooks in MPA into OTX common mmcv adapter (#1922) * Add explanation for XAI & minor doc fixes (#1923) * [CI] Updated daily workflow (#1904) Updated daily workflow - remove if statement to allow running on any branch by manually * [FIX] re-bugfix: ATSS head loss (#1907) re bugfix * Fix typos * Explanation of Explanation * Add images & typo fixes * Fixes from comments * Add accuracy for OD explanation * Tutorial update * Add accuracy for BCCD and WGISD * Fix * Move utils to OTX common & adapter (#1931) * [CI] Updated daily workflow (#1904) Updated daily workflow - remove if statement to allow running on any branch by manually * [FIX] re-bugfix: ATSS head loss (#1907) re bugfix * Added security.md Added the security related notification. * Update security.md fix prettier issue * Move utils * Fix typo * Move classification unit tests to OTX tests (#1932) * [CI] Updated daily workflow (#1904) Updated daily workflow - remove if statement to allow running on any branch by manually * [FIX] re-bugfix: ATSS head loss (#1907) re bugfix * Added security.md Added the security related notification. * Update security.md fix prettier issue * Move classification unit tests * Apply mocker in RandAugment TC * Move otx.mpa.modules.ov -> otx.core.ov (#1929) * First Refactoring * Refactor mpa.modules.ov -> otx.core.ov * Remove otx.mpa.modules.ov * Fix docs & unit test paths * Fix some import * Add TODO comments * Add github icon link in doc head (#1934) * Fix invalid temp_dir inside unit tests (#1937) * Move all model utils and dataloaders/samplers in mpa to otx adapter (#1930) * Move stages to each algorithms (#1938) * Move stages to each algorithms * Change docs for action recognition (#1940) * Change docs for action recognition * Fix typo * Update MoViNet related parts * Add MoViNet performance * Revert table include Complexity and Model size * Fix tox to run tests on non-editable install * Fix packaging errors including cython module build / import issues (#1936) * Avoid cychon compile on Windows * Fix packaging errors * Fix pre-commit * Remove use_develop option from tox tests * Fix pre-commit * Fix tox -e -> -re for clean build * Fix import error * Add missing __init__.py * Fix relative -> absolute import * Revert to editable package in tox tests * Release 1.1.0 (#1933) * Release OTX 1.1.0 * Update known issues in CHANGELOG.md * Fix PR / issue # to links * Fix pre-commit * Bump-up version to 1.2 * Fix pre-commit --------- Signed-off-by: Songki Choi <songki.choi@intel.com> Co-authored-by: Yunchu Lee <yunchu.lee@intel.com> Co-authored-by: Harim Kang <harim.kang@intel.com> Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com> Co-authored-by: Emily Chun <emily.chun@intel.com> Co-authored-by: emily.chun <chuneuny@ikvensx012> Co-authored-by: Sungman Cho <sungman.cho@intel.com> Co-authored-by: Inhyuk Andy Cho <andy.inhyuk.jo@intel.com> Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com> Co-authored-by: Jihwan Eom <jihwan.eom@intel.com> Co-authored-by: Soobee Lee <soobee.lee@intel.com> Co-authored-by: Galina Zalesskaya <galina.zalesskaya@intel.com>

Add mmcls transformer backbones

85f033a

harimkang requested a review from a team as a code owner March 20, 2023 04:44

harimkang changed the title ~~[ENHANCE] Transformer backbone support~~ [ENHANCE] Transformer mmcls backbone support Mar 20, 2023

github-actions bot added ALGO Any changes in OTX Algo Tasks implementation CLI Any changes in OTE CLI labels Mar 20, 2023

harimkang added this to the 1.1.0 milestone Mar 20, 2023

goodsong81 previously approved these changes Mar 20, 2023

View reviewed changes

sungmanc reviewed Mar 20, 2023

View reviewed changes

otx/algorithms/classification/configs/configuration.yaml Show resolved Hide resolved

Fix VisionTransformeroutput check

3b0c148

harimkang dismissed goodsong81’s stale review via 3b0c148 March 20, 2023 05:21

harimkang requested review from sungchul2 and JihwanEom March 20, 2023 05:24

harimkang changed the title ~~[ENHANCE] Transformer mmcls backbone support~~ [ENHANCE] mmcls.VisionTransformer backbone support Mar 20, 2023

harimkang added the DO NOT MERGE label Mar 20, 2023

goodsong81 requested changes Mar 20, 2023

View reviewed changes

chuneuny-emily added the ENHANCE Enhancement of existing features label Mar 20, 2023

harimkang added 2 commits March 20, 2023 22:03

Add changes

86baffc

Disable recording forward hooks in inferrer

d300f7c

harimkang commented Mar 20, 2023

View reviewed changes

otx/mpa/cls/inferrer.py Show resolved Hide resolved

Remove unused import

6310554

harimkang removed the DO NOT MERGE label Mar 20, 2023

goodsong81 approved these changes Mar 21, 2023

View reviewed changes

goodsong81 changed the title ~~[ENHANCE] mmcls.VisionTransformer backbone support~~ Add mmcls.VisionTransformer backbone support Mar 21, 2023

goodsong81 added the RELEASE label Mar 21, 2023

sungchul2 reviewed Mar 21, 2023

View reviewed changes

otx/algorithms/classification/configs/configuration.yaml Show resolved Hide resolved

otx/cli/builder/builder.py Show resolved Hide resolved

otx/mpa/cls/inferrer.py Show resolved Hide resolved

sungchul2 approved these changes Mar 21, 2023

View reviewed changes

harimkang merged commit f7e4799 into openvinotoolkit:releases/1.1.0 Mar 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add mmcls.VisionTransformer backbone support #1908

Add mmcls.VisionTransformer backbone support #1908

harimkang commented Mar 20, 2023 •

edited by JihwanEom

goodsong81 left a comment

sungmanc left a comment

harimkang commented Mar 20, 2023

harimkang commented Mar 20, 2023

codecov-commenter commented Mar 20, 2023 •

edited

goodsong81 left a comment

goodsong81 left a comment

sungchul2 left a comment

harimkang commented Mar 21, 2023

Add mmcls.VisionTransformer backbone support #1908

Add mmcls.VisionTransformer backbone support #1908

Conversation

harimkang commented Mar 20, 2023 • edited by JihwanEom

Summary

How to test

Checklist

License

goodsong81 left a comment

Choose a reason for hiding this comment

sungmanc left a comment

Choose a reason for hiding this comment

harimkang commented Mar 20, 2023

harimkang commented Mar 20, 2023

codecov-commenter commented Mar 20, 2023 • edited

Codecov Report

goodsong81 left a comment

Choose a reason for hiding this comment

goodsong81 left a comment

Choose a reason for hiding this comment

sungchul2 left a comment

Choose a reason for hiding this comment

harimkang commented Mar 21, 2023

harimkang commented Mar 20, 2023 •

edited by JihwanEom

codecov-commenter commented Mar 20, 2023 •

edited