Add visual prompting task

[JihwanEom]

Summary

This PR is to add visual prompting task (reference paper: https://arxiv.org/abs/2304.12306).
This PR includes train and eval tasks, and others (deploy, export, ...) will be proceeded in the next PRs.

TODOs

• SAM model implementation
• Create VisualPromptingDatasetAdapter
  • SAM receives images, bboxes, and points, (optionally masks and texts) as inputs, but the current DetectionDatasetAdapter handles only images as inputs and others as annotations. It is required to create a new dataset adapter that can handle multiple inputs for visual prompting task.
• otx functions
  • train
    • w/ bboxes
    • w/ points -> will be implemented
    • w/ both prompts + cyclic training -> will be implemented
  • eval
    • (same with DatasetAdapter) When otx eval, with_empty_annotations() is applied and inputs of visual prompting but assigned as annotations are removed.
      

      training_extensions/otx/cli/tools/eval.py

      Lines 135 to 138 in 6b045da

      	predicted_validation_dataset = task.infer(
      	validation_dataset.with_empty_annotations(),
      	InferenceParameters(is_evaluation=False),
      	)

• Support various callbacks
  • checkpoint
  • early stopping
  • inference
  • ...
• Tests
  • Unit test
  • Integration test
• Documentation -> to be proceeded in other PR

How to test

Unit tests (local execution)

• otx/algorithms/visual_prompting/utils/visual_prompting_utils.py will be implemented if needed.

Name                                                                                                            Stmts   Miss  Cover   Missing
---------------------------------------------------------------------------------------------------------------------------------------------
$ tox -vv -e tests-all-py38 -- tests/unit/algorithms/visual_prompting
======================= 127 passed, 93 warnings in 7.87s =======================

$ tox -vv -e tests-all-py310 -- tests/unit/algorithms/visual_prompting
======================= 127 passed, 93 warnings in 8.17s =======================
otx/algorithms/visual_prompting/adapters/pytorch_lightning/callbacks/inference.py                                  40      0   100%
otx/algorithms/visual_prompting/adapters/pytorch_lightning/config/visual_prompting_config.py                       47      1    98%   89
otx/algorithms/visual_prompting/adapters/pytorch_lightning/datasets/dataset.py                                    112      9    92%   95, 99, 112, 169, 225, 269-272
otx/algorithms/visual_prompting/adapters/pytorch_lightning/datasets/pipelines/transforms.py                        90      9    90%   66, 100-107, 118-119
otx/algorithms/visual_prompting/adapters/pytorch_lightning/models/backbones/vit.py                                122      6    95%   197-198, 203, 267, 292, 319
otx/algorithms/visual_prompting/adapters/pytorch_lightning/models/decoders/sam_mask_decoder.py                    153      1    99%   199
otx/algorithms/visual_prompting/adapters/pytorch_lightning/models/encoders/sam_image_encoder.py                    11      0   100%
otx/algorithms/visual_prompting/adapters/pytorch_lightning/models/encoders/sam_prompt_encoder.py                   94      0   100%
otx/algorithms/visual_prompting/adapters/pytorch_lightning/models/utils/layer_norm.py                              14      0   100%
otx/algorithms/visual_prompting/adapters/pytorch_lightning/models/utils/mlp_block.py                               10      0   100%
otx/algorithms/visual_prompting/adapters/pytorch_lightning/models/visual_prompters/segment_anything.py            180      7    96%   172-173, 362-363, 374-376
otx/algorithms/visual_prompting/configs/base/configuration.py                                                       8      0   100%
otx/algorithms/visual_prompting/tasks/inference.py                                                                130     23    82%   90-91, 139, 158-164, 169-170, 175, 215, 224, 256, 260-269, 276-277, 281-282
otx/algorithms/visual_prompting/tasks/train.py                                                                     37      4    89%   66-67, 86-87
otx/algorithms/visual_prompting/utils/visual_prompting_utils.py                                                     3      3     0%   17-22

$ tox -vv -e tests-all-py310 -- tests/unit/api/entities/test_model_template.py
======================== 25 passed, 1 warning in 0.84s ========================
otx/api/entities/model_template.py                                                                                229      8    97%   234, 560-568

$ tox -vv -e tests-all-py310 -- tests/unit/cli/manager/test_config_manager.py
======================== 27 passed, 1 warning in 3.53s ========================
otx/cli/manager/config_manager.py                                                                                 377     82    78%   139-145, 240, 310-311$ 366, 391, 395, 399, 404, 420-441, 479, 490, 492, 494, 496-507, 511-513, 542, 545, 557-568, 582, 585, 587, 589, 613, 617, 655-656, 662-677, 681-689, 709

$ tox -vv -e tests-all-py310 -- tests/unit/core/data/adapter/test_visual_prompting_adapter.py
======================== 6 passed, 9 warnings in 3.34s ========================
otx/core/data/adapter/visual_prompting_dataset_adapter.py                                                          54      2    96%   64, 80

$ tox -vv -e tests-all-py310 -- tests/unit
FAILED tests/unit/algorithms/common/adapters/mmcv/nncf/test_mmcv_nncf_runners.py::TestAccuracyAwareRunner::test_run
FAILED tests/unit/algorithms/detection/test_xai_detection_validity.py::TestExplainMethods::test_saliency_map_det[Custom_Object_Detection_YOLOX]
FAILED tests/unit/algorithms/detection/test_xai_detection_validity.py::TestExplainMethods::test_saliency_map_det[Custom_Object_Detection_Gen3_SSD]
= 3 failed, 2259 passed, 13 skipped, 9 xfailed, 5 xpassed, 1167 warnings in 320.69s (0:05:20) =

Integration test (local execution)

$ tox -vv -e tests-all-py310 -- tests/integration/cli/visual_prompting
================== 6 passed, 1 warning in 3278.82s (0:54:38) ===================

Checklist

• I have added unit tests to cover my changes.​
• I have added integration tests to cover my changes.​
• I have added e2e tests for validation.
• I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
• I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
• I have linked related issues.

License

• I submit my code changes under the same Apache License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below).

# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

[eunwoosh]

python package bson is newly used in this update. How about adding bson in requirement file?

[eunwoosh]

visual prompting seems new task like classification or anomaly tasks. Should user install otx with pip install otx[full] to use this task? If then, I think that a way to install this task only is necessary like other tasks should be provided.

[sungmanc]

python package bson is newly used in this update. How about adding bson in requirement file?

Isn't it the default package from python? If so, I think we don't need to.

[sungchul2]

python package bson is newly used in this update. How about adding bson in requirement file?

Isn't it the default package from python? If so, I think we don't need to.

Yes, it is already used in otx.

[sungchul2]

visual prompting seems new task like classification or anomaly tasks. Should user install otx with pip install otx[full] to use this task? If then, I think that a way to install this task only is necessary like other tasks should be provided.

Yes, it is required, but I think it would be better to implement it after implementing other tasks, export, deploy, and optimize.
What do you think?

[eunwoosh]

python package bson is newly used in this update. How about adding bson in requirement file?

Isn't it the default package from python? If so, I think we don't need to.

Yes, it is already used in otx.

I don't think so. When I use pure python, there is no bson. I think packages OTX uses has dependency with bson. But I think it should be added regardless of python package OTX uses is uses bson or not, because OTX can not use that package later.
I think you will provide a way to install visual prompt task independently and need to make a requirement file for that. How about to add it later then?

[eunwoosh]

visual prompting seems new task like classification or anomaly tasks. Should user install otx with pip install otx[full] to use this task? If then, I think that a way to install this task only is necessary like other tasks should be provided.

Yes, it is required, but I think it would be better to implement it after implementing other tasks, export, deploy, and optimize. What do you think?

Yes, I agree with you.

[eunwoosh]

Thanks for your huge effort :) I left some comments although I didn't check all the files. please take a look.

[sungchul2]

python package bson is newly used in this update. How about adding bson in requirement file?

Isn't it the default package from python? If so, I think we don't need to.

Yes, it is already used in otx.

I don't think so. When I use pure python, there is no bson. I think packages OTX uses has dependency with bson. But I think it should be added regardless of python package OTX uses is uses bson or not, because OTX can not use that package later. I think you will provide a way to install visual prompt task independently and need to make a requirement file for that. How about to add it later then?

I'll consider it.

[jaegukhyun]

Generally LGTM, I left a few comments.

[jaegukhyun]

LGTM, Please clean commits, when you merge this PR