Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement XAI for tiling detection #2240

Merged
merged 28 commits into from
Jul 4, 2023

Conversation

GalyaZalesskaya
Copy link
Contributor

@GalyaZalesskaya GalyaZalesskaya commented Jun 13, 2023

Summary

This PR implements the functionality to use saliency maps for tiles and merge them in the unite saliency map. The current approach is to resize a whole image and use its saliency map, which suffers from the lack of details.
The results of the new approach and the current one are below.

image

Also, the exact same merging mechanism was applied for Mask RCNN (Instance Segmentation) Pytorch inference:
image

For Mask RCNN (Instance Segmentation) OpenVINO inference saliency maps aren't returned for each tole. So, the merged masks for the united image were used.
image

What was added:

  • Merging saliency maps functionality, using both saliency maps from tiles and a whole image, add non-linear normalization afterward

  • Add implementation for PyTorch inference and OpenVINO inference for all detection models

  • Add implementation to merge saliency maps for Pytorch task for Instance Segmentation task (Mask RCNN)

  • Updated saliency map generation for OpenVINO inference task for Instance Segmentation

    • Changed a way to merge masks for each objects, added confidence scores to mask value
    • Moved saliency map generation process for Instance Segmentation from /model_wrappers/openvino_models.py to tiler.py implementation to keep tiled saliency map generation for OpenVINO task in one file (negotiable)
  • Add parameter in BaseRecordingForwardHook whether to return normalized saliency maps

  • Add parameter in DetClassProbabilityMapHook whether to use softmax in saliency maps postprocessing

  • Fixed the issue there during workplace creation for Mask RCNN model deployment_tile_classifier.py config file wasn't created

  • Unit tests

  • e2e tests

CVS-108355

Checklist

  • I have added unit tests to cover my changes.​
  • I have added integration tests to cover my changes.​
  • I have added e2e tests for validation.
  • I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
  • I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
  • I have linked related issues.

License

  • I submit my code changes under the same Apache License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below).
# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

@github-actions github-actions bot added ALGO Any changes in OTX Algo Tasks implementation API Any changes in OTX API TEST Any changes in tests labels Jun 13, 2023
@github-actions github-actions bot removed the TEST Any changes in tests label Jun 14, 2023
@github-actions github-actions bot added the DOC Improvements or additions to documentation label Jun 14, 2023
@GalyaZalesskaya GalyaZalesskaya marked this pull request as ready for review June 15, 2023 21:43
@GalyaZalesskaya GalyaZalesskaya requested a review from a team as a code owner June 15, 2023 21:43
@eunwoosh
Copy link
Contributor

Could you add and update unit test?

Copy link
Contributor

@eunwoosh eunwoosh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for good enhancement! I left some comments. please take a look.

otx/algorithms/detection/adapters/mmdet/task.py Outdated Show resolved Hide resolved
otx/api/utils/tiler.py Outdated Show resolved Hide resolved
@jaegukhyun
Copy link
Contributor

Overally LGTM, As Eunwoo said, could you add unit test to cover your PR? And is there no need to add intg or e2e test for this feature?

@negvet
Copy link
Contributor

negvet commented Jun 16, 2023

Thank you for a PR!
Could you please rebase on develop (#2227 was merged), and take a look on how it works for m-rcnn (both torch and OV tasks)? thanks!

CHANGELOG.md Outdated Show resolved Hide resolved
@jaegukhyun
Copy link
Contributor

Could you resolve conflicts?

@github-actions github-actions bot added the CLI Any changes in OTE CLI label Jun 29, 2023
@github-actions github-actions bot added the TEST Any changes in tests label Jun 29, 2023
@GalyaZalesskaya
Copy link
Contributor Author

@eunwoosh @jaegukhyun @negvet @sovrasov
Sorry for the delay, I was updating my PR to cover case with Mask RCNN Instance Segmentation model and to cover latest changes in develop.
Thank you all for your valuable comments. Can you please look again?

@GalyaZalesskaya
Copy link
Contributor Author

@negvet @sovrasov Also would like to hear your opinion about moving saliency map generation for Mask RCNN to tiler.py

otx/api/utils/tiler.py Outdated Show resolved Hide resolved
otx/api/utils/tiler.py Outdated Show resolved Hide resolved
sovrasov
sovrasov previously approved these changes Jul 3, 2023
@eunwoosh
Copy link
Contributor

eunwoosh commented Jul 4, 2023

Could you check fail test cases?

@sovrasov sovrasov enabled auto-merge (squash) July 4, 2023 14:45
@sovrasov sovrasov merged commit ec41b6c into openvinotoolkit:develop Jul 4, 2023
14 checks passed
@GalyaZalesskaya GalyaZalesskaya mentioned this pull request Jul 5, 2023
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ALGO Any changes in OTX Algo Tasks implementation API Any changes in OTX API CLI Any changes in OTE CLI DOC Improvements or additions to documentation TEST Any changes in tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants