Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Image Explorer] CanvasTools Image Loading support for Object Detection #2097

Merged
merged 31 commits into from
Jun 13, 2023
Merged

[Image Explorer] CanvasTools Image Loading support for Object Detection #2097

merged 31 commits into from
Jun 13, 2023

Conversation

Advitya17
Copy link
Collaborator

@Advitya17 Advitya17 commented Jun 6, 2023 •
edited
Loading

This PR replaces the existing image loading logic in the Flyout (using <Image> from FluentUI) to vott-ct (CanvasTools) for the Object Detection scenario.

This PR also fixes some minor typos on the mean_pixel_value and rounds off their values for readability.

Description

Advantages:

  1. Reusability with Data Labeling team's work, which unlocks loading and overlaying elements on images for CV tasks.
  2. Unlocks the potential to draw bounding boxes in the frontend, which'll enable on-the-fly functionality of filtering/displaying bounding boxes based on user inputs related to error types, classes etc. Aligned with customer feedback & feature requests.
  3. Piggybacks on CanvasTools' accessibility standards (including for overlaid elements)
  4. Slightly higher res display compared to <Image>

This support may also be emulated for other scenarios of the dashboard in future PRs as desired.

image

image

Checklist

  • I have added screenshots above for all UI changes.
  • I have added e2e tests for all UI changes.
  • Documentation was updated if it was needed.

@github-actions
Copy link

github-actions bot commented Jun 6, 2023

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/agemawat/bbox_vott/dashboard/index.html

@natalie-isak
Copy link
Collaborator

I'm a little confused by the second UI screenshot in the description. Why are there such large bounding boxes?

(Also, are those UI screenshots created by the changes in this PR? I thought the bboxes were magenta/blue?)

@natalie-isak
Copy link
Collaborator

I remember in an initial demo of CanvasTools, there was the possibility of adding shading to bboxes. How difficult is this to add, maybe can be a consideration for a future PR?

@Advitya17
Copy link
Collaborator Author

I remember in an initial demo of CanvasTools, there was the possibility of adding shading to bboxes. How difficult is this to add, maybe can be a consideration for a future PR?

@natalie-isak tags support adding a color. Seems like those are reflected in the box as well, but may have to check when the boxes get rendered.

@Advitya17
Copy link
Collaborator Author

I'm a little confused by the second UI screenshot in the description. Why are there such large bounding boxes?

(Also, are those UI screenshots created by the changes in this PR? I thought the bboxes were magenta/blue?)

The bounding boxes are from the backend itself. Those big white lines are simply tracing where the users hovers on the image (seems like default canvastools behavior). Hence tried to draw a red arrow (as the arrow can't be trapped in a screenshot) to show that's where I was hovering during the screenshot :)

@github-actions
Copy link

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/agemawat/bbox_vott/dashboard/index.html

1 similar comment
@github-actions
Copy link

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/agemawat/bbox_vott/dashboard/index.html

@github-actions
Copy link

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/agemawat/bbox_vott/dashboard/index.html

@github-actions
Copy link

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/agemawat/bbox_vott/dashboard/index.html

@github-actions
Copy link

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/agemawat/bbox_vott/dashboard/index.html

@github-actions
Copy link

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/agemawat/bbox_vott/dashboard/index.html

1 similar comment
@github-actions
Copy link

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/agemawat/bbox_vott/dashboard/index.html

imatiach-msft
imatiach-msft approved these changes Jun 13, 2023
View reviewed changes
Copy link
Contributor

@imatiach-msft imatiach-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

signing off except for the 300 line limit issue

@github-actions
Copy link

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/agemawat/bbox_vott/dashboard/index.html

2 similar comments
@github-actions
Copy link

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/agemawat/bbox_vott/dashboard/index.html

@github-actions
Copy link

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/agemawat/bbox_vott/dashboard/index.html

imatiach-msft
imatiach-msft approved these changes Jun 13, 2023
View reviewed changes
Copy link
Contributor

@imatiach-msft imatiach-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re-approving

@imatiach-msft imatiach-msft merged commit f55e4be into main Jun 13, 2023
@imatiach-msft imatiach-msft deleted the agemawat/bbox_vott branch June 13, 2023 22:07
vinuthakaranth pushed a commit that referenced this pull request Jul 5, 2023
@Advitya17 @vinuthakaranth
[Image Explorer] CanvasTools Image Loading support for Object Detecti…
3609179 
…on (#2097)

* bbox vott template ckpt

* canvastools image load ckpt

* loading image from scratch ckpt

* canvastools image loading support

* coordinate fixes

* regiondata call + path fix

* code cleanup

* image loading ckpt

* callback image loading support

* lint fixes

* disabling internal imports for vott

* file refactor lint fix

* auto lint fixes

* lint fixes

* image dims arg for frontend bbox

* lint fixes

* lint fixes

* image dimension support

* canvas module added

* comment fix

* async image loading support

* lint fixes

* lint fixes

* max-lines lint fixes

* lint fixes
imatiach-msft added a commit that referenced this pull request Jul 10, 2023
@vinuthakaranth @JarvisG495
@RubyZ10 @natalie-isak @tongyu-microsoft @gaugup @imatiach-msft @Advitya17 @sreeja-g @mehektulsyan @akhil-2709
Individual feature importance interpret QA - part1 (#2167)
5828c21 
* add individual feature importance for QA

temporally use a dropdown to select token

* make QA UI compatible with text classification

temporally using hardcode isQA flag as the prop for identifying qa from an upper level is TBD

* fix left textbox data

fix left textbox importance data and add description for qa interpret

* add isAllDataCohort (#2094)

Signed-off-by: RubyZ10 <zhenzhu@microsoft.com>

* retain selections (#2096)

Signed-off-by: RubyZ10 <zhenzhu@microsoft.com>

* Adding Class Name To OD Dropdown (#2098)

* changing dropdown to include class name

* prettier

* Update erroranalysis-dashboard-regression-superconductor.ipynb (#2102)

* Port over logic of `load_adult_income_dataset` from `dice-ml` to unblock gates (#2106)

* Port over logic of `load_adult_income_dataset` from `dice-ml` to unblock gates

Signed-off-by: Gaurav Gupta <gaugup@microsoft.com>

* Fix linting and add TODO

Signed-off-by: Gaurav Gupta <gaugup@microsoft.com>

* Add fixture for dataset fetch

Signed-off-by: Gaurav Gupta <gaugup@microsoft.com>

---------

Signed-off-by: Gaurav Gupta <gaugup@microsoft.com>

* refactor cohortlist (#2103)

Signed-off-by: RubyZ10 <zhenzhu@microsoft.com>

* Fix dimension mismatch error when dropped_features contains cat_features (#2099)

* fix error when dropped_features contains cat_features

* fix lint

* add tests

* add tests

* fix UT error

* Add more dataset utilities in `rai_test_utils` (#2101)

Signed-off-by: Gaurav Gupta <gaugup@microsoft.com>

* release erroranalysis 0.4.4 (#2105)

* update responsibleai and raiwidgets to erroranalysis 0.4.4 (#2107)

* enable confusion matrix for binary and multiclass text and vision RAI dashboards (#2108)

* release raiwidgets and responsibleai 0.28.0 (#2109)

* Release `rai-test-utils` 0.3.0 (#2110)

* Update rai_test_utils in erroranalysis and responsibleai to 0.3.0 (#2111)

* Update rai_test_utils in erroranalysis and responsibleai to 0.3.0

* Update requirements-dev.txt

* update econml to latest 0.14.1 release (#2112)

* causal fix (#2113)

Signed-off-by: vinutha karanth <vinutha.karanth@gmail.com>

* [Image Explorer] CanvasTools Image Loading support for Object Detection (#2097)

* bbox vott template ckpt

* canvastools image load ckpt

* loading image from scratch ckpt

* canvastools image loading support

* coordinate fixes

* regiondata call + path fix

* code cleanup

* image loading ckpt

* callback image loading support

* lint fixes

* disabling internal imports for vott

* file refactor lint fix

* auto lint fixes

* lint fixes

* image dims arg for frontend bbox

* lint fixes

* lint fixes

* image dimension support

* canvas module added

* comment fix

* async image loading support

* lint fixes

* lint fixes

* max-lines lint fixes

* lint fixes

* Add columns names to `create_multiclass_classification_dataset` (#2114)

Signed-off-by: Gaurav Gupta <gaugup@microsoft.com>

* [Object Detection] [Image Explorer] Bounding Box Support with CanvasTools on Flyout (#2116)

* bbox vott template ckpt

* canvastools image load ckpt

* loading image from scratch ckpt

* canvastools image loading support

* coordinate fixes

* regiondata call + path fix

* code cleanup

* image loading ckpt

* callback image loading support

* lint fixes

* bounding box working ckpt for fridge

* disabling internal imports for vott

* freeze rough ckpt

* file refactor lint fix

* auto lint fixes

* freeze cleanup

* lint fixes

* image dims arg for frontend bbox

* lint fixes

* lint fixes

* image dimension support

* imagedim support comments

* image dimension support

* image dimension support

* canvas module added

* image dimension working ckpt

* comment fix

* cleanup

* async image loading support

* lint fixes

* lint fixes

* reverted max line override

* code cleanups

* lint fixes

* auto lint fixes

* comment fixes

* add ResponsibleAI Text package as initial commit

* update responsibleai-text package to responsibleai v0.25.0

* add model-agnostic explanation support for wrapped models with simple text tokenizer (#1)

* add build and release github actions to responsibleai-text package (#2)

* Enable error analysis for QA (#5)

* add error analysis for QA model

* validate github actions, fix linting and builds (#4)

* added question type feature extractor (#6)

Added regex based question type feature extractor

---------

Co-authored-by: Jarvis Gao <jarvisg495@gmail.com>
Co-authored-by: Jarvis Gao <86736305+JarvisG495@users.noreply.github.com>

* remove train data parameter from RAITextInsights (#7)

* release responsibleai-text v0.1.0 (#8)

* cache error analysis predictions for RAITextInsights to optimize performance when retraining tree or creating new matrix (#9)

* update responsibleai-text package to responsibleai v0.27.0 (#11)

* release responsibleai-text v0.1.1 (#12)

* Add QA feature context overlap and improve question type regex (#14)

* Update .gitignore

* Improve regex matching

modify regex to cover more cases

* add feature context_overlap

* add unit tests for QA feature extractors

* fix CI

* fix test case

* fix task type for multilabel and multiclass classification when passed to UX (#15)

* release responsibleai-text 0.1.2 (#16)

* Python backend metrics for question answering scenario (#17)

* Fixed metrics and added bertscore

* Update setup.py to add extras for qa metrics

* Update CI-python.yml to install qa extras

* fix linting error

* linting fixup

* fix lint

* fix flake8 linting

* fix install command in CI

---------

Co-authored-by: akhil-2709 <aanthireddy@umass.edu>
Co-authored-by: Neutrinowo <86736305+JarvisG495@users.noreply.github.com>

* added-syntactic-complexity (#18)

* added-syntactic-complexity-feature

Added syntactic complexity features, i.e., average and maximum parse tree depth for both Context and Questions.

* fixed-lint

* add metadata support to RAITextInsights (#19)

* fix feature metadata serialization logic in RAITextInsights which was causing exceptions in error analysis (#22)

* add text_column parameter to RAITextInsights to allow user to specify the text columns to differentiate from metadata (#24)

* release responsibleai-text 0.1.3 (#21)

* fix responsibleai-text linting and move directory to
responsible-ai-toolbox

* Add loading spinner for causal policy tab (#2121)

* Add loading spinner for causal policy tab

Signed-off-by: Gaurav Gupta <gaugup@microsoft.com>

* Address code review comments

Signed-off-by: Gaurav Gupta <gaugup@microsoft.com>

---------

Signed-off-by: Gaurav Gupta <gaugup@microsoft.com>

* Refactor- Add datasetCohorts to ModelAssessementContext && create datasetCohort && add compositeFilter in DatasetCohort (#2119)

* refactor

Signed-off-by: RubyZ10 <zhenzhu@microsoft.com>

* address comments

Signed-off-by: RubyZ10 <zhenzhu@microsoft.com>

---------

Signed-off-by: RubyZ10 <zhenzhu@microsoft.com>

* remove console log (#2125)

Signed-off-by: RubyZ10 <zhenzhu@microsoft.com>

* Update msdata-rai-widgets-onelocbuild.yml for Azure Pipelines

* fix build failures due to lint errors for onelocbuild yml (#2137)

* Skipping python 3.7 tests on MACOS (#2129)

* [WIP] Debugging _bz2 failures

* Update CI-python.yml

* Update CI-python.yml

* Update CI-python.yml

* Update CI-python.yml

* Update CI-python.yml

* Update CI-python.yml

* `rai_test_utils`: Convert all namespaces in `__all__` list as strings (#2128)

Signed-off-by: Gaurav Gupta <gaugup@microsoft.com>

* User Flows / Architectures

* [Object Detection] DFD Documentation (#2138)

* dfd

* yarn lint

* [Object Detection] Explanation Documentation (#2136)

* docs for expl

* fixing name

* docs correcting image

* yarn lintfix

* Update docs/explanation-object-detection-README.md

Co-authored-by: Advitya Gemawat <agemawat@microsoft.com>

* removing dpv2

---------

Co-authored-by: Advitya Gemawat <agemawat@microsoft.com>

* CanvasTools documentation

* isQA as prop

Signed-off-by: vinutha karanth <vinutha.karanth@gmail.com>

* update and refine

Signed-off-by: vinutha karanth <vinutha.karanth@gmail.com>

* remove state isQA

Signed-off-by: vinutha karanth <vinutha.karanth@gmail.com>

* clean

Signed-off-by: vinutha karanth <vinutha.karanth@gmail.com>

---------

Signed-off-by: RubyZ10 <zhenzhu@microsoft.com>
Signed-off-by: Gaurav Gupta <gaugup@microsoft.com>
Signed-off-by: vinutha karanth <vinutha.karanth@gmail.com>
Co-authored-by: Neutrinowo <86736305+JarvisG495@users.noreply.github.com>
Co-authored-by: Ruby Zhu <76190371+RubyZ10@users.noreply.github.com>
Co-authored-by: natalie-isak <123979655+natalie-isak@users.noreply.github.com>
Co-authored-by: tongy-msft <91754176+tongyu-microsoft@users.noreply.github.com>
Co-authored-by: Gaurav Gupta <47334368+gaugup@users.noreply.github.com>
Co-authored-by: Ilya Matiach <ilmat@microsoft.com>
Co-authored-by: Advitya Gemawat <agemawat@microsoft.com>
Co-authored-by: Sreeja Gaddamidi <37956427+sreeja-g@users.noreply.github.com>
Co-authored-by: Jarvis Gao <jarvisg495@gmail.com>
Co-authored-by: mehektulsyan <31842027+mehektulsyan@users.noreply.github.com>
Co-authored-by: akhil-2709 <aanthireddy@umass.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@code-vicar code-vicar code-vicar left review comments

@imatiach-msft imatiach-msft imatiach-msft approved these changes

@natalie-isak natalie-isak natalie-isak approved these changes

@gaugup gaugup Awaiting requested review from gaugup gaugup is a code owner

@tongyu-microsoft tongyu-microsoft Awaiting requested review from tongyu-microsoft tongyu-microsoft is a code owner

@xuke444 xuke444 Awaiting requested review from xuke444 xuke444 is a code owner

@RubyZ10 RubyZ10 Awaiting requested review from RubyZ10 RubyZ10 is a code owner

@vinuthakaranth vinuthakaranth Awaiting requested review from vinuthakaranth vinuthakaranth is a code owner

@hawestra hawestra Awaiting requested review from hawestra

@romanlutz romanlutz Awaiting requested review from romanlutz

@kicha0 kicha0 Awaiting requested review from kicha0 kicha0 is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Advitya17 @natalie-isak @codecov-commenter @code-vicar @imatiach-msft