Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/make sv.mask annotator as fast as annotators in popular detection frameworks #472 #606

Conversation

xenteros
Copy link
Contributor

@xenteros xenteros commented Nov 16, 2023

Description

  • Combined all colored masks before applying them to the scene
  • Made sure that returned ndarray is np.uint8 for video writer simplicity

#472

  • Bug fix (non-breaking change which fixes an issue)

How has this change been tested, please provide a testcase or example of how you tested the change?

I profiled the code with VizTracer and compared outputs frame by frame and they are equal.

* Combined all colored masks before applying them to the scene
* Made sure that returned ndarray is np.uint8 for video writer simplicity
@CLAassistant
Copy link

CLAassistant commented Nov 16, 2023

CLA assistant check
All committers have signed the CLA.

@SkalskiP SkalskiP self-requested a review November 17, 2023 14:06
@SkalskiP SkalskiP added enhancement New feature or request api:annotator Annotators labels Nov 17, 2023
@SkalskiP
Copy link
Collaborator

Hi, @xenteros! 👋🏻 Thanks for your interest in Supervision. I'm benchmarking your solution right now.

@SkalskiP
Copy link
Collaborator

Hi @xenteros 👋🏻, I've been busy making tests of this change. Here is what I got:

  1. Consistency of obtained results
  • develop

download - 2023-11-17T160729 509

  • xenteros:feature/Make_sv.MaskAnnotator_as_fast_as_annotators_in_popular_detection_frameworks_#472

download - 2023-11-17T152113 491

  1. Drawing speed

Annotation speed depends on external factors - other processes running on the machine and varies between test runs. To validate the results, I performed 5 tests with the code in the current develop and 5 with the code from xenteros:feature/Make_sv.MaskAnnotator_as_fast_as_annotators_in_popular_detection_frameworks_#472. I averaged the results.

  • develop

    reference (only inference) inference + box annotation inference + mask annotation
    123 sec 125 sec 298 sec
    125 sec 125 sec 300 sec
    124 sec 125 sec 301 sec
    124 sec 125 sec 299 sec
    124 sec 125 sec 301 sec
    124 sec (average) 125 sec (average) 300 sec (average)

    Annotation overhead: 300 - 124 = 176
    Per frame annotation overhead: 350 / 341 = 0.51
    Google Colab - link

  • xenteros:feature/Make_sv.MaskAnnotator_as_fast_as_annotators_in_popular_detection_frameworks_#472

    reference (only inference) inference + box annotation inference + mask annotation
    125 sec 125 sec 159 sec
    124 sec 125 sec 159 sec
    124 sec 125 sec 159 sec
    124 sec 125 sec 162 sec
    124 sec 126 sec 161 sec
    124 sec (average) 125 sec (average) 160 sec (average)

    Annotation overhead: 160 - 124 = 36
    Per frame annotation overhead: 36 / 341 = 0.11
    Google Colab - link

  1. Summary:
  • Results are different - in the case of overlapping masks, we see only the one that was applied to the area last
  • Updated MaskAnnotator is 5x faster

@SkalskiP
Copy link
Collaborator

Speed comparison across all three MaskAnnotator versions

MaskAnnotator_Chart (1)

@SkalskiP
Copy link
Collaborator

LGTM. Merging! @xenteros, thanks a lot for your contribution. This annotator was a big performance bottleneck.

@SkalskiP SkalskiP merged commit 145313a into roboflow:develop Nov 17, 2023
6 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api:annotator Annotators enhancement New feature or request
Projects
Status: Current Release: Done
Development

Successfully merging this pull request may close these issues.

Make sv.MaskAnnotator as fast as annotators in popular detection frameworks
3 participants