Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Semi-automatic tools enhancements (Non-blocking UI, tips) #3473

Merged
merged 40 commits into from
Aug 6, 2021

Conversation

bsekachev
Copy link
Member

@bsekachev bsekachev commented Jul 28, 2021

Motivation and context

Related #2936
Resolved #2515

Peek 2021-08-02 12-25
Screenshot from 2021-08-02 12-26-25

How has this been tested?

Manual testing

Checklist

License

  • I submit my code changes under the same MIT License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below)
# Copyright (C) 2021 Intel Corporation
#
# SPDX-License-Identifier: MIT

@bsekachev bsekachev added the enhancement New feature or request label Jul 28, 2021
ActiveChooN
ActiveChooN previously approved these changes Aug 4, 2021
@nmanovic
Copy link
Contributor

nmanovic commented Aug 4, 2021

@bsekachev , I like the experience, especially with DEXTR. But I'm missing a feature. When I'm annotating using DEXTR, it doesn't respect "selected opacity" parameter. In some cases it is really difficult to annotate if opacity is 0 during drawing.

image

@bsekachev
Copy link
Member Author

@bsekachev , I like the experience, especially with DEXTR. But I'm missing a feature. When I'm annotating using DEXTR, it doesn't respect "selected opacity" parameter. In some cases it is really difficult to annotate if opacity is 0 during drawing.

image

Added feature: "Selected opacity" slider now defines opacity level of shapes being drawn (works when draw shapes, or work with interactors | trackers)

@nmanovic nmanovic merged commit 59af610 into develop Aug 6, 2021
@nmanovic nmanovic deleted the bs/semi_enhancements_1 branch August 6, 2021 12:04
@bsekachev
Copy link
Member Author

bsekachev commented Aug 6, 2021

@aschernov @TOsmanov

Could you please update the user guide about using interactors according to new changes (non-blocking UI, Selected opacity slider)? Contact @azhavoro to get a CVAT instance with these changes.

Also, we expect a feedback from the DA team about the previous PR (with points minimizer) and this PR, since they were implemented to satisfy your requests.

@dvkruchinin

Could you please prepare a test for:

  • Added feature: "Selected opacity" slider now defines opacity level of shapes being drawn (probably we already have a test with this slider, better to update it)

@aschernov
Copy link
Contributor

@bsekachev , we will update the documentation.
Also, we will prepare a feedback about the implementations you mentioned.

@aschernov
Copy link
Contributor

@bsekachev , we made one more annotation experiment with another but similar annotation task we used before, and got the following conclusions:
After test annotation, comparing the traditional method (manual) and the semi-automatic (DEXTR) method with the points minimizer, we got the following data:
1
Speed of manual annotation is faster than getting a polygon with a semi-automatic annotation tool + in most cases we have to correct it.
Examples of annotation with DEXTR with and without points minimizer:
In this example, there are inaccuracies in the result of work of the semi-automatic annotation tool. If reduce the number of points with the points minimizer, we will need to correct only 3 points.
2
3
Often, a semi-automatic annotation tool annotated an object badly. A lot of corrections are required, even if we set a lot of points.
4
5
We can also use the minimizer to increase the number of points to get better polygon quality for objects that are well annotated with semi-automatic annotation tool, where no additional corrections are required, if the task requires it. It took 8 seconds to receive a response from the server, excluding the time to set the first 4 points. Annotating such objects could possibly take less time than manual annotating if the request to the server took less time.
6
Some complex objects require a lot of points. Using points minimizer in this case increased the time a bit for correcting the polygon. If the task does not require excessive accuracy, then using points minimizer will spend less time than without it.
7
8
In general, using points minimizer in many cases reduces the time for correcting the polygon created with semi-automatic annotation tool. But manual annotation is faster, mainly this may be related with a long wait for a response from the server. For simple objects, reducing the server response time would bring the speed annotation with a semi-automatic annotation tool closer to manual annotating speed.
Perhaps, absence of blocking UI has made the annotation with a semi-automatic annotation tool a little faster. Blocking UI didn't let additional viewing of the polygon while awaiting a response from the server. Working with a clear and large object, seeing some inaccuracies in the polygon, we can add additional points to those places that were not very well annotated with a semi-automatic annotation tool. We can also see an incorrectly placed point and we can remove it without waiting for a response from the server, this is convenient.

@nmanovic
Copy link
Contributor

@aschernov , can we try the same experiments on objects like persons? Each DL model is trained on a specific dataset. DEXTR was trained on Pascal or COCO. It means that the model didn't see classes which you try to annotate. This is why results are extremely bad (my guess). Probably it should be mentioned in documentation. When we are able to retrain the model on the fly, it makes sense to come back to experiments which were described in the comment above.

@aschernov
Copy link
Contributor

@nmanovic , sure. We'll prepare a corresponding test task and repeat the experiment. I'll let you know about the results.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CVAT AI Tools: dextr usage
4 participants