Skip to content

Commit

Permalink
Release v2.5.2 (cvat-ai#6570)
Browse files Browse the repository at this point in the history 
## \[2.5.2\] - 2023-07-27

### Added

- We've added support for multi-line text attributes
(<cvat-ai#6458>)
- You can now set a default attribute value for SELECT, RADIO types on
UI
  (<cvat-ai#6474>)
- \[SDK\] `cvat_sdk.datasets`, is now available, providing a
framework-agnostic alternative to `cvat_sdk.pytorch`
  (<cvat-ai#6428>)
- We've introduced analytics for Jobs, Tasks, and Project
(<cvat-ai#6371>)

### Changed

- \[Helm\] In Helm, we've added a configurable default storage option to
the chart (<cvat-ai#6137>)

### Removed

- \[Helm\] In Helm, we've eliminated the obligatory use of hardcoded
traefik ingress (<cvat-ai#6137>)

### Fixed

- Fixed an issue with calculating the number of objects on the
annotation view when frames are deleted
  (<cvat-ai#6493>)
- \[SDK\] In SDK, we've fixed the issue with creating attributes with
blank default values
  (<cvat-ai#6454>)
- \[SDK\] We've corrected a problem in SDK where it was altering input
data in models (<cvat-ai#6455>)
- Fixed exporting of hash for shapes and tags in a specific corner case
(<cvat-ai#6517>)
- Resolved the issue where 3D jobs couldn't be opened in validation mode
(<cvat-ai#6507>)
- Fixed SAM plugin (403 code for workers in organizations)
(<cvat-ai#6514>)
- Fixed the issue where initial frame from query parameter was not
opening specific frame in a job
  (<cvat-ai#6506>)
- Corrected the issue with the removal of the first keyframe
(<cvat-ai#6494>)
- Fixed the display of project previews on small screens and updated
stylelint & rules (<cvat-ai#6551>)
- Implemented server-side validation for attribute specifications
  (<cvat-ai#6447>)
- \[API\] Fixed API issue related to file downloading failures for
filenames with special characters
(<cvat-ai#6492>)
- \[Helm\] In Helm, we've resolved an issue with multiple caches
in the same RWX volume, which was preventing db migration from starting
(<cvat-ai#6137>)

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Anastasia Yasakova <yasakova.an@gmail.com>
Co-authored-by: yasakova-anastasia <anastasia@cvat.ai>
Co-authored-by: Roman Donchenko <roman@cvat.ai>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Boris Sekachev <boris.sekachev@yandex.ru>
Co-authored-by: Maxim Zhiltsov <zhiltsov.max35@gmail.com>
Co-authored-by: Kirill Sizov <kirill.sizov@cvat.ai>
Co-authored-by: Nikita Manovich <nikita@cvat.ai>
Co-authored-by: Mariia Acoca <39969264+mdacoca@users.noreply.github.com>
Co-authored-by: Kirill Lakhov <kirill.9992@gmail.com>
Co-authored-by: Michael Kirpichev <mkirpic+github@gmail.com>
Co-authored-by: Michael Kirpichev <m.kirpichev@haut.ai>
Co-authored-by: Boris Sekachev <boris@cvat.ai>
  • Loading branch information
@azhavoro @yasakova-anastasia
@SpecLad @dependabot @bsekachev @zhiltsov-max @nmanovic @mdacoca @klakhov @Keramblock @PMazarovich
14 people authored and PMazarovich committed Aug 15, 2023
1 parent 63283a4 commit dc701dd
Showing 1 changed file with 232 additions and 0 deletions.
232 changes: 232 additions & 0 deletions site/content/en/docs/manual/advanced/annotation-quality.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,232 @@
---
title: 'Annotation quality & Honeypot'
linkTitle: 'Annotation quality'
weight: 14
description: 'How to check the quality of annotation in CVAT'
---

In CVAT, it's possible to evaluate the quality of annotation through
the creation of a **Ground truth** job, referred to as a Honeypot.
To estimate the task quality, CVAT compares all other jobs in the task against the
established **Ground truth** job, and calculates annotation quality
based on this comparison.

> **Note** that quality estimation only supports
> 2d tasks. It supports all the annotation types except 2d cuboids.
> **Note** that tracks are considered separate shapes
> and compared on a per-frame basis with other tracks and shapes.
See:

- [Ground truth job](#ground-truth-job)
- [Managing Ground Truth jobs: Import, Export, and Deletion](#managing-ground-truth-jobs-import-export-and-deletion)
- [Import](#import)
- [Export](#export)
- [Delete](#delete)
- [Assessing data quality with Ground truth jobs](#assessing-data-quality-with-ground-truth-jobs)
- [Quality data](#quality-data)
- [Annotation quality settings](#annotation-quality-settings)
- [GT conflicts in the CVAT interface](#gt-conflicts-in-the-cvat-interface)
- [Annotation quality \& Honeypot video tutorial](#annotation-quality--honeypot-video-tutorial)

## Ground truth job

A **Ground truth** job is a way to tell CVAT where to store
and get the "correct" annotations for task quality estimation.

To estimate task quality, you need to
create a **Ground truth** job in the task,
and annotate it. You don’t need to
annotate the whole dataset twice,
the annotation quality of a small part of
the data shows the quality of annotation for
the whole dataset.

For the quality assurance to function correctly, the **Ground truth** job must
have a small portion of the task frames and the frames must be chosen randomly.
Depending on the dataset size and task complexity,
**5-15% of the data is typically good enough** for quality estimation,
while keeping extra annotation overhead acceptable.

For example, in a typical **task with 2000 frames**, selecting **just 5%**,
which is 100 extra frames to annotate, **is enough** to estimate the
annotation quality. If the task contains **only 30 frames**, it's advisable to
select **8-10 frames**, which is **about 30%**.

It is more than 15% but in the case of smaller datasets,
we need more samples to estimate quality reliably.

To create a **Ground truth** job, do the following:

1. Create a [task](/docs/manual/basics/create_an_annotation_task/), and open the task page.
2. Click **+**.

![Create job](/images/honeypot01.jpg)

3. In the **Add new job** window, fill in the following fields:

![Add new job](/images/honeypot02.jpg)

- **Job type**: Use the default parameter **Ground truth**.
- **Frame selection method**: Use the default parameter **Random**.
- **Quantity %**: Set the desired percentage of frames for the **Ground truth** job.
<br>**Note** that when you use **Quantity %**, the **Frames** field will be autofilled.
- **Frame count**: Set the desired number of frames for the "ground truth" job.
<br>**Note** that when you use **Frames**, the **Quantity %** field will be will be autofilled.
- **Seed**: (Optional) If you need to make the random selection reproducible, specify this number.
It can be any integer number, the same value will yield the same random selection (given that the
frame number is unchanged). <br> **Note** that if you want to use a
custom frame sequence, you can do this using the server API instead,
see [Jobs API #create](https://opencv.github.io/cvat/docs/api_sdk/sdk/reference/apis/jobs-api/#create).

4. Click **Submit**.
5. Annotate frames, save your work.
6. Change the status of the job to **Completed**.
7. Change **Stage** to **Accepted**.

The **Ground truth** job will appear in the jobs list.

![Add new job](/images/honeypot03.jpg)

## Managing Ground Truth jobs: Import, Export, and Deletion

Annotations from **Ground truth** jobs are not included in the dataset export,
they also cannot be imported during task annotations import
or with automatic annotation for the task.

Import, export, and delete options are available from the
job's menu.

![Add new job](/images/honeypot04.jpg)

### Import

If you want to import annotations into the **Ground truth** job, do the following.

1. Open the task, and find the **Ground truth** job in the jobs list.
2. Click on three dots to open the menu.
3. From the menu, select **Import annotations**.
4. Select import format, and select file.
5. Click **OK**.

> **Note** that if there are imported annotations for the frames that exist in the task,
> but are not included in the **Ground truth** job, they will be ignored.
> This way, you don't need to worry about "cleaning up" your **Ground truth**
> annotations for the whole dataset before importing them.
> Importing annotations for the frames that are not known in the task still raises errors.
### Export

To export annotations from the **Ground truth** job, do the following.

1. Open the task, and find a job in the jobs list.
2. Click on three dots to open the menu.
3. From the menu, select **Export annotations**.

### Delete

To delete the **Ground truth** job, do the following.

1. Open the task, and find the **Ground truth** job in the jobs list.
2. Click on three dots to open the menu.
3. From the menu, select **Delete**.

## Assessing data quality with Ground truth jobs

Once you've established the **Ground truth** job, proceed to annotate the dataset.

CVAT will begin the quality comparison between the annotated task and the
**Ground truth** job in this task once it is finished (on the `acceptance` stage and in the `completed` state).

> **Note** that the process of quality calculation may take up to several hours, depending on
> the amount of data and labeled objects, and is **not updated immediately** after task updates.
To view results go to the **Task** > **Actions** > **View analytics**> **Performance** tab.

![Add new job](/images/honeypot05.jpg)

### Quality data

The Analytics page has the following fields:

<!--lint disable maximum-line-length-->

| Field | Description |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Mean annotation quality | Displays the average quality of annotations, which includes: the count of accurate annotations, total task annotations, ground truth annotations, accuracy rate, precision rate, and recall rate. |
| GT Conflicts | Conflicts identified during quality assessment, including extra or missing annotations. Mouse over the **?** icon for a detailed conflict report on your dataset. |
| Issues | Number of [opened issues](/docs/manual/advanced/review/). If no issues were reported, will show 0. |
| Quality report | Quality report in JSON format. |
| Ground truth job data | "Information about ground truth job, including date, time, and number of issues. |
| List of jobs | List of all the jobs in the task |

<!--lint enable maximum-line-length-->

### Annotation quality settings

If you need to tweak some aspects of comparisons, you can do this from
the **Annotation Quality Settings** menu.

You can configure what overlap
should be considered low or how annotations must be compared.

The updated settings will take effect
on the next quality update.

To open **Annotation Quality Settings**, find
**Quality report** and on the right side of it, click on
three dots.

The following window will open.
Hover over the **?** marks to understand what each field represents.

![Add new job](/images/honeypot08.jpg)

Annotation quality settings have the following parameters:

<!--lint disable maximum-line-length-->

| Field | Description |
| ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Min overlap threshold | Min overlap threshold(IoU) is used for the distinction between matched / unmatched shapes. |
| Low overlap threshold | Low overlap threshold is used for the distinction between strong/weak (low overlap) matches. |
| OKS Sigma | IoU threshold for points. The percent of the box area, used as the radius of the circle around the GT point, where the checked point is expected to be. |
| Relative thickness (frame side %) | Thickness of polylines, relative to the (image area) ^ 0.5. The distance to the boundary around the GT line inside of which the checked line points should be. |
| Check orientation | Indicates that polylines have direction. |
| Min similarity gain (%) | The minimal gain in the GT IoU between the given and reversed line directions to consider the line inverted. Only useful with the Check orientation parameter. |
| Compare groups | Enables or disables annotation group checks. |
| Min group match threshold | Minimal IoU for groups to be considered matching, used when the Compare groups are enabled. |
| Check object visibility | Check for partially-covered annotations. Masks and polygons will be compared to each other. |
| Min visibility threshold | Minimal visible area percent of the spatial annotations (polygons, masks) |
| For reporting covered annotations, useful with the Check object visibility option. |
| Match only visible parts | Use only the visible part of the masks and polygons in comparisons. |

<!--lint enable maximum-line-length-->

### GT conflicts in the CVAT interface

To see GT Conflicts in the CVAT interface, go to **Review** >
**Issues** > **Show ground truth annotations and conflicts**.

![GT conflict](/images/honeypot06.gif)

The ground truth (GT) annotation is depicted as
a dotted-line box with an associated label.

Upon hovering over an issue on the right-side panel with your mouse,
the corresponding GT Annotation gets highlighted.

Use arrows in the Issue toolbar to move between GT conflicts.

To create an issue related to the conflict,
right-click on the bounding box and from the
menu select the type of issue you want to create.

![GT conflict](/images/honeypot07.jpg)

## Annotation quality & Honeypot video tutorial

This video demonstrates the process:

<iframe width="560" height="315" src="https://www.youtube.com/embed/0HtBnr_CZAM" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

0 comments on commit dc701dd

Please sign in to comment.