Shot dataset: create annotation aggregation script (consider annotator agreement etc)

[tyiannak]

No description provided.

[tyiannak]

@apoman38 @electrasif
Create a new script e.g. in shot_annotation/aggregate_annotations.py

Given the annotation csv this script should:
a) get generic statistics: e.g.
- 10 annotators

• 1921 annotations in total
• 500 videos in the annotation pool
• 101 videos annotated by 1 user
• 221 videos annotated by 2 user2
• ....
  b) aggregate to FINAL annotations: (a) read the CSV (b) map each file (shot) to a final (aggregated decision) and respective confidence. E.g. video 124 has been annotated by thod, ilektra, elektra, kostas with the following annotations: zoom in, zoom in, N/A, zoom in. Then, video 124 will be mapped to , confidence:0.75

[apoman38]

Create a new script aggregate_annotations.py
Let me explain the problems I encountered while implementing the code. The annotations_database.txt file in the first line has no delimiter between words and pandas did not recognize them so I added comma. In addition the N / A class is recognized by the pandas as NaN and so in the final file created later (I will explain) it does not exist as a class. So in order to create the script I renamed NaN to Nothing. In the folder there are 2 versions of the file where V1 the old and V2 the new.
The code prints:

1. Total number of annotations
2. Annotations of every user
3. Create a file detailed_confidence.csv. Contains in detail the confidence of each video to which class it belongs
4. Create a file confidence.txt. It was created to be more readable and contains the class to which each video belongs

For sure the script must be update. Please express your ideas for how many files must be created (more,less?) and bags that may be in the code that affect the end result.

[tyiannak]

@apoman38 thanx for the update
I did not undarstand exactly what is the problem with the annotations_database.txt file, why can't we just define the delimiter while uploading the csv file?
Also, have you tested that the final annotation actually works? (you could test by making some "artificiall" annotations for specific files)

[apoman38]

The file has been updated, the file seems to be working properly without any incorrect results.
See here " #20 (comment) " for details about the output files.
I created a new file hello.txt with false annotations and i added one specific video to many classes to see if the confidence works as expected. Please run the code with hello.txt file to see the results.