Questions about evaluation indicators

[Bin-ze]

I downloaded the UAVDT data set and evaluation tool, but I did not find out how to use it to generate test results from the tool. I did a lot of searches, and there are no similar tutorials on the Internet. I searched the original paper and did not find the author. Contact information.
But I really need the same evaluation criteria as the open source papers to evaluate the quality of my network. You published your paper with this data set. I wonder if you can send me the testing tools you use. I want to use this to evaluate your own network.

[hu64]

Hi, after some research, I could find the dataset page. Maybe it changed location since I used it? I don't know. Anyway, it is there now: https://sites.google.com/view/grli-uavdt. Tell me if you have any problems accessing it. If you scroll to the bottom of the page, the evaluation tool is the link titled "[DET/MOT toolkit (Google Drive)]", at this URL: (https://drive.google.com/file/d/19498uJd7T9w4quwnQEy62nibt3uyT9pq/view. After that, you can see that the testing set is defined in the file utils/CalculateDetectionPR_overall.m. Good luck!

[Bin-ze]

I did download the tool in that data set and run the MATLAB file. I don't know how to evaluate my network using the matlab tool. There seems to be no similar tutorial on the Internet. I'm really sorry. I may not have described it clearly before. I'm sorry.

…

在 2021年12月1日，上午10:06，Hughes Perreault ***@***.***> 写道：  Hi, after some research, I could find the dataset page. Maybe it changed location since I used it? I don't know. Anyway, it is there now: https://sites.google.com/view/grli-uavdt. Tell me if you have any problems accessing it. If you scroll to the bottom of the page, the evaluation tool is the link titled "[DET/MOT toolkit (Google Drive)]", at this URL: (https://drive.google.com/file/d/19498uJd7T9w4quwnQEy62nibt3uyT9pq/view. After that, you can see that the testing set is defined in the file utils/CalculateDetectionPR_overall.m. Good luck! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[Bin-ze]

Do you use read_csv to generate the test text file on the entire data set? I see that such a function is written in your code. So can I copy this part of your code to generate the text file needed for testing. Can you tell me which of these is the script you used to generate the txt file in the end? Also, can I get so many txt files directly by running this script?

[hu64]

Ah okay I see... You have to run the file CalculateDetectionsPR. I suggest you try to run results for Faster R-CNN first, as it is the one that's in the file by default. You can change the name at line 2 in this file, and it will change the folder that is evaluated in "RES_DET". Also, if you have trouble, and you only want the overall results, you can comment the other lines, like this. If I remember correctly, you should be able to reproduce the results in the original UAVDT paper like that:

%% calculate the AP scores of detectors in terms of different attributes
detection = 'det_FRCNN'; % detector names, i.e., det_FRCNN, det_RFCN, det_RON, det_SSD
CalculateDetectionPR_overall(detection); % calculate the AP scores of the test-set.
% CalculateDetectionPR_seq(detection); % calculate the AP scores in sequence-level attribute

% calculate the AP scores in object-level attribute
% for obj_attr = 1:3
    % 1 for Vehicle Category;
    % 2 for Vehicle Occlusion;
    % 3 for Out-of-view;
    % CalculateDetectionPR_obj(detection, obj_attr);
% end

I hope this helps! If you have any problem about the format of your results, just follow an example at: RES_DET\det_FRCNN\M0203.txt.

[Bin-ze]

/run_on_csv_method.py Is it a script that generates test results? It seems that he should not be able to generate all the txt files, how do you operate？

[hu64]

This script generates test results, but in an incorrect format for this tool. I had another very simple script which I no longer have access to. But really if you check RES_DET\det_FRCNN\M0203.txt, you will see that the format is simple, you can write a conversion script very easily yourself. Some values are unused for object detection evaluation, and are set to 1 or -1.

[Bin-ze]

SPLIT = 'uav-test'

if SPLIT == 'test':
    source_lines = open(os.path.join(DATASET_DIR, 'test-tf-all.csv'), 'r').readlines()
    target_file = open(os.path.join(base_dir, exp_id, 'ua-test.csv'), 'a')
elif SPLIT == 'train1on10':
    source_lines = open(os.path.join(DATASET_DIR, 'train-tf.csv'), 'r').readlines() # + open('/store/datasets/UA-Detrac/val-tf-all.csv', 'r').readlines()
    target_file = open(os.path.join(base_dir, exp_id, 'ua-train1on10.csv'), 'w')
elif SPLIT == 'trainval':
    source_lines = open(os.path.join(DATASET_DIR, 'train-tf-all.csv'), 'r').readlines() + open('/store/datasets/UA-Detrac/val-tf-all.csv', 'r').readlines()
    target_file = open(os.path.join(base_dir, exp_id, 'ua-trainval.csv'), 'w')
elif SPLIT == 'uav-test':
    source_lines = open('/store/datasets/UAV/val.csv', 'r').readlines()
    target_file = open(os.path.join(base_dir, exp_id, 'uav-test.csv'), 'w')
elif SPLIT == 'uav-val':
    source_lines = open('/store/datasets/UAV/val-sub.csv', 'r').readlines()
    target_file = open(os.path.join(base_dir, exp_id, 'uav-val2.csv'), 'w')
elif SPLIT == 'changedetection':
    source_lines = open('/store/datasets/changedetection/changedetection.csv', 'r').readlines()
    target_file = open(os.path.join(base_dir, exp_id, 'changedetection.csv'), 'w')
elif SPLIT == 'ped1':
    source_lines = open('/store/datasets/ped1/csv.csv', 'r').readlines()
    target_file = open(os.path.join(base_dir, exp_id, 'results.csv'), 'w')

Have you saved the path of the picture in a csv file?
Read file used to generate txt file?
So how to generate the txt file needed for matlab tool evaluation?
Are all data used for evaluation?
I'm still not sure?
I only need to know how to generate the txt file needed for matlab tool testing to realize the evaluation of my network.

I'm really sorry to disturb you, but it is still difficult for me to understand the generation logic of a series of txt files and what kind of data it uses only from your code.
If it is convenient, can you give me your contact information, wecat or Telegram or other, I only have a little question? But it seems difficult to say clearly in the mail?

[Bin-ze]

I can write my own conversion tool. It is really simple.
But do I need to use all the data in the results I generate? Including the pictures of each frame and even the data used in training. If this is the case, then I think I understand.

[hu64]

For my research I converted all the datasets in the same simple CSV format, which is:

path_to_file, x0, y0, x1, y1, label

and then when I generate the results:

path_to_file, x0, y0, x1, y1, label, score

I later use a custom script to reconvert each dataset into their own format for evaluation. The advantage for me is that all my data is in the same format during the work.

[hu64]

Actually I found the script for UAVDT to CSV, here it is: https://1drv.ms/u/s!Ahf1HD3oV-4xgtwyi3vRFiAE5lwBbA?e=FRiIus. Hope this helps!

[Bin-ze]

First of all, thank you for your help, I have solved the previous problem.
However, new problems have emerged. I used centernet as my baseline to train on this data set, using coco indicators, but I found that the indicators change very strangely during training. Loss decreases with training, but the accuracy of the network on the test set gets worse with training, which is far from the results in your paper. I did not use the cneternet you used, but I am sure that there is no problem with my network, and it can get normal results on the coco data set.
I used the ignore.txt file of the dataset to process the data, because I think that unlabeled instances may cause instability in network training. I don't know what kind of treatment you have done?
I currently have no way to get a relatively normal baseline indicator. It makes it difficult for me to continue my work. Do you have any good suggestions?
For small goals, have you made improvements to the official code (it does not seem to be from the paper)?
What index did you use during training, VOC or COCO?