KITTI Results

[CuongNguyen218]

@AlexeyAB ,
Hi Alex. Thanks for your amazing code. But I have a trouble when Training your repo with KITTI Dataset.
When I use your repo and trained around 50k iter, I get a bad result with car AP detection when test with KITTI Devkit. Details in image below.

And my result at final iter with your repo below

And chart image of training process below:

I see a gap between the result of car mAP when I use your repo and GaussianYolo repo is large. Notice that I don't use Gaussian Yolo Layer. Result from Gaussian Yolo repo in the image below

Can you explain it for me ?
This is my config file
yolov3-kitti.txt
:
And this is the number of box of KITTI dataset in training set and test set:

[AlexeyAB]

@CuongNguyen218 Hi,

1. There was a bug in the data augmentation from 26 oct to 4 dec 2019. So you should train it again in this repo.

2. Also what Darknet-repo and what command do you use to get car_detection AP?

[CuongNguyen218]

@AlexeyAB , Hi,
Thank for your quick reply. Darknet repo what i use to reproduce this result below is Gaussian Yolo Repo ( i used yolo layer not gaussian Yolo Layer).

Gaussian Repo: https://github.com/jwchoi384/Gaussian_YOLOv3
and I used official evaluate code from KITTI dataset webpage without any change: http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=2d

[AlexeyAB]

Darknet repo what i use to reproduce this result below is Gaussian Yolo Repo

Try to use this repo: https://github.com/AlexeyAB/darknet

I just added ./darknet detector valid for KITTI and BDD: aaba21b

[CuongNguyen218]

@AlexeyAB ,
thank for your support but I think there is a misunderstanding between us. Let me explain:

1. I trained KITTI dataset with your repo get weight file A. Then, I use weight file A to valid on testset and result of weight A in gaussian repo and get results in image below when test with devkit code.
   The order is E M H left to right
   

I see it is an unreasonable result follow gaussian paper:

2. To check problems, I retrained KITTI dataset with Gaussian Yolov3 Repo and get weight file B (it don't have GIOU value in log file and I don't know that GIOU affects to this result when i use your repo). Then, I used weight B to valid on testset and get results in image below with devkit code.
   
   Results and trend seem good.
   My cmd line is: ./darknet detector valid cfg/datafile cfg/cfgfile weights
   Maybe i will retry with your newest repo. Thanks

[AlexeyAB]

My cmd line is: ./darknet detector valid cfg/datafile cfg/cfgfile weights
Maybe i will retry with your newest repo. Thanks

1. Yes. You should do ./darknet detector valid on the same repo where did you do ./darknet detector train There are different resize approaches in different frameworks with pros and cons: Resizing : keeping aspect ratio, or not #232 (comment)

2. Also there was a bug 26 oct -> 4 dec in training: KITTI Results #4454 (comment)

[CuongNguyen218]

@AlexeyAB ,
I will retrain tomorrow. Thanks you. One more question, I want to use GIOU with Gaussian Yolo Layer as a metrics instead of IOU ( do not use GIOU loss). How can I do that ?

[CuongNguyen218]

@AlexeyAB ,
Hi i want to read about data augment bug. Maybe you prefer wrong comment above?

[AlexeyAB]

I will retrain tomorrow. Thanks you. One more question, I want to use GIOU with Gaussian Yolo Layer as a metrics instead of IOU ( do not use GIOU loss). How can I do that ?

There isn't mAP that is based on GIoU instead of IoU. Try to ask it in GIoU-repository.

Hi i want to read about data augment bug. Maybe you prefer wrong comment above?

Just Bilateral-blurring will be used for 50% of images even if you didn't enable it.

[CuongNguyen218]

@AlexeyAB ,
In #3114 inyolo_v3_tiny_pan5 matrix_gaussian_GIoU aa_ae_mixup_new.cfg , I saw you use GIOU for Gaussian Yolo layer. I think something not correct when see NLL loss in Gaussian Yolo Paper because NLL loss in Gaussian Yolo Layer is loss function for bounding box and GIOU also Bounding Box loss. Can you explain it for me ?
iou_loss=giou

[AlexeyAB]

@CuongNguyen218

GIoU just replaces NLL loss for b-box.
So there will be GIoU-loss for bbox-loss and Gaussian_yolo-loss for Uncertainty.

[CuongNguyen218]

@AlexeyAB ,
Hi. I think something wrong with valid

I retrained KITTI dataset with your new code and your valid code but something when wrong with car class. The trend is Easy > Moderate > Hard but my result what I get from your code is not follow that.
I don't know why ?

[AlexeyAB]

@CuongNguyen218

Also you can try to set in your cfg-file

[net]
letter_box=1

Then train usin AlexeyAB repo, and valid by using Jwchoi repo.

[CuongNguyen218]

@AlexeyAB , Thanks for your support. and i want to find commit of bag of freebie. Can you find it for me?

[AlexeyAB]

Serach commit with MixUp.

or just use

[net]
mixup=1

[CuongNguyen218]

@AlexeyAB , i saw bag_freebies=1 in #3272 . does it equal to mixup = 1 ?

[AlexeyAB]

Yes.

[AlexeyAB]

But you should know, that mixup=1 improves accuracy only for Classifier. (and the use Classifier pre-trained weights file for Detector).

For training Detector is recommended to use mosaic=1. Use the latest version of Darknet,.

[CuongNguyen218]

okay, I got it. Thanks you. I want to try it because i see it improve performance of yolov3 in gluoncv +5% on COCO.
Paper: https://arxiv.org/pdf/1902.04103.pdf
Maybe it included a bag of trick for object detection.

[AlexeyAB]

BoF (Bag of Freebies) includes 5 features - there are implemented 5 of 5 features in this repository, read: #3272 (comment)

There is implemented mixup=1 that gives +1.5 - +2.0% mAP for Yolov3, but:

mixup=1 for Detector gives only +0.3% mAP@0.5...0.95 on MS COCO

Paper: https://arxiv.org/pdf/1902.04103.pdf

---

To get +1.5 - +2.0% mAP you must first train the Classifier with [net] mixup=1 https://github.com/AlexeyAB/darknet/blob/master/cfg/darknet53.cfg and then use this pre-trained weights file for training Detector Yolo with [net] mixup=1 https://github.com/AlexeyAB/darknet/blob/master/cfg/yolov3.cfg then you will get +2% MS COCO

---

To get another +2.0% you should use: mixup=1, SGDR (cosine lr), random=1, and GPU-processing + CPU-RAM - it takes a lot of time to training.

---

There are better approaches than MixUP:

• CutMix: Classifier - CutMix: Regularization Strategy to Train Strong Classifiers #4419 - for Classifier
• Mosaic: Detector - Mosaic data augmentation  #4264 - for Classifier and Detector

[CuongNguyen218]

@AlexeyAB ,
I retrained my model with your repo and setting letter_box = 1 in my cfg file, and bad results for car class still appear.
This is my result:

[AlexeyAB]

What width= and height= do you use in both cfg-files?
What random= do you use in the last [yolo] layer in both cfg-files?

[CuongNguyen218]

Hi @AlexeyAB ,
I set width = height = 512 in both cfg file.
random = 1, ignore_thresh = 0.5 in both.

[AlexeyAB]

I retrained my model with your repo and setting letter_box = 1 in my cfg file, and bad results for car class still appear.

Did you set [net] letter_box=1 in cfg, then train with AlexeyAB, then valid with Jwchoi and get Car AP = 59.56%?

Can you attach both cfg-files?

There is also different network resize approaches:

• AlexeyAB:

  darknet/src/detector.c

  Lines 169 to 171 in 318919e

  	float random_val = rand_scale(1.4); // *x or /x
  	int dim_w = roundl(random_val*init_w / 32 + 1) * 32;
  	int dim_h = roundl(random_val*init_h / 32 + 1) * 32;

• pjreddie that is doesn't depend on the initial network size: https://github.com/pjreddie/darknet/blob/61c9d02ec461e30d55762ec7669d6a1d3c356fb2/examples/detector.c#L65

[CuongNguyen218]

@AlexeyAB

Did you set [net] letter_box=1 in cfg, then train with AlexeyAB, then valid with Jwchoi and get Car AP = 59.56%?

Yes.
Here is my both cfg-files?
yolov3-kiit-jwchoi.txt
yolov3-kitti-AlexeyAB.txt

[AlexeyAB]

Did you train both models with the same batch and subdivisions?

batch=64
subdivisions=16

Try to change these lines:

darknet/src/detector.c

Lines 169 to 171 in 318919e

	float random_val = rand_scale(1.4); // *x or /x
	int dim_w = roundl(random_val*init_w / 32 + 1) * 32;
	int dim_h = roundl(random_val*init_h / 32 + 1) * 32;

To these

            int dim = (rand() % 10 + 10) * 32;
            if (get_current_batch(net)+200 > net->max_batches) dim = 608;
int dim_w = dim;
int dim_h = dim;

recompile and train without letter_box=1

[CuongNguyen218]

Did you train both models with the same batch and subdivisions?

batch=64
subdivisions=16

Yes, I set both are Batch=64 and sub =16
i will retry this with 10000k iter.

[CuongNguyen218]

@AlexeyAB ,
I change these line above and get an error

Maybe related to this line

darknet/src/detector.c

Line 83 in 2a873f3

network net = nets[0];

[AlexeyAB]

Use if (get_current_batch(net) + 200 > net.max_batches) dim = 608; instead of if (get_current_batch(net) + 200 > net->max_batches) dim = 608;

Also removes these lines:

darknet/src/detector.c

Lines 173 to 177 in 318919e

	// at the beginning
	if (avg_loss < 0) {
	dim_w = roundl(1.4*init_w / 32 + 1) * 32;
	dim_h = roundl(1.4*init_h / 32 + 1) * 32;
	}

[CuongNguyen218]

@AlexeyAB

This is new error.

Here is my change in code. Maybe I miss a '(' right?
Sorry i'm not familiar with C

[AlexeyAB]

Use // for commenting instad of #

[CuongNguyen218]

@AlexeyAB , thank you it work. As I feel, training speed is faster.

[CuongNguyen218]

@AlexeyAB ,
I trained my model with 10k iter and finally it works.

Thank you for your support.

[err-or]

Hi, if you are using ./darknet detector valid for kitti, what do you use for eval= in obj,data? If you are using the devkit to evaluate yolo for kitti, can you please tell me how you are using the devkit? Thanks!

[AlexeyAB]

Use eval=kitti in the obj.data file.

Can be: voc, coco, imagenet, kitty, bdd:

darknet/src/detector.c

Lines 662 to 708 in 49bff0e

	char *type = option_find_str(options, "eval", "voc");
	FILE *fp = 0;
	FILE **fps = 0;
	int coco = 0;
	int imagenet = 0;
	int bdd = 0;
	int kitti = 0;
	if (0 == strcmp(type, "coco")) {
	if (!outfile) outfile = "coco_results";
	snprintf(buff, 1024, "%s/%s.json", prefix, outfile);
	fp = fopen(buff, "w");
	fprintf(fp, "[\n");
	coco = 1;
	}
	else if (0 == strcmp(type, "bdd")) {
	if (!outfile) outfile = "bdd_results";
	snprintf(buff, 1024, "%s/%s.json", prefix, outfile);
	fp = fopen(buff, "w");
	fprintf(fp, "[\n");
	bdd = 1;
	}
	else if (0 == strcmp(type, "kitti")) {
	char buff2[1024];
	if (!outfile) outfile = "kitti_results";
	printf("%s\n", outfile);
	snprintf(buff, 1024, "%s/%s", prefix, outfile);
	int mkd = make_directory(buff, 0777);
	snprintf(buff2, 1024, "%s/%s/data", prefix, outfile);
	int mkd2 = make_directory(buff2, 0777);
	kitti = 1;
	}
	else if (0 == strcmp(type, "imagenet")) {
	if (!outfile) outfile = "imagenet-detection";
	snprintf(buff, 1024, "%s/%s.txt", prefix, outfile);
	fp = fopen(buff, "w");
	imagenet = 1;
	classes = 200;
	}
	else {
	if (!outfile) outfile = "comp4_det_test_";
	fps = (FILE**) xcalloc(classes, sizeof(FILE *));
	for (j = 0; j < classes; ++j) {
	snprintf(buff, 1024, "%s/%s%s.txt", prefix, outfile, names[j]);
	fps[j] = fopen(buff, "w");
	}
	}

[err-or]

Hi Alexey, thank you for your response. I did what you said but I get an error while running ./darknet detector valid where eval = kitti in obj.data. The error is this:

To give you an idea of what I did:

1. I downloaded the training images of KITTI 2D object detection
2. I have converted the labels according to yolo
3. I have taken around 6k images as train.txt
4. The remaining images are in valid.txt
5. I trained the model up to 16k iterations

Can you please tell me how can I evaluate the model according to kitti 2D detection benchmark?

[AlexeyAB]

• Can you detect successfully by using your model?

• do you have /results/ folder?

• do you have folder /results/kitti_results/data/ ?

• show obj.data file, did you set eval=kitti ?

• show command that you used

• Can you successfully run ./darknet detector map ... on kitti val dataset?

---

If you can't get Kitti-evalutation, also you can try to use BDD-evalutation methond

• eval=bdd in obj.data file

• ./darknet detector valid cfg/kitty.data cfg/kitty.cfg kitty.weights

• try to download gt_KITTI_valid_for_bdd_eval_method.json from https://github.com/jwchoi384/Gaussian_YOLOv3/tree/master/bdd_evaluation

• and try to use this script https://github.com/jwchoi384/Gaussian_YOLOv3/blob/master/bdd_evaluation/evaluate.py

cd bdd_evaluation/

python evaluate.py det gt_KITTI_valid_for_bdd_eval_method.json ../results/bdd_results.json

[err-or]

Hi Alexey,

1. Yes I can successfully detect with my model
2. The /results/ folder was already there even before running the training
3. I get the /results/kitti_results/data/ folder when I run ./darknet detector valid with eval = kitti in obj.data but I encounter the error that I mentioned. I only get a single .txt file one of the image file but that .txt file is blank.
4. This is the command that I used: ./darknet detector valid data/obj.data cfg/yolo-obj.cfg backup/yolo-obj_final.weights
5. Yes I can successfully run ./darknet detector map ... and this is what I get after running this: