Running the YOLO9000 network with hierarchical softmax support and detections

[relh]

This PR adds in the hierarchical softmax functionality and processes object detections from the network per the YOLO9000 paper and darknet implementation.

It adds an additional second threshold, the 'hierarchythreshold' that specifies the conditional probability cutoff for traversing down the tree for choosing an object label.

It adds a data file in cfg/ that corresponds to the paths for map and tree that are already part of the config.

It adds a preprocessing step to the yolo9000 network that builds two data structures (hyponym_tree and coco_map) from these files stored in cfg/data.

A normal threshold of 0.1 and a hierarchythreshold of 0.6 produced decent results on the sample data set. Results aren't as great as I had hoped for but I'm not sure what the cause there might be.

For the NMS9000 function, I suppress any boxes with IOU outputs > 0.25 because there was a tendency for animals to have their bodies and their heads classified as the same thing.

fixes #49

[codecov-io]

Codecov Report

Merging #257 into master will decrease coverage by 0.35%.
The diff coverage is 34.48%.

@@            Coverage Diff             @@
##           master     #257      +/-   ##
==========================================
- Coverage   67.51%   67.15%   -0.36%     
==========================================
  Files          33       33              
  Lines        2087     2113      +26     
==========================================
+ Hits         1409     1419      +10     
- Misses        678      694      +16

Impacted Files	Coverage Δ
setup.py	0% <ø> (ø)	⬆️
darkflow/defaults.py	67.41% <100%> (+0.37%)	⬆️
darkflow/net/yolo/misc.py	47.14% <12.5%> (-10.27%)	⬇️
darkflow/net/yolo/__init__.py	93.1% <50%> (-6.9%)	⬇️
darkflow/net/yolov2/predict.py	90.74% <62.5%> (+0.74%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ae3ac33...321d2d6. Read the comment docs.

[abagshaw]

@relh Awesome! Can't wait to give this a shot when it's merged. When I have some time I'll build a test for inference with YOLO9000 to increase coverage as most of the new code is not covered - but that's not a big issue right now.

One other thing - if you have time, it might be helpful to add some information to the README.md on the two different thresholds and maybe some more specifics about using YOLO9000...if there are any? If you don't have time to do this I probably can (although I'm certainly not the expert on this).

[halt9]

@relh I cloned your repo and gave it a shot, but here's what I get when I try to run the demo:
Press [ESC] to quit demo Traceback (most recent call last): File "/usr/local/bin/flow", line 6, in <module> cliHandler(sys.argv) File "/usr/local/lib/python2.7/dist-packages/darkflow/cli.py", line 25, in cliHandler tfnet.camera() File "/usr/local/lib/python2.7/dist-packages/darkflow/net/help.py", line 122, in camera single_out, img, False) File "/usr/local/lib/python2.7/dist-packages/darkflow/net/yolov2/predict.py", line 40, in postprocess boxes = self.findboxes(net_out) File "/usr/local/lib/python2.7/dist-packages/darkflow/net/yolov2/predict.py", line 28, in findboxes boxes=box_constructor9000(meta,net_out) File "darkflow/cython_utils/cy_yolo9000_findboxes.pyx", line 113, in darkflow.cython_utils.cy_yolo9000_findboxes.box_constructor (darkflow/cython_utils/cy_yolo9000_findboxes.c:3610) float hthreshold = meta['hierarchythreshold'] KeyError: 'hierarchythreshold'

This is after the initial setup phase.

[relh]

I think it's a real issue that I don't initialize the hierarchythreshold in defaults. I'll actually test it with the demo rather than my external server I was running it with and update the PR!

[abagshaw]

@relh Thanks for your fix - that does seem to fix the issue, but now it removes the user's ability to change the value of hierarchythreshold for themselves in a flag.

I think the best/easiest way to fix this would just be to set the default value for hierarchythreshold in defaults.py to something like 0.6 like you suggested instead of -0.1. Also might as well get rid of the condition: if FLAGS.hierarchythreshold > 0.0: in darkflow/net/yolo/__init__.py as that should always evaluate to true - so self.meta['hierarchythreshold'] = FLAGS.hierarchythreshold can always run. What do you think?

Personally I think setting hardcoded default values buried in the codebase is not great practice as it's hard to track down if it ever needs to be changed...but that's just my opinion.

[relh]

@abagshaw I realized that I think I was misusing meta, as I believe you guys have been only having meta store keys that are present in the cfg file. Since the hierarchy threshold isn't in the cfg (and hence meta), it might make more sense for it to be solely in flags.

I agree adding it to defaults.py with a sensible default value makes sense: the key is just to make sure it's included as part of the parameters that get passed to the box_constructor(meta,np.ndarray[float,ndim=3] net_out_in) (so basically part of meta in some fashion)

I reverted it now to be similar to the darknet repo, where the value is hard coded in at 0.5.

I figured it would be worth talking about first because there is also a third threshold, the IOU threshold that is hard coded in darknet but could be another FLAGS parameter if we were interested in letting people control that. Adding the NMS IOU threshold however makes us have to add another parameter to the NMS function call..

I already added two other keys to meta, 'hyponym_tree' and 'coco_map' and I know there is also the 'labels' key and maybe a few others added in preprocessing. I think it would be great if meta and flags could practically become one, and maybe there is a section in defaults.py that adds/overwrites all of the appropriate keys in the meta dict with what has been passed in as a flag, similar to how 'thresh' works now in yolo/__init__.py

[abagshaw]

@relh Could we not just pass FLAGS to box_constructor as another argument? I'll let others chime in on how they think this should be best implemented.

On another note I've been playing around this for the past little while and just as you mentioned - detection accuracy isn't amazing. It appears to me that it's not even doing as well as YOLOv2 on the sample images. I'm going to try to compare these detections to the ones I get with the original darknet code to see if there are any noticeable differences. Maybe this is just the tradeoff you get with having tons of detection classes...

[relh]

I'll add in these flags in a way that makes sense.

Beyond the flags, tomorrow I'm planning to test out the original darknet repo for both the hierarchical 200 and 9000 class to see if the results from here are consistent. I know the 200 has existing detection examples online and it performs reasonably, but the real test is with the same images on a local copy with the same tuning of parameters.

[halt9]

Just wanted to point out the recent commits have indeed solved the issue that I was talking about above. However, when I run a video, I'm not getting any bounding boxes.

[relh]

Ok, so I've restructured my PR.

Now, net/yolo/__init__.py adds the new keys to meta, similar to how the meta['labels'] key gets added.

I also add the hierarchical threshold per @abagshaw 's suggestion, taking it from the flags.

Beyond this I've removed the separate cy_yolo9000_findboxes.pyx file as it wasn't necessary and put the added functionality back in cy_yolo2_findboxes.pyx.

Currently it checks the meta to see if certain keys exist to determine what kind of object detection to perform, this means it defaults to the softmax across the 200 items as described in the paper with the yolo9000 weights as both the tree and map meta keys are present.

It might make more sense to provide a configuration option to choose between yolo2/yolo200/yolo9000. I'd hope for some input from you all on this one.

It is also worth noting that in this thread https://groups.google.com/forum/#!topic/darknet/Cw99SpxKK2A they comment on the not excellent/improved accuracy of the 9000 network.

I ran darknet for comparisons against my results and didn't find many differences (see philipperemy/yolo-9000#3).

I'm still noting some weirdness with the 200 class version in one of the demo images with a horse categorizing as an apple rather than a horse.

@halt9 Are you using the default threshold values? if you are running flow to test with a video, you won't see many bounding boxes unless you use a lower threshold. Here is a command I ran with a demo video I had:

flow --model cfg/yolo9000.cfg --load bin/yolo9000.weights --threshold 0.1 --hierarchyThreshold 0.1 --demo videofile.avi

I've got to leave this alone for a few days for work, but think it's getting closer to being able to join the main repository.

[cjosephson]

What is the status on this? Would love to see it added.

Also, I ran your flow command with threshold 0.1 on this video and didn't see any bounding boxes:
https://drive.google.com/file/d/1rqXVvFxpwdOHU1MH-hDQpHIotMByvXo-/view?usp=sharing

If I use a higher resolution video, I do get some bounding boxes but it is pretty inaccurate: