Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update crowd label handeling #984

Merged

Conversation

bluerythem
Copy link
Contributor

The previous way to thresh anchor boxes based on iou is not a good metric. For example, small anchors inside large crowd box would have small iou so not being ignored.

This PR update crowd label handling logic. Here, we use ioa instead of iou so that anchors that overlap with crowd boxes could be properly filtered out.

@ppwwyyxx
Copy link
Collaborator

As said in the code, we did not do this filtering because the official implementation (Detectron) did not do this.
And in practice I've found no differences.
Did you find any accuracy difference by using your filtering?

For example, small anchors inside large crowd box would have small iou so not being ignored.

It's arguable whether small anchors inside large crowd box should be ignored.

@bluerythem
Copy link
Contributor Author

I think it is wrong not to handle crowd label even Detectron did not. The network would be penalized for doing true detection inside the crowd label region.

I have not tried to train on Coco but for my private dataset which includes about 3% crowd label, it improve about 2 points on overall mAP.

If it did not make a difference for your former training, is it possible that former code based on iou did not properly ignore anchors in crowd box region?
Another possible theory is the crowd region would be marked as bgs if not taken cared of. But later the code samples fgs and bgs and there are many bg labels so some mistakes in crowd label region is may be statistically not significant.

I am curious why ignoring small anchors inside large crowd box is arguable? Isn't true positives inside crowd box are usually small? Like a person in a crowd of people?

@ppwwyyxx
Copy link
Collaborator

Isn't true positives inside crowd box are usually small? Like a person in a crowd of people?

Crowd is labeled as boxes. In coco the labeled boxes are sometimes unnecessarily big due to lazy labelers. The crowd box may as as result overlap meaningful objects, especially objects of a different category (e.g. a car in a crowd of people).

What you said about crowd boxes all make sense -- that's why there is crowd box handling code, though disabled. And they are disabled because there is importance in maintaining consistency with official code.

Therefore how about let's add your improvement but at the same time still keep things consistent by default.
Could you change the default CROWD_OVERLAP_THRESH to something like 1.0 or larger (since it's not used at all for now) and add a comment that this is by default disabled?

@ppwwyyxx
Copy link
Collaborator

Just made the above changes to this PR. Let me know if you found anything is incorrect

@ppwwyyxx ppwwyyxx merged commit c842bf5 into tensorpack:master Nov 18, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants