-
Notifications
You must be signed in to change notification settings - Fork 52
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
When analyze coco2017,this code ap is a little higher than original ap? #17
Comments
Hi @baolinhu, could you please be more specific? |
@matteorr Thanks for reply.I use this code to analyze coco2017val.Use the same prediction result JSON file.
|
Same question, 0.7 point on mAP. |
@DouYishun, thanks for adding info. I assume you mean 0.007? Right now, the only results that seem affected are What is surprising is that the overall AP on medium and large instances is the same, but when using area=all there is a difference. This makes me think the problem might have to do with a possible difference in the annotation of small objects (which are not considered in the COCO Keypoints, but my eval code might be looking at how many small objects there are when computing precision). I'm currently looking into it and will post updates as soon as possible. |
@matteorr Yes, I mean 0.007%.
|
As I suspected in the previous post, the culprit is the different definition of area ranges during evaluation. When wrapping the
These values are initialized in the Params class and my default values for the area ranges are different from the values that are defined in the original cocoeval repo. Specifically, since COCO keypoints don't have small instances I believe that the The definition discrepancy results in the fact that the number of ground truth instances counted is different for the two evaluations, resulting in different AP (higher for I think my solution makes more sense and reached out about it in the past, but they didn't change their code. You can easily choose the one you prefer though, by just changing the default param values in the Params class, or by overwriting them after you instantiated an object of class COCOanalyze and just accessing its params, i.e.:
After this change the results will match. I'm closing this issue, please put a thumbs up if you think its solved, or feel free to reopen if you have further comments. |
@matteorr Thanks. I know your mean.But I still don't fully understand.Since COCO keypoints don't have small instances,the number of ground truth instances should be the same.Because the number of ground truth instances whose areas less than 32**2 pixels should be 0.So |
@baolinhu - With So in this particular case, detections smaller than 32**2 will be ignored by my evaluation code. To convince yourself, try removing all the small detections before loading them in the
You'll see that the results in this case are exactly the same regardless of the value This makes sense to me. But if you still don't agree please post back, maybe I am missing your point. |
|
@baolinhu - Glad the issue is resolved. To follow up one last time on what might be the "best" evaluation strategy, my interpretation is that since we know that COCO does not have ground truth keypoints for instances with area smaller that 32**2 it is better to ignore detections that have area too small, as they most likely will not be good because of the lack of training data. I agree this strategy might penalize algorithms that are making good keypoint predictions for small instances. An interesting approach could be to ignore all detections with an area smaller than the minimum area with which an IoU of 0.5 is possible if the detection is perfectly overlapped with a ground truth of size 32**2. In conclusion, I think there is no definitive "right" or "wrong" way of doing it. As long as you are aware about what are the consequences with either approach, and compare all algorithms using the same technique it shouldn't matter too much. |
Yeah,I agree with your conclusion.Thanks for your patience. |
Worked for me, but |
When analyze coco2017,this code ap is a little higher than original ap?
The text was updated successfully, but these errors were encountered: