How to evaluate for coco dataset? #253

kimna4 · 2018-10-24T14:07:48Z

Hello,

I hope to evaluate my trained model with coco dataset.
But there is no code for evaluation of coco.

With google, I realize that I have to evaluate the result using coco server.
But.. I don't know exactly. How to make the out for server using this code.

Could you tell me about it??

Thank you

lpd2csx · 2018-12-31T02:41:32Z

I modify my test.py in the master folder,output format of which is according to coco official website. I copy the result in .txt to a file in .jason file,because the eval code need the .jason file rest data.Then you can refer cocoapi/PythonAPI/pycocoEvalDemo.ipynb to eval you output.
Here is my modified test.py just for reference.
'''
from future import print_function
import sys
import os
import argparse
import torch
import torch.nn as nn
import torch.backends.cudnn as cudnn
import torchvision.transforms as transforms
from torch.autograd import Variable
from data import COCO_ROOT, COCO_CLASSES as labelmap
from PIL import Image
from data import VOCAnnotationTransform, VOCDetection, BaseTransform, VOC_CLASSES
from data import COCO_CLASSES, COCOAnnotationTransform, COCODetection
import torch.utils.data as data
from ssd import build_ssd

COCO_change_category = ['0','1','2','3','4','5','6','7','8','9','10','11','13','14','15','16','17','18','19','20',
'21','22','23','24','25','26','27','28','31','32','33','34','35','36','37','38','39','40',
'41','42','43','44','46','47','48','49','50','51','52','53','54','55','56','57','58','59',
'60','61','62','63','64','65','67','70','72','73','74','75','76','77','78','79','80','81',
'82','84','85','86','87','88','89','90']

parser = argparse.ArgumentParser(description='Single Shot MultiBox Detection')
parser.add_argument('--trained_model', default='weights/ssd300_COCO_395000.pth',
type=str, help='Trained state_dict file path to open')
parser.add_argument('--save_folder', default='eval/', type=str,
help='Dir to save results')
parser.add_argument('--visual_threshold', default=0.25, type=float,
help='Final confidence threshold')
parser.add_argument('--cuda', default=True, type=bool,
help='Use cuda to train model')
parser.add_argument('--coco_root', default=COCO_ROOT, help='Location of VOC root directory')
parser.add_argument('-f', default=None, type=str, help="Dummy arg so we can load in Jupyter Notebooks")
args = parser.parse_args()

if args.cuda and torch.cuda.is_available():
torch.set_default_tensor_type('torch.cuda.FloatTensor')
else:
torch.set_default_tensor_type('torch.FloatTensor')

if not os.path.exists(args.save_folder):
os.mkdir(args.save_folder)

def test_net(save_folder, net, cuda, testset, transform, thresh):
# dump predictions and assoc. ground truth to text file for now
filename = save_folder + 'result.txt'
num_images = len(testset)
for i in range(num_images):
print('Testing image {:d}/{:d}....'.format(i+1, num_images))
img = testset.pull_image(i)
x = torch.from_numpy(transform(img)[0]).permute(2, 0, 1)
x = Variable(x.unsqueeze(0))

    if cuda:
        x = x.cuda()

    y = net(x)      # forward pass
    detections = y.data
    # scale each detection back up to the image
    scale = torch.Tensor([img.shape[1], img.shape[0],
                         img.shape[1], img.shape[0]])

    # ii -> category id
    for ii in range(detections.size(1)):
        j = 0
        while detections[0, ii, j, 0] >= thresh:

            score = detections[0, ii, j, 0].cpu().data.numpy()
            pt = (detections[0, ii, j, 1:]*scale).cpu().numpy()
            coords = (pt[0], pt[1], pt[2], pt[3])

            # standard format of coco ->
            # [{"image_id":42,"category_id":18,"bbox":[258.15,41.29,348.26,243.78],"score":0.236},{...},...]
            with open(filename, mode='a') as f:
                f.write(
                    '{"image_id":' + str(testset.pull_anno(i)[0]['image_id']) +
                    ',"category_id":' + str(COCO_change_category[ii]) +
                    ',"bbox":[' + ','.join(str(c) for c in coords) + ']'
                    ',"score":')
                f.write('%.2f' %(score))
                f.write('},')
                # you need to delete the last ',' of the last image output of test image
            j += 1

def test_voc():
# load net
num_classes = 81 # change
net = build_ssd('test', 300, num_classes) # initialize SSD
net.load_state_dict(torch.load(args.trained_model))
net.eval()
print('Finished loading model!')
# load data
testset = COCODetection(args.coco_root, 'trainval35k', None, COCOAnnotationTransform)
if args.cuda:
net = net.cuda()
cudnn.benchmark = True
# evaluation
test_net(args.save_folder, net, args.cuda, testset,
BaseTransform(net.size, (104, 117, 123)),
thresh=args.visual_threshold)

if name == 'main':
test_voc()
'''

ShabnamBehnam · 2019-01-06T12:42:04Z

Hi,
I changed your 'test.py' code as you said, but I encountered the following error, which is related to the number of COCO dataset classes. Do you know the cause of this erosion?
" RuntimeError: While copying the parameter named conf.0.weight, whose dimensions in the model are torch.Size([324, 512, 3, 3]) and whose dimensions in the checkpoint are torch.Size([804, 512, 3, 3]). "
The number 324 is the product of the number of classes and 4.(now the number of classes in the code is 81 but if you change it, 324 changes according to it)
but I do not know what exactly 804 is?
Can you help me to solve this problem?
Thank You

eng100200 · 2019-02-09T08:50:54Z

@lpd2csx hello, have you trained SSD on coco human keypoints dataset?

Hliang1994 · 2019-04-24T03:08:51Z

@lpd2csx hi, I have use you test code, and converted txt to json. And then I use PythonAPI/pycocoEvalDemo to eval the result. But the map is much lower compared to corrent one. My train loss of coco is about 3.8. Could I have you result about eval?
my result is as follow:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.005
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.017
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.002
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.002
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.009
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.014
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.016
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.016
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.035

banxia1994 · 2019-04-26T05:18:52Z

@lpd2csx hi, I have use you test code, and converted txt to json. And then I use PythonAPI/pycocoEvalDemo to eval the result. But the map is much lower compared to corrent one. My train loss of coco is about 3.8. Could I have you result about eval?
my result is as follow:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.005
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.017
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.002
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.002
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.009
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.014
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.016
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.016
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.035

Hi, Do you have any idea about this results right now? same as you

Ai-is-light · 2019-05-24T14:22:05Z

ztianlin · 2019-07-19T03:50:19Z

@Hliang1994@banxia1994 Hi~have you solved the COCO evaluation problem?

Damon2019 · 2019-08-09T12:59:57Z

@ShabnamBehnam @lpd2csx @Hliang1994 @eng100200 hi,can you help me? have you solved the COCO evaluation problem? or other help,thanks

sauravsolanki · 2019-09-02T10:59:12Z

Hi,
I changed your 'test.py' code as you said, but I encountered the following error, which is related to the number of COCO dataset classes. Do you know the cause of this erosion?
" RuntimeError: While copying the parameter named conf.0.weight, whose dimensions in the model are torch.Size([324, 512, 3, 3]) and whose dimensions in the checkpoint are torch.Size([804, 512, 3, 3]). "
The number 324 is the product of the number of classes and 4.(now the number of classes in the code is 81 but if you change it, 324 changes according to it)
but I do not know what exactly 804 is?
Can you help me to solve this problem?
Thank You

The Problem is with num_class . change it and load respective dataset

eng100200 · 2019-12-26T03:36:39Z

@lesonly its a txt file, right? if it is txt format, very easy, just read each line and find how many features or paramters it has then save each of them in a dictionary , i assume each line has many parameters,..and so on,,,then dump the dictionary in a list (append list), finally save in a json file which you want.

eng100200 · 2019-12-26T07:03:04Z

@lesonly ok, let me see your file.

eng100200 · 2019-12-27T01:47:27Z

@lesonly its not hard problem you have list of dictionaries in result.txt and each dictionary has three keys. Its not hard to read and make a json file. I will contact you later today or tomorrow.

lesonly · 2019-12-27T01:52:28Z

@eng100200 Very appreciate for your help!

lesonly · 2019-12-31T12:16:38Z

@lpd2csx hi, I have use you test code, and converted txt to json. And then I use PythonAPI/pycocoEvalDemo to eval the result. But the map is much lower compared to corrent one. My train loss of coco is about 3.8. Could I have you result about eval?
my result is as follow:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.005
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.017
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.002
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.002
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.009
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.014
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.016
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.016
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.001
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.035

@Hliang1994 @banxia1994 Hi, Do you have any idea about this results ? same as you

rebeen · 2020-04-25T11:56:28Z


from future import print_function
import sys
import os
import argparse
import torch
import torch.nn as nn
import torch.backends.cudnn as cudnn
import torchvision.transforms as transforms
from torch.autograd import Variable
from data import COCO_ROOT, COCO_CLASSES as labelmap
from PIL import Image
from data import VOCAnnotationTransform, VOCDetection, BaseTransform, VOC_CLASSES
from data import COCO_CLASSES, COCOAnnotationTransform, COCODetection
import torch.utils.data as data
from ssd import build_ssd

COCO_change_category = ['0','1','2','3','4','5','6','7','8','9','10','11','13','14','15','16','17','18','19','20',
'21','22','23','24','25','26','27','28','31','32','33','34','35','36','37','38','39','40',
'41','42','43','44','46','47','48','49','50','51','52','53','54','55','56','57','58','59',
'60','61','62','63','64','65','67','70','72','73','74','75','76','77','78','79','80','81',
'82','84','85','86','87','88','89','90']

parser = argparse.ArgumentParser(description='Single Shot MultiBox Detection')
parser.add_argument('--trained_model', default='weights/ssd300_COCO_395000.pth',
type=str, help='Trained state_dict file path to open')
parser.add_argument('--save_folder', default='eval/', type=str,
help='Dir to save results')
parser.add_argument('--visual_threshold', default=0.25, type=float,
help='Final confidence threshold')
parser.add_argument('--cuda', default=True, type=bool,
help='Use cuda to train model')
parser.add_argument('--coco_root', default=COCO_ROOT, help='Location of VOC root directory')
parser.add_argument('-f', default=None, type=str, help="Dummy arg so we can load in Jupyter Notebooks")
args = parser.parse_args()

if args.cuda and torch.cuda.is_available():
torch.set_default_tensor_type('torch.cuda.FloatTensor')
else:
torch.set_default_tensor_type('torch.FloatTensor')

if not os.path.exists(args.save_folder):
os.mkdir(args.save_folder)

def test_net(save_folder, net, cuda, testset, transform, thresh):
# dump predictions and assoc. ground truth to text file for now
    filename = save_folder + 'result.txt'
    num_images = len(testset)
    for i in range(num_images):
    print('Testing image {:d}/{:d}....'.format(i+1, num_images))
    img = testset.pull_image(i)
    x = torch.from_numpy(transform(img)[0]).permute(2, 0, 1)
    x = Variable(x.unsqueeze(0))

    if cuda:
    x = x.cuda()

    y = net(x)      # forward pass
    detections = y.data
    # scale each detection back up to the image
    scale = torch.Tensor([img.shape[1], img.shape[0],
                         img.shape[1], img.shape[0]])

    # ii -> category id
    for ii in range(detections.size(1)):
        j = 0
        while detections[0, ii, j, 0] >= thresh:

            score = detections[0, ii, j, 0].cpu().data.numpy()
            pt = (detections[0, ii, j, 1:]*scale).cpu().numpy()
            coords = (pt[0], pt[1], pt[2], pt[3])

            # standard format of coco ->
            # [{"image_id":42,"category_id":18,"bbox":[258.15,41.29,348.26,243.78],"score":0.236},{...},...]
            with open(filename, mode='a') as f:
                f.write(
                    '{"image_id":' + str(testset.pull_anno(i)[0]['image_id']) +
                    ',"category_id":' + str(COCO_change_category[ii]) +
                    ',"bbox":[' + ','.join(str(c) for c in coords) + ']'
                    ',"score":')
                f.write('%.2f' %(score))
                f.write('},')
                # you need to delete the last ',' of the last image output of test image
            j += 1
def test_voc():
# load net
    num_classes = 81 # change
    net = build_ssd('test', 300, num_classes) # initialize SSD
    net.load_state_dict(torch.load(args.trained_model))
    net.eval()
    print('Finished loading model!')
    # load data
    testset = COCODetection(args.coco_root, 'trainval35k', None, COCOAnnotationTransform)
    if args.cuda:
    net = net.cuda()
    cudnn.benchmark = True
    # evaluation
    test_net(args.save_folder, net, args.cuda, testset,
    BaseTransform(net.size, (104, 117, 123)),
    thresh=args.visual_threshold)

if name == 'main':
test_voc()

hjlee9182 · 2020-10-27T05:53:40Z

When inference ssd using coco dataset.
Some box element has minus value ...
How to solve problem..?

This was referenced Jun 2, 2019

How to train own datasets? #16

Open

How eval my trained model by my custom data, which is transformed like coco format? #359

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to evaluate for coco dataset? #253

How to evaluate for coco dataset? #253

kimna4 commented Oct 24, 2018

lpd2csx commented Dec 31, 2018

ShabnamBehnam commented Jan 6, 2019 •

edited

eng100200 commented Feb 9, 2019

Hliang1994 commented Apr 24, 2019

banxia1994 commented Apr 26, 2019

Ai-is-light commented May 24, 2019

ztianlin commented Jul 19, 2019

Damon2019 commented Aug 9, 2019

sauravsolanki commented Sep 2, 2019

eng100200 commented Dec 26, 2019

eng100200 commented Dec 26, 2019

eng100200 commented Dec 27, 2019

lesonly commented Dec 27, 2019

lesonly commented Dec 31, 2019

rebeen commented Apr 25, 2020

hjlee9182 commented Oct 27, 2020

How to evaluate for coco dataset? #253

How to evaluate for coco dataset? #253

Comments

kimna4 commented Oct 24, 2018

lpd2csx commented Dec 31, 2018

ShabnamBehnam commented Jan 6, 2019 • edited

eng100200 commented Feb 9, 2019

Hliang1994 commented Apr 24, 2019

banxia1994 commented Apr 26, 2019

Ai-is-light commented May 24, 2019

ztianlin commented Jul 19, 2019

Damon2019 commented Aug 9, 2019

sauravsolanki commented Sep 2, 2019

eng100200 commented Dec 26, 2019

eng100200 commented Dec 26, 2019

eng100200 commented Dec 27, 2019

lesonly commented Dec 27, 2019

lesonly commented Dec 31, 2019

rebeen commented Apr 25, 2020

hjlee9182 commented Oct 27, 2020

ShabnamBehnam commented Jan 6, 2019 •

edited