How to add evaluation?

[wangyakunn]

Thanks a lot!
I find "#no evaluation" in custom_dataset.py in line 357. That means this is a no evaluation version.And I want to know how to fix it to add the evaluation?

[github-actions]

Message that will be displayed on users first issue

[OrangeSodahub]

Yes, we didn't produce the validation datasets nor the evaluation script. You could refer to the pcdet/tools/pred.py for how to get customed predictions and other evaluation scripts in pcdet for the evaluation metrics caculation.

[OrangeSodahub]

BTW, we use our customed evaluation tools in here , however, not recommended for you, you'd better refer to official evaluation tools in KITTI, Waymo, etc.

[wangyakunn]

Thanks for your reply!

# 2022.04.30: Remove evaluation
 
   logger.info('**********************Start evaluation %s/%s(%s)**********************' %
                (cfg.EXP_GROUP_PATH, cfg.TAG, args.extra_tag))
    test_set, test_loader, sampler = build_dataloader(
        dataset_cfg=cfg.DATA_CONFIG,
        class_names=cfg.CLASS_NAMES,
        batch_size=args.batch_size,
        dist=dist_train, 
        workers=args.workers, 
        logger=logger, 
        training=False
    )
    print("len(test_loader) = ",len(test_loader))
    eval_output_dir = output_dir / 'eval' / 'eval_with_train'
    eval_output_dir.mkdir(parents=True, exist_ok=True)
    args.start_epoch = max(args.epochs - args.num_epochs_to_eval, 0)  # Only evaluate the last args.num_epochs_to_eval epochs

    repeat_eval_ckpt(
        model.module if dist_train else model,
        test_loader, args, eval_output_dir, logger, ckpt_dir,
        dist_test=dist_train
    )
    logger.info('**********************End evaluation %s/%s(%s)**********************' %
                (cfg.EXP_GROUP_PATH, cfg.TAG, args.extra_tag))

I tried to unravel the comments in this code but it looks like doesn't work.I find that this build_dataloader() cannot load val data.

test_set, test_loader, sampler = build_dataloader(
        dataset_cfg=cfg.DATA_CONFIG,
        class_names=cfg.CLASS_NAMES,
        batch_size=args.batch_size,
        dist=dist_train, 
        workers=args.workers, 
        logger=logger, 
        training=False
    )

Why it works in train dataset but test dataset? And I would like to know if it is complex to add evaluatation based on your code.

[OrangeSodahub]

What does I find that this build_dataloader() cannot load val data. mean?

Our code is mainly based on OpenPCDet, add evaluation is not difficult, you could just regard our code same as the OpenPCDet and add it. Two steps suggested: get the predictions from model (refer to pred.py, different from OpenPCDet), and evaluate them (same as the OpenPCDet but need to fit your customed dataset also.

[wangyakunn]

I'm sorry i made a little mistake about "cannot load val data" and now I have solved it.
I read part of Openpcdet code about eval carefully. Thanks for your suggestions and I got it.