Understanding MMPose better (weights and scores)

[korneliaWatson]

Hi all,

I have some general questions about the workings of MMPose that I don't seem to be able to find answers to using google.

1. I noticed that every time I train a model a best_coco_AP_epoch_10.pth is created. Occasionally I had best_coco_AP_epoch_20.pth. How come best coco AP is not created for every epoch? Is it possible to capture weights that generated the best coco AP across all epochs? If so, how?
2. It's not quite clear to me how keypoints are scored. Some scores are over 1, which makes me think it doesn't translate to over 100% confidence score. Is it possible to get more information around this topic?
3. All coco/AP and coco/AR scores are 1, even though on evaluation I can see there are discrepancies. If I'm understanding it correctly, coco/AR is 1 because all keypoints are being detected, but why would coco/AP be 1 as well when precision is not 100% correct?

Any clues are much appreciated. Thank you. :-)

[Ben-Louis]

1. If you set train_cfg.val_interval to 1 in the configuration, validation will be performed after each training epoch and the checkpoint with the best AP across all epochs will be saved.
2. The keypoint score varies depending on the algorithm used. For instance, heatmap-based methods generate scores between 0 and 1, while simcc models do not have an upper bound.
3. The metric 'coco/AP' is sensitive to the sigma of each keypoint, which is defined in the dataset metafile. If you didn't collect this statistic, it is recommended to use other metrics like PCK.

[korneliaWatson]

Thank you. :-)

I've just tried using PCK metric.
To do it I set:

default_hooks = dict(
    checkpoint=dict(
        interval=20, rule='greater', save_best='PCK', type='CheckpointHook'),
    logger=dict(interval=50, type='LoggerHook'),
    param_scheduler=dict(type='ParamSchedulerHook'),
    sampler_seed=dict(type='DistSamplerSeedHook'),
    timer=dict(type='IterTimerHook'),
    visualization=dict(enable=False, type='PoseVisualizationHook'))

and both test_evaluator and val_evaluator have:

val_evaluator = dict(
    ann_file='coco_ps_256x192/front_val.json',
    type='PCKAccuracy')

When running the training script I get an error:

TypeError: PCKAccuracy.__init__() got an unexpected keyword argument 'ann_file'

I'm not seeing an example with PCK metric online so that I could compare it against my code. Would you be able to point out what I'm doing wrong? Thank you :-)

[Ben-Louis]

Please refer to this example

mmpose/configs/hand_2d_keypoint/topdown_heatmap/onehand10k/td-hm_hrnetv2-w18_8xb64-210e_onehand10k-256x256.py

Lines 152 to 158 in 537bd8e

	# evaluators
	val_evaluator = [
	dict(type='PCKAccuracy', thr=0.2),
	dict(type='AUC'),
	dict(type='EPE'),
	]
	test_evaluator = val_evaluator

You can easily configure it by setting

val_evaluator=dict(type='PCKAccuracy')
test_evaluator=val_evaluator

in your config file. The ann_file argument is not required when you're using PCKAccuracy.

[korneliaWatson]

Ah, got it. :-) Thank you.

[korneliaWatson]

Hello again. :-)

I'm getting some interesting results that I wanted to run by you, if possible.

I changed my cfg.train_interval to 1. I noticed, that despite running the training for a maximum of 40 epochs, best_coco_AP weights are generated quite early on, within the first 10 epochs. But when I compare mean absolute errors of best_coco_AP vs the last weights generated, the latter are much better than what is claimed to be the best. Any ideas why this might be?

Actually, I'm seeing similar behaviour for PCK metric, although not on as big scale as with coco/AP.