3DHP evaluation

[alecda573]

Hi nice work and thanks for releasing your code.

could you please point me to what data for MPI you use to train and validate on? Also, can you tell me what file contains your skeleton and MPI specific dataset class?

Also, do you only evaluate with MpJpE or do you use PcK amd any other metrics on MPI?

[Garfield-kh]

Hi, thank you for the interest!

could you please point me to what data for MPI you use to train and validate on?

For 3DHP, we train on H36M training set, and evaluate on 3DHP test set for cross-evaluation.

can you tell me what file contains your skeleton and MPI specific dataset class?

The code for 3DHP data preparation is borrowed from SPIN. You may refer to DATASETS.md for more details.

do you only evaluate with MpJpE or do you use PcK amd any other metrics on MPI?

We evaluated the PCK, AUC, and MPJPE following previous work.

[alecda573]

@Garfield-kh Thanks for the very informative response! I thought you all performed this cross set evaluation to ensure that you were not overfitting to H36M. This seems to be the standard protocol although some recent papers are choosing to train on MPI and then validate leading to large performance increases.

Could you possibly elaborate on what code needs to be modified to process the training data from MPI?

Also, if I am reading your code correctly do you guys process the whole evaluation set or only a specific action from the validation set?

[Garfield-kh]

Could you possibly elaborate on what code needs to be modified to process the training data from MPI?

Maybe you can check this mpi_inf_3dhp.py from SPIN for the training data.

Also, if I am reading your code correctly do you guys process the whole evaluation set or only a specific action from the validation set?

It is downsampled from the original one, following SPIN setting.

[alecda573]

@Garfield-kh what do you guys set your threshold at for the evaluation of PCK on MPI? Also, what do you use for your scales argument?

Specifically in this method and then subsequently compute_AUC:

`def compute_PCK(gts, preds, scales=1000, eval_joints=None, threshold=150):
PCK_THRESHOLD = threshold
sample_num = len(gts)
total = 0
true_positive = 0
if eval_joints is None:
eval_joints = list(range(gts.shape[1]))

for n in range(sample_num):
    gt = gts[n]
    pred = preds[n]
    # scale = scales[n]
    scale = 1000
    per_joint_error = np.take(np.sqrt(np.sum(np.power(pred - gt, 2), 1)) * scale, eval_joints, axis=0)
    true_positive += (per_joint_error < PCK_THRESHOLD).sum()
    total += per_joint_error.size

pck = float(true_positive / total) * 100
return pck

`

[Garfield-kh]

The threshold is 150, the scale 1000 is to convert unit m to mm.