Results Mismatch compare paper's(?) on Trainig setting 3 and test setting 3

[chojinie]

Hi, @linjing7

I bring some result from guiding instruction on paper and github pages.
I have some question about that.

1. Followed by github's guide like setting 3(Train on MSCOCO, Human3.6m, MPII, UBody-Train and Test on UBody-val) :
   [Train] It took a 60hours by A60002way(64G*2) gpu:0,1 / Command : python train.py --gpu 0,1 --lr 1e-4 --exp_name output/train_setting3 --train_batch_size 16 --ubody_benchmark --decoder_setting wo_decoder

--> I got snapshot_13.pth.tar

[Test] Same Hardware / Command : python test.py --gpu 0,1,2,3 --exp_name output/train_setting3/ --pretrained_model_path ../output/train_setting3/model_dump/snapshot_13.pth --testset UBody --test_batch_size 64 --decoder_setting wo_decoder

--> Then I got result

and Paper's result just like that

It seems very different, even setting3 result is better than paper's result.

Does this event is right? then why? I'm merely speculating that the difference in performance arises from whether osx_l was fine-tuned with the U-body dataset or initially trained from scratch including the U-body dataset.

2. Followed by github's Testing OSX guideline,
   I test the model with command
   python test.py --gpu 0,1 --exp_name /output/test_setting3 --pretrained_model_path ../pretrained_models/osx_l_wo_decoder.pth.tar --testset UBody --test_batch_size 64 --decoder_setting wo_decoder

--> I got this result

It is more near the result of paper's.
In testing OSX, they use the pretrained model osx_l.pth.tar for testing. Is this the experimental approach referred to in Table 5 of the paper? And is this osx_l.pth.tar a model that was pretrained on existing datasets (MPII, H36M, MSCOCO-Wholebody) and then fine-tuned on the U-body dataset?
However, since this is a without decoder model, the performance is slightly lower than what is presented in the paper, right?

[linjing7]

Hi, thanks for your interest! Yes, the result of Table 5 refers to the osx_l.pth.tar model that pretrained on existing datasets and fine-tuned on the UBody trainset. I'm happy that you can achieve a better result than our paper by designing a better training scheme.

[chojinie]

Thanks for rapid and accurate reply!!

I have another questions about correlation of setting with performance.

In my command "[Train] It took a 60hours by A60002way(64G*2) gpu:0,1 / Command : python train.py --gpu 0,1 --lr 1e-4 --exp_name output/train_setting3 --train_batch_size 16 --ubody_benchmark --decoder_setting wo_decoder"

The only difference between the command I used and setting3 command you provided is that I used A6000 GPUs (with a memory capacity of 64GB) and utilizes two GPUs.

[linjing7]

Okay, it seems that there do not exist significant differences between your setting and the original setting. But I'm pleased that you could achieve a better performance than our paper :)

[chojinie]

Thanks!!

[chojinie]

Hi @linjing7.
I'm Jin.

I reopen issue to clarify the problem.

It seems like an extension of what we discussed earlier. I followed the Training Setting 3 protocol, with the only differences being that I used only two GPUs and my GPUs are A6000s.

I used the following command:
python train.py --gpu 0,1 --lr 1e-4 --exp_name output/train_setting3 --train_batch_size 16 --ubody_benchmark --decoder_setting wo_decoder

And, inspecting train.py, it appears that osx_l.pth.tar is added by default as the pretrained_model_path.

I was digging some code and module, and find that in train.py, there are --continue flag,

and it is connected to base.py with _make_model function

Does 'pretrained_model_path' not activate when i try to train setting 3?

because, in command there is not use --continue. so _make_model function didn't call the pretrained_model_path(osx_l.pth.tar)

[linjing7]

Yes, pretrained_model_path is not activated. Only the backbone pretrained model is loaded here.

[chojinie]

Thanks for timey reply!