Implementation is different with the paper?

[niujinshuchong]

Hi, thank you for sharing you work.
I read the TPAMI paper, but i didn't found the two stream architecture in this repositories. The paper propose Pose Adaptive Convolution and Projected Normalized Coordinate Code which are not contained in this repositories either.
Well, from the main.py, it just crops face region in image and feeds to mobilenet. I wonder why using simple regression strategy can produce promising results as you provided? Is data augmentation matters?

[cleardusk]

Well, good question, you are the first one to propose it.

To be honest, the experiments in this project's conclusion is not completely consistent with the original paper.

1. I use the 3x4 matrix (11 degrees if more precise) form to substitute the original R (rotation, 3 degrees) and t_2d (2 degrees) in the original paper.
2. In this repo, VDC finetuned from WPDC loss is better than WPDC loss, while there is no such experiment in the original paper. The original paper's conclusion is that WPDC is better than VDC.
3. PAC and PNCC are all implemented, but the two-stream cascade structure does not bring any performance improvement under my circumstance.
4. The augmentation part has no big difference with the original paper. But it matters.

Why simple regression strategy can produce promising results? The method still needs improvement.

[niujinshuchong]

@cleardusk Thank you for your quick reply. I am very interesting in PAC and PNCC since it may work in human pose estimation.
From the params.py, you load some .npy file. How do you generate those file?
From main.py, you said the network's output control shape, pose and expression of the face. I wonder how they can control the expression, because as I know, there is no such label in the training set.

[cleardusk]

I use BFM face model, the network's output is pose, shape, expression parameter. 3DDFA is parameter regression.
I will update the code of PAC and PNCC, but it may require a little time to release as I am busy these days.
*.npy are generated from BFM (original form is *.mat)

[developer-mayuan]

Well, good question, you are the first one to propose it.

To be honest, the experiments in this project's conclusion is not completely consistent with the original paper.

1. I use the 3x4 matrix (11 degrees if more precise) form to substitute the original R (rotation, 3 degrees) and t_2d (2 degrees) in the original paper.
2. In this repo, VDC finetuned from WPDC loss is better than WPDC loss, while there is no such experiment in the original paper. The original paper's conclusion is that WPDC is better than VDC.
3. PAC and PNCC are all implemented, but the two-stream cascade structure does not bring any performance improvement under my circumstance.
4. The augmentation part has no big difference with the original paper. But it matters.

Why simple regression strategy can produce promising results? The method still needs improvement.

I also wonder why regression strategy works but your result proved it, nice work. What I care about is that since your current implementation only need parameters as the ground truth, does this mean I can prepare the training data by just taking down the first 40 attributes of the shape info and the first 10 attributes of the expression info with 12 elements of the projection matrix from 300W_LP dataset?

[cleardusk]

Yes, but if you do some pre-process to the original image, the corresponding param should be modified, such as scale, offset.

[KeyKy]

@cleardusk How do you generate param_all_norm.pkl? Do you release the code for generating param_all_norm.pkl?

[cleardusk]

Sorry for that, the training details will be updated in the next two months.

[KeyKy]

@cleardusk How do you generate param_all_norm.pkl? Do you open source the code to create the param_all_norm.pkl