TotalCapture Distortion

[yohanshin]

Hi, I wonder how you dealt with the distortion of cameras.

The given distortion parameter is too small, so using 1st order radial distortion, that parameter seems like the system is undistorted at all.

I wonder if you suffered the same issue, and if yes, how did you deal with that? Doesn't it affect a lot while reconstructing 3D pose from given 2D points?

Thank you very much in advance!

[zhezh]

hi @yohanshin we have observed the same. The distortion is small, you can either use it or not.

For our work, we did not use the distortion for fundamental matrix calculation

[yohanshin]

Dear @zhezh , Thank you very much for your explanation. When I project given 3D ground-truth onto each camera, and I can see that 3D->2D projection does not align well with the original image. I think there might be some miss-calibration of the dataset or maybe it is just me doing a wrong projection?

If you also found this issue, I believe your works, reconstructing 3D keypoints from multi-view may suffer a lot from this since detected 2D keypoints may not recover the accurate 3D triangulation. Or was it okay since you used PSM?

[zhezh]

@yohanshin Yes, in some sequences there is misalignment. We have contacted the TotalCapture authors, the acknowledged it and have not a method to complement it.

However, if your method is improved and fix cases of large mpjpe, you will still see gain in the mpjpe metric despite the misalignments. This is to say that mpjpe gain is consistent with better model.

The similar misalignment also happens in Human3.6M.

In learnable triangulation paper, they calculate relative mpjpe, it is also applicable.

[yohanshin]

@zhezh Thanks for your detailed perspective on this problem. I also used Human3.6M, but I think this misalignment issue is not that much severe at that dataset, is it? I agree that relative MPJPE will help to somehow solve the misalignment issue in terms of evaluation, but not sure if the given calibration is incorrect, would it be still reasonable to use volumetric aggregation from learnable triangulation or cross-view fusion from your work.

I will try to figure this out and thank you so much for providing us with this preprocessing tool-box! It is super helpful.

[zhezh]

@yohanshin

• misalignment is smaller in Human3.6M than TotalCapture
• volumetric and crossview fusion are both reasonable when misalignments happened in training data. The supervision signals for volumetric method and crossview fusion are 3D and 2D coordinates respectively. The misalignment will be averaged across the whole dataset while training. It could result in some offset in final pose predictions but will not lead to corrupted results.

I assume it mainly affects the bias but not the weight for the conv or linear network (just my hypothesis).