About the data format for 3D and 2D joints #22

tqtrunghnvn · 2021-07-12T07:10:35Z

Hi authors,

I found that the original 3DPW dataset has:

3D joints with the shape of 24x3
2D joints with the shape of 18x3

However, your customized 3DPW dataset has:

3D joints with the shape of 24x4
2D joints with the shape of 24x3

Therefore, my question is "what is the fourth element of your customized 3D joints"?
Thank you!

kevinlin311tw · 2021-07-12T23:48:46Z

Thanks for the questions.

In our code, we use a unified data format for 2D/3D joints for different datasets.

For 3D joints, the format is [x, y, z, a], where the fourth element indicates the availability of the joint. It is a binary flag. For most cases (such as 3DPW dataset with SMPL style 3D joints), a is always 1. We keep this format for future adaptation to other different 3D joints/mesh topology.

For 2D joints, the format is [x, y, v], where v is the visibility flag.

tqtrunghnvn · 2021-07-13T02:59:31Z

Thanks so much for your quick reply!

I found that you already reorganized the original 3DPW dataset and managed your own 3DPW dataset, consisting of .tsv files.

The 3DPW dataset originally has train/validation/test split. However, in your customized 3DPW dataset, I only saw train.img.tsv and test.img.tsv.
Could you please clarify this?

tqtrunghnvn · 2021-07-13T04:45:28Z

Moreover, I also compared the original 3DPW dataset and your customized 3DPW dataset, and I found the difference.
I took the same person and the same frame. The data is much different:

In the original 3DPW dataset:

joint2d = [[761.0245361328125, 1427.6376953125, 1.0], 
				[763.7735595703125, 1254.625, 1.0], 
				[757.8124389648438, 1021.6619262695312, 1.0], 
				[871.8953247070312, 1010.0325927734375, 1.0], 
				[838.7183837890625, 1243.81982421875, 1.0], 
				[812.2440185546875, 1417.615966796875, 1.0], 
				[709.1945190429688, 1071.94677734375, 1.0], 
				[708.8104248046875, 942.5911254882812, 1.0], 
				[724.396728515625, 823.7357177734375, 1.0], 
				[883.9208984375, 818.1397094726562, 1.0], 
				[888.2723388671875, 938.8115234375, 1.0], 
				[897.5230102539062, 1059.4345703125, 1.0], 
				[807.9381103515625, 771.5740966796875, 1.0], 
				[802.7290649414062, 636.399658203125, 1.0], 
				[810.5869140625, 1011.4450073242188, 1.0], 
				[805.8743896484375, 803.0057373046875, 1.0], 
				[802.6295166015625, 891.9404907226562, 1.0], 
				[818.800537109375, 722.0154418945312, 1.0], 
				[804.0947265625, 668.561279296875, 1.0], 
				[820.7557983398438, 697.0828857421875, 1.0], 
				[833.2789916992188, 680.14111328125, 1.0], 
				[798.1824951171875, 682.2127685546875, 1.0], 
				[843.4369506835938, 701.8033447265625, 1.0], 
				[772.8783569335938, 705.4711303710938, 1.0]]

In your customized 3DPW dataset:

joint2d = [[ 7.67384253e-01  8.47698757e-02 -5.60344048e-01]
                               [ 8.28211824e-01  3.11791144e-03 -5.87908769e-01]
                               [ 7.10723322e-01 -9.30041110e-04 -5.61701921e-01]
                               [ 7.58508424e-01  1.81683114e-01 -5.69663679e-01]
                               [ 8.45361280e-01 -3.59436410e-01 -5.95123847e-01]
                               [ 6.94116843e-01 -3.71307514e-01 -5.74784826e-01]
                               [ 7.57282539e-01  3.13208635e-01 -5.72228007e-01]
                               [ 8.15533953e-01 -7.35199899e-01 -6.38437082e-01]
                               [ 7.13418712e-01 -7.46048192e-01 -6.23979217e-01]
                               [ 7.60892928e-01  3.61799676e-01 -5.52873324e-01]
                               [ 8.30624588e-01 -7.90706438e-01 -5.26462391e-01]
                               [ 7.28885027e-01 -7.93635708e-01 -5.09829213e-01]
                               [ 7.48760967e-01  5.54727070e-01 -6.02829374e-01]
                               [ 8.22302452e-01  4.70086415e-01 -5.97221059e-01]
                               [ 6.85288513e-01  4.64970268e-01 -5.87709265e-01]
                               [ 7.54457942e-01  6.29499406e-01 -5.69330585e-01]
                               [ 9.07146496e-01  4.77772425e-01 -6.03743600e-01]
                               [ 5.96054458e-01  4.70962621e-01 -5.83015465e-01]
                               [ 9.31566392e-01  2.32074144e-01 -6.41171805e-01]
                               [ 5.78144159e-01  2.32812816e-01 -6.19179812e-01]
                               [ 9.56104924e-01  6.07977247e-04 -6.08673423e-01]
                               [ 5.75637751e-01 -4.62294618e-03 -5.72327194e-01]
                               [ 9.68516244e-01 -7.55253650e-02 -6.00078079e-01]
                               [ 5.71770439e-01 -8.07533496e-02 -5.57852445e-01]]

These two vectors have not only different dimensions but also very different values. The same situation for joint3d.
Could you please clarify this difference?

Thank you so much, and hope to see your answer soon!

kevinlin311tw · 2021-07-13T06:39:19Z

We don't use validation set in our experiments. We don't have validation tsv file.

When I look at the first sample in train.label.tsv, I see the expected annotations:

joint2d = [[761.0245361328125, 1427.6376953125, 1.0], 
				[763.7735595703125, 1254.625, 1.0], 
				[757.8124389648438, 1021.6619262695312, 1.0], 
				[871.8953247070312, 1010.0325927734375, 1.0], 
				[838.7183837890625, 1243.81982421875, 1.0], 
				[812.2440185546875, 1417.615966796875, 1.0], 
				[709.1945190429688, 1071.94677734375, 1.0], 
				[708.8104248046875, 942.5911254882812, 1.0], 
				[724.396728515625, 823.7357177734375, 1.0], 
				[883.9208984375, 818.1397094726562, 1.0], 
				[888.2723388671875, 938.8115234375, 1.0], 
				[897.5230102539062, 1059.4345703125, 1.0], 
				[807.9381103515625, 771.5740966796875, 1.0], 
				[802.7290649414062, 636.399658203125, 1.0], 
				[810.5869140625, 1011.4450073242188, 1.0], 
				[805.8743896484375, 803.0057373046875, 1.0], 
				[802.6295166015625, 891.9404907226562, 1.0], 
				[818.800537109375, 722.0154418945312, 1.0], 
				[804.0947265625, 668.561279296875, 1.0], 
				[820.7557983398438, 697.0828857421875, 1.0], 
				[833.2789916992188, 680.14111328125, 1.0], 
				[798.1824951171875, 682.2127685546875, 1.0], 
				[843.4369506835938, 701.8033447265625, 1.0], 
				[772.8783569335938, 705.4711303710938, 1.0]]

I couldn't find your provided example in our tsv file.

If you find the vector somewhere in our training pipeline (such as model outputs), it would be more explainable. Please note, following the literature, we did some coordinate normalizations for better training (see https://github.com/microsoft/MeshTransformer/blob/main/metro/datasets/human_mesh_tsv.py#L123).

During inference, we will undo the normalization in order to visualize the joints in the image (see https://github.com/microsoft/MeshTransformer/blob/main/metro/utils/renderer.py#L276).

The operations are commonly used in the literature. We mainly follow GraphCMR for the data pre-processing.

tqtrunghnvn · 2021-07-13T07:02:02Z

Oh I am sorry for the mistake. The data I got must be:

In your customized 3DPW dataset:

joint2d = [[761.0245361328125, 1427.6376953125, 1.0], 
				[763.7735595703125, 1254.625, 1.0], 
				[757.8124389648438, 1021.6619262695312, 1.0], 
				[871.8953247070312, 1010.0325927734375, 1.0], 
				[838.7183837890625, 1243.81982421875, 1.0], 
				[812.2440185546875, 1417.615966796875, 1.0], 
				[709.1945190429688, 1071.94677734375, 1.0], 
				[708.8104248046875, 942.5911254882812, 1.0], 
				[724.396728515625, 823.7357177734375, 1.0], 
				[883.9208984375, 818.1397094726562, 1.0], 
				[888.2723388671875, 938.8115234375, 1.0], 
				[897.5230102539062, 1059.4345703125, 1.0], 
				[807.9381103515625, 771.5740966796875, 1.0], 
				[802.7290649414062, 636.399658203125, 1.0], 
				[810.5869140625, 1011.4450073242188, 1.0], 
				[805.8743896484375, 803.0057373046875, 1.0], 
				[802.6295166015625, 891.9404907226562, 1.0], 
				[818.800537109375, 722.0154418945312, 1.0], 
				[804.0947265625, 668.561279296875, 1.0], 
				[820.7557983398438, 697.0828857421875, 1.0], 
				[833.2789916992188, 680.14111328125, 1.0], 
				[798.1824951171875, 682.2127685546875, 1.0], 
				[843.4369506835938, 701.8033447265625, 1.0], 
				[772.8783569335938, 705.4711303710938, 1.0]]

In the original 3DPW dataset:

joint2d = [[8.169990e+02 6.940840e+02 8.994710e-01]
                               [8.064530e+02 8.138730e+02 9.828750e-01]
                               [7.282780e+02 8.140330e+02 9.386620e-01]
                               [7.127130e+02 9.444220e+02 8.730780e-01]
                               [7.127340e+02 1.064240e+03 9.139100e-01]
                               [8.845230e+02 8.138130e+02 9.476240e-01]
                               [8.898530e+02 9.389390e+02 9.635330e-01]
                               [8.950580e+02 1.048680e+03 8.936900e-01]
                               [7.648070e+02 1.064310e+03 8.733710e-01]
                               [7.596690e+02 1.252160e+03 8.256300e-01]
                               [7.645820e+02 1.424360e+03 7.860840e-01]
                               [8.532480e+02 1.059310e+03 8.713770e-01]
                               [8.428340e+02 1.231570e+03 8.600970e-01]
                               [8.066020e+02 1.408730e+03 8.674500e-01]
                               [8.064390e+02 6.885450e+02 1.951078e-01]
                               [8.376640e+02 6.884630e+02 1.843966e-01]
                               [7.651140e+02 6.991450e+02 1.813974e-01]
                               [8.479240e+02 6.939170e+02 3.753400e-02]]

tqtrunghnvn · 2021-07-13T07:12:19Z

By the way, the data for 3D joints is:

In your customized 3DPW dataset:

joint3d = [[-0.07414655387401581, 0.6259320974349976, 0.23802196979522705, 1.0], 
				 [-0.08293654024600983, 0.2553229033946991, 0.11177857965230942, 1.0], 
				 [-0.11118228733539581, -0.20798233151435852, -0.03652437776327133, 1.0], 
				 [0.11696193367242813, -0.22830316424369812, 0.030360819771885872, 1.0], 
				 [0.07074201852083206, 0.23932282626628876, 0.14836804568767548, 1.0], 
				 [0.03360944241285324, 0.6109831929206848, 0.2613641917705536, 1.0], 
				 [-0.1982978880405426, -0.10928139090538025, 0.02480170875787735, 1.0], 
				 [-0.1994788944721222, -0.3582003712654114, 0.01969926245510578, 1.0], 
				 [-0.17634113132953644, -0.5815433859825134, -0.05405966565012932, 1.0], 
				 [0.13607390224933624, -0.5964640974998474, 0.006664540618658066, 1.0], 
				 [0.1584412306547165, -0.36617332696914673, 0.08553362637758255, 1.0], 
				 [0.18082615733146667, -0.12912505865097046, 0.108730249106884, 1.0], 
				 [-0.017434636130928993, -0.6804583072662354, -0.047672443091869354, 1.0], 
				 [-0.02729572169482708, -0.9356586933135986, -0.0475996695458889, 1.0], 
				 [-0.0037099719047546387, -0.2259705662727356, 0.015443656593561172, 1.0], 
				 [-0.018986718729138374, -0.6225076913833618, -0.03026910126209259, 1.0], 
				 [-0.02061682939529419, -0.4550009071826935, 0.003700997680425644, 1.0], 
				 [-0.005796360783278942, -0.7664101123809814, -0.11061951518058777, 1.0], 
				 [-0.02451862022280693, -0.8751493692398071, -0.04619435593485832, 1.0], 
				 [-0.012012489140033722, -0.8034844398498535, -0.1795082986354828, 1.0], 
				 [0.01684357412159443, -0.8400061726570129, -0.1395934820175171, 1.0], 
				 [-0.04841940104961395, -0.835761547088623, -0.1426447182893753, 1.0], 
				 [0.049128368496894836, -0.8115767240524292, -0.05223245173692703, 1.0], 
				 [-0.08516306430101395, -0.8038434386253357, -0.058558084070682526, 1.0]]

In the original 3DPW dataset:

joint3d = [[ 7.67384253e-01  8.47698757e-02 -5.60344048e-01]
                                 [ 8.28211824e-01  3.11791144e-03 -5.87908769e-01]
                                 [ 7.10723322e-01 -9.30041110e-04 -5.61701921e-01]
                                 [ 7.58508424e-01  1.81683114e-01 -5.69663679e-01]
                                 [ 8.45361280e-01 -3.59436410e-01 -5.95123847e-01]
                                 [ 6.94116843e-01 -3.71307514e-01 -5.74784826e-01]
                                 [ 7.57282539e-01  3.13208635e-01 -5.72228007e-01]
                                 [ 8.15533953e-01 -7.35199899e-01 -6.38437082e-01]
                                 [ 7.13418712e-01 -7.46048192e-01 -6.23979217e-01]
                                 [ 7.60892928e-01  3.61799676e-01 -5.52873324e-01]
                                 [ 8.30624588e-01 -7.90706438e-01 -5.26462391e-01]
                                 [ 7.28885027e-01 -7.93635708e-01 -5.09829213e-01]
                                 [ 7.48760967e-01  5.54727070e-01 -6.02829374e-01]
                                 [ 8.22302452e-01  4.70086415e-01 -5.97221059e-01]
                                 [ 6.85288513e-01  4.64970268e-01 -5.87709265e-01]
                                 [ 7.54457942e-01  6.29499406e-01 -5.69330585e-01]
                                 [ 9.07146496e-01  4.77772425e-01 -6.03743600e-01]
                                 [ 5.96054458e-01  4.70962621e-01 -5.83015465e-01]
                                 [ 9.31566392e-01  2.32074144e-01 -6.41171805e-01]
                                 [ 5.78144159e-01  2.32812816e-01 -6.19179812e-01]
                                 [ 9.56104924e-01  6.07977247e-04 -6.08673423e-01]
                                 [ 5.75637751e-01 -4.62294618e-03 -5.72327194e-01]
                                 [ 9.68516244e-01 -7.55253650e-02 -6.00078079e-01]
                                 [ 5.71770439e-01 -8.07533496e-02 -5.57852445e-01]]

Could you please elaborate on this difference and share with us your code for creating 3DPW tsv files?

kevinlin311tw · 2021-07-13T09:24:27Z

I am not very sure about the format as you pointed out here. I think there are three possible reasons for this problem.

The joint order may be different.
We may use different normalizations (with image size and/or root coordinate).
We may use different coordinate systems.

As I can recall, I actually tried to normalize all these factors before. It has been a while, but I remember that our 3D joints will look similar to the original 3DPW ones as you post here.

Unfortunately, the tsv generation code is missing. The code was written 1 year ago, and I can't find it now. I need some time to dig it out. I may need to rewrite a new code when I have time. I will add it to the repo then.

Thanks a lot for your interests in our work :)

tqtrunghnvn · 2021-07-13T09:32:32Z

You mean you normalized both joint_2d and joint_3d before writing them into tsv file.

If that is right, why do you still normalize joint_2d and joint_3d in your dataloader code?

kevinlin311tw · 2021-07-13T09:36:42Z

We did normalization in the dataloader. Please stay tuned for the tsv generation code!

tqtrunghnvn · 2021-07-13T09:40:31Z

As far as I figure out, you only provided the tsv generation code for pseudo-3d-dataset (https://github.com/microsoft/MeshTransformer/blob/main/metro/tools/tsv_demo.py)

There is no tsv code for 3DPW dataset. If you can provide it soon, it would be highly appreciated!

Thank you so much!

kevinlin311tw · 2021-07-31T01:33:47Z

The 3dpw tsv example code can be found at https://github.com/microsoft/MeshTransformer/blob/main/metro/tools/tsv_demo_3dpw.py

wjingdan · 2021-10-25T07:57:03Z

The 3dpw tsv example code can be found at https://github.com/microsoft/MeshTransformer/blob/main/metro/tools/tsv_demo_3dpw.py

Can you provide the code to generate the TSV file of the human36m dataset? Thank you very much.

Hao-hui · 2022-02-24T08:49:10Z

The 3dpw tsv example code can be found at https://github.com/microsoft/MeshTransformer/blob/main/metro/tools/tsv_demo_3dpw.py

Can you provide the code to generate the TSV file of the human36m dataset? Thank you very much.

Hi, wjingdan
I am interested in the authors' code to generate the TSV file of the human36m dataset. Or do you know the difference between TSV file of human36m and the original human36m.
Thank you!

kevinlin311tw · 2022-04-19T23:53:19Z

Hi all,
You may find the example script to reproduce Human3.6M tsv files below:
https://github.com/microsoft/MeshTransformer/blob/main/metro/tools/tsv_demo_h36m.py

WZS666 · 2022-07-19T09:18:46Z

Hello @kevinlin311tw ,
Have you ever get Out of memory (killed) and end the process while writing things in to tsv?
I'm trying to build my own tsv files, but trouble with OOM (killed) problem.
If you have any suggestion. Please let's me know.
Thanks a lot !

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

About the data format for 3D and 2D joints #22

About the data format for 3D and 2D joints #22

tqtrunghnvn commented Jul 12, 2021

kevinlin311tw commented Jul 12, 2021 •

edited

Loading

tqtrunghnvn commented Jul 13, 2021

tqtrunghnvn commented Jul 13, 2021

kevinlin311tw commented Jul 13, 2021 •

edited

Loading

tqtrunghnvn commented Jul 13, 2021

tqtrunghnvn commented Jul 13, 2021

kevinlin311tw commented Jul 13, 2021

tqtrunghnvn commented Jul 13, 2021

kevinlin311tw commented Jul 13, 2021

tqtrunghnvn commented Jul 13, 2021

kevinlin311tw commented Jul 31, 2021

wjingdan commented Oct 25, 2021

Hao-hui commented Feb 24, 2022

kevinlin311tw commented Apr 19, 2022

WZS666 commented Jul 19, 2022

About the data format for 3D and 2D joints #22

About the data format for 3D and 2D joints #22

Comments

tqtrunghnvn commented Jul 12, 2021

kevinlin311tw commented Jul 12, 2021 • edited Loading

tqtrunghnvn commented Jul 13, 2021

tqtrunghnvn commented Jul 13, 2021

kevinlin311tw commented Jul 13, 2021 • edited Loading

tqtrunghnvn commented Jul 13, 2021

tqtrunghnvn commented Jul 13, 2021

kevinlin311tw commented Jul 13, 2021

tqtrunghnvn commented Jul 13, 2021

kevinlin311tw commented Jul 13, 2021

tqtrunghnvn commented Jul 13, 2021

kevinlin311tw commented Jul 31, 2021

wjingdan commented Oct 25, 2021

Hao-hui commented Feb 24, 2022

kevinlin311tw commented Apr 19, 2022

WZS666 commented Jul 19, 2022

kevinlin311tw commented Jul 12, 2021 •

edited

Loading

kevinlin311tw commented Jul 13, 2021 •

edited

Loading