[FAQ] Common mistakes when parsing SMPL UV data

[Lotayou]

@Lotayou (a.k.a myself) I just found that my SMPL UV data at hand is messed up, UV vertices' topology does not match that of SMPL 3D mesh. Here's the result:

This error hampers my further development since the color inconsistency induced by UV data error serious affects the interpolation accuracy of (x,y,z) coords. I'm afraid I cannot do anything further until this problem is solved.

Any help would be deeply appreciated!

[radvani]

How did you extract the UV map? From the FBX file?

In the code it looks like you're parsing a file named 'SMPL_template_UV_map.obj'. Do you have that file in the repository?

[Lotayou]

@radvani I downloaded the FBX file from official website and convert it into .obj using an online converter , the result is the aforementioned SMPL_template_UV_map.obj.
I'll upload the obj file for your evaluation.

[radvani]

Interesting, today I'm trying to extract the UV map directly from the FBX file, using the FBX SDK. I will let you know if it looks different than yours (perhaps the FBX -> OBJ conversion corrupted something?)

[Lotayou]

Cool! You can visualize the results with the following code snippets:

# create a half-red-half-blue 3D mesh
ones_vec = np.ones(verts.shape[0], dtype=verts.dtype)
    zeros_vec = np.zeros(verts.shape[0], dtype=verts.dtype)
    verts = np.stack((
        np.where(verts[:,1] > 0.5, ones_vec, zeros_vec),
        zeros_vec,
        np.where(verts[:,1] > 0.5, zeros_vec, ones_vec),
        ),axis=1
    )

# write colorized point cloud file 
def write_ply(ply_name, coords, rgbs):
    if rgbs.max() < 1.000001:
        rgbs = (rgbs * 255.).astype(np.uint8)
    
    with open(ply_name, 'w') as f:
        # headers
        f.writelines([
            'ply\n'
            'format ascii 1.0\n',
            'element vertex {}\n'.format(coords.shape[0]),
            'property float x\n',
            'property float y\n',
            'property float z\n',
            'property uchar red\n',
            'property uchar green\n',
            'property uchar blue\n',
            'end_header\n',
            ]
        )
        
        for i in range(coords.shape[0]):
            str = '{:10.6f} {:10.6f} {:10.6f} {:d} {:d} {:d}\n'\
                .format(coords[i,0],coords[i,1],coords[i,2],
                    rgbs[i,0], rgbs[i,1], rgbs[i,2])
            f.write(str)
    

[radvani]

Which FBX file were you using from the official website? Was it basicModel_f_lbs_10_207_0_v1.0.2, or basicModel_m_lbs_10_207_0_v1.0.2? My preliminary printout of texcoords from FBX show that they're different from what's in your OBJ file.

Also, regarding visualizing the results, I think I'm missing something. It looks like you're generating the human model (shown above) by creating the PLY file using the snippet in the previous comment, but how are you generating the images of the red/blue UV map?

[radvani]

Nevermind my first question above, it looks like you're using the female model. I generated a new OBJ and UV map directly from the FBX file and will try to test it now (still working through how to do that exactly).

[Lotayou]

Weird. As I recall SMPL model is gender-independent, and I never bother to find the difference... I guess male model has an extra "part" eh?
It's not necessary to generate the whole map (that requires barycentric interpolation which is kinda tricky), I think the scatter plot (like the one on the right) will be adequate. For that you can just quantize the UV texture coordinates to integers (denpending on your output solution, I use 300), and colorize each UV texture vertex with [0,1] normalized 3D coordinates of the corresponding mesh vertex.

[radvani]

Actually it may take me a bit to test this myself, but I've attached the OBJ I generated from the FBX SDK in case you want to try out. You can use your usual methods to generate the UV map from this (in get_SMPL_UV_map.py).

smpl_torch.obj.zip

[radvani]

I'm getting the following when using your OBJ file, and it seems fairly consistent?

I wonder if there's actually an issue in get_SMPL_UV_map, in that in your version you're creating a 1-1 mapping between texcoord indices and vertices. I don't think that's correct, in that a single texcoord index can correspond to multiple vertices (there are no restrictions against this in OBJ). I changed this so that instead there's a mapping between faces: e.g. each tuple of texcoords maps to the corresponding tuple of vertices. This resulted in my graphic above.

[Lotayou]

I think you got a point, it's indeed much safer to use face correspondences for UV plot... Think you can submit a PR?

[radvani]

Sure, I think my changes broke the render method in that class but I'll fix that and submit a PR tonight.

[radvani]

Submitted the PR! There's a new method in there as well for rendering the scatter plot; when you run get_SMPL_UV_map.py it spits out an image like the following. Let me know if looks right to you.

[Lotayou]

Works perfectly! Thanks a lot for your help! Closing this issue now.

[Lotayou]

@radvani You are absolutely right that texcoords and vertices are not in one-to-one correspondance. But what's puzzling me is that even the same texture vertex could also correspond to multiple 3D vertices, as shown here:

There are five such points in total: [4085, 4088, 4107, 4108, 4109] (0-index) as shown in red dots. Again I cannot observe a pattern here, it seems that the faulty texcoords all gather around the right knee...

That's weird as hell, it seems that the UV texture generation process does not just cut mesh into pieces, but also attach some pieces together at these weird points... I wonder what do you make of this?

[radvani]

That's interesting, my first guess it that the online OBJ converter you're using is trying to minimize the number of unique texture coordinates using too large of an epsilon, causing it to merge together texture coordinates that are in fact distinct.

The reason this can happen is that the FBX format actually doesn't even define a mapping between texture coordinates and vertices. It just defines:

1. A list of vertices
2. A mapping between faces and vertices
3. Three texture coordinates for each face

(Unless I'm missing something, but I've spent awhile staring at this cryptic FBX SDK)

For our model, that means there are 6890 vertices defined in the FBX file, and 41319 texture coordinates. This results in a massive amount of duplicated texture coordinates, which is why most converters minimize them.

I can try to do my own OBJ export with smarter texture coordinate minimization, but I realized I don't have the gender neutral FBX file that corresponds to neutral_smpl_with_cocoplus_reg.pkl. I only have the male and female versions. Do you have the FBX from which you generated the OBJ?

[Lotayou]

Sure, you can find the FBX file via this link, it contains three animations, I used this one:
mtllib mosh_cmu_0511_f_lbs_10_207_0_v1.0.2.obj.mtl
but I'm afraid that all three files are female (judging by the name)...

Also, here's some of the test result I generated from h36m dataset: Google link

It contains multiple things, including some real 3D meshes sampled and aligned with the corresponding 2D image. Can you test your render_point_cloud function with these real human meshes included here, and see if the results are the same with mine?

Now I'm not sure if that online converter I used is faulty, or the .fbx file is corrupted in the first place. Anyway, I'll try to use FBX-SDK for generation, and try to use the other two fbx files for UV data extraction. I'll keep you informed about any new findings.

[radvani]

That sounds good. Actually the OBJ I pasted above (converted from FBX) which didn't do any texture coordinate minimization (and therefore had 41319 texture coordinates), looks quite a bit different from yours. Most of this is probably due to the fact that I used per-corner normals instead of per-vertex, but it goes to show how the conversion can really make an impact.

[Lotayou]

The two models actually looks quite the same to me, apart from the shades... perhaps my eyes are not that sensitive to naked female bodies (lol).
I opened my obj and your obj models together in meshlab, apart from a global translation I think they are actually the same...

[radvani]

Haha you're right they're quite the same on the surface, it's just the normals and (probably) the texture coordinates that are different.

[Lotayou]

BTW, did you notice that in the fbx file there are only 41319 // 3 = 13773 faces, where in the original mesh there are actually 13776 faces? I guess the face 1-1 correspondance is not valid as well...

If the .fbx file is incorrect, I would have to use LSCM to calculate the UV mapping myself... guess you can think of better alternatives rather than this?

[Lotayou]

@radvani I think we can draw the conclusion on this issue, but I still feel safer if you can double check my findings below:

Today I converted the fbx file into .ply format, which generates completely different results:

Some of the original 6890 vertices are duplicated, making a total of 8964, and formed an exact 1-1 correspondance with UV vertices. Now every uv vertex is assigned to a unique 3d vertex, and the color of each uv vertex is uniquely determined without any ambiguity. However, the generated UV maps still suffers from the color inconsistency, like this:


as can be seen, the error is persistent, even the pattern is the same as before (please refer to the results in the mainpage).

By now I'm pretty certain that the real problem lies in the vt_faces representation: some edges in the uv domain actually connects two uv vertices that are far away on real 3D mesh. The only uncertainty remains, is whether the original fbx file contains the wrong vt_face info in the first place, or some error happens during the process of format conversion?

To verify that, I've also tested my algorithm with your attached obj file that you converted from FBX-SDK, and the error remains the same.

I feel it safe to say that the official SMPL data is corrupted in the first place.
However, I still feel safer if someone can double check the above assumptions again.
Can I bother you to test your point cloud rendering algorithm for some unorthodox (not in the rest shape) meshes with your UV data directly extracted from SDK, and see if the color pattern is the same as mine? You can generate some SMPL models with random theta and betas with any SMPL implementation (including mine here) Thanks!

[radvani]

Interesting! Yes, I'll take a look on Monday and see if my results are similar to yours.

[Lotayou]

@radvani Hey, I managed to retrieve some information from the .fbx file with both FBX C++ SDK and Python SDK, but the results seem weird:

For python results
It seems that each "polygon" contains 4 vertices... You can refer to the zip for my test fbx and parsed results (The visualization script is a direct modification from FBX Python SDK\2019.0\samples\ImportScene. I suppressed some redundant skeleton info).
python_ImportScene.zip

For C++ Results
Here's a snapshot of loaded mesh object under VS2015 Debug mode.

The result is also very weird, the mPolygons, which I persume is the 3D vertices, contains 6900 elements instead of 6890, and other attributes doesn't match to correct numbers as well...

Just wonder what kind SDK did you use to parse the following obj file? Is it possible to share me the conversion code so I can cross check it? Thanks:)

Actually it may take me a bit to test this myself, but I've attached the OBJ I generated from the FBX SDK in case you want to try out. You can use your usual methods to generate the UV map from this (in get_SMPL_UV_map.py).

smpl_torch.obj.zip

[radvani]

Hey! I noticed the same thing regarding the use of quads instead of triangles, so I run it through the FBX SDK's triangulator. You can do this like so:

    FbxGeometryConverter converter(_fbxManager);
    converter.Triangulate(scene, true);

Where the scene is your root FbxScene, and fbxManager is your FBXManager. This is using the C++ FBX SDK. I can send over my conversion code soon I'm just cleaning it up a bit.

I'm going to check on my vertex count now...

[radvani]

Pre-triangulation, I'm getting 6890 vertices, and 6900 polygons (quads). Post-triangulation, I get 6890 vertices and 13773 polygons (triangles).

[radvani]

Here's an example I ran against an SMPL file. I'm still wrapping my head around what these UV maps are supposed to look like. Do you see the same color inconsistency here? How do you identify if the UV map looks correct?

[radvani]

Attached is my C++ FBX to OBJ converter. It's meant to run on a Mac but you can get it to run on Linux pretty easily by just changing the log function it's using at the top of the implementation file. Most of the magic happens in VROFbxToObjConverter::convertToObj. Before that it's just a lot of setup.

I ripped this out of a library I have that does more generalized FBX export (e.g. skeletal animations, etc.) so there may be some things in there that aren't strictly necessary.

To use, just create a VROFbxToObjConverter and call convertToObj with source path and destination path, like this:

VROFbxToObjConverter *converter = new VROFbxToObjConverter();
converter->convertToObj(fbxPath, outPath);

FBXConverter.zip

Note that this converter does no collapsing of duplicate texture coordinates, so it ends up with a ton of them. The only dependencies are STL and the FBX SDK.

[radvani]

Ok I think I see what you mean by the color maps being off. When I render just the X values to the scatter plot -- encoding X in the 'R' channel of the UV map, and leaving G and B at zero -- I would expect red to increase from left to right (for my model, where arms are outstretched and the character is facing the camera), but it doesn't. See first image.

However, when I do this same thing using just the mesh's vertices (as opposed to the positions in the 2D image), it works better, but maybe still not perfect (second image). Looking into it now.

[radvani]

Update: the issue appears to be that the SMPL models we generate at runtime (through smpl_torch_generator.py) do not match the template SMPL from which we derived the UV coordinates.

More specifically, we rely on the correspondence from the texcoord indices to the vertex coord indices to be identical in both the template SMPL and the runtime SMPL. We require this so that we can iterate through each face in the template SMPL and find the texture coordinates in the template SMPL and the corresponding vertices in the runtime SMPL. We then write this correspondence to the UV position map.

This is why the scatter plot of the template OBJ's own vertices looks good, but as soon as we pass in another OBJ, we have discontinuities.

[radvani]

Ok, so I had a 3D modeling friend create a new 2D UV map for us. This is not the same as the template SMPL UV map (the parts are arranged in different places) but it does maintain the correspondence between the runtime SMPL and the template SMPL. The results on running it from a generated SMPL model are shown here (ignore the fact that my model is poorly aligned, that's a separate bug).

[radvani]

To use this one, use get_SMPL_UV_map on the OBJ I attached here, then run create_UV_position_maps as normal... just make sure you delete the barycentric coordinate cache file first!

template.obj.zip

[Lotayou]

That’s very inspiring news! I should have thought about it when I first notice the difference between UV maps from SMPL templates and generated meshes. It’s really nice to have a working UV maps though. I will merge this into the current framework ASAP. This could be the game changer for our projects:)

[Lotayou]

@radvani I just finished baseline training with some interesting results. I've sent you an email at radvani@viromedia.com to discuss about my findings. Just wanna holla at you in case my email get spammed:)

[ypflll]

@Lotayou Good job.
But seems your SMPL's UV map is not extracted correctly and I am not sure where is the problem. The way I extract it seems simple. Here are some details maybe help:

1. I load basicModel_f_lbs_10_207_0_v1.0.2.FBX;
2. Then got 6900 quadrilateral and cut them into triangles(13776);
3. Some vertices correspond to more than one(4 at most) position on the UV map.
   My rendered UV map is like this:
   

[radvani]

We actually tried that, but the UV map extracted from the FBX templates doesn't share the same vertex to texture coordinate mapping as the meshes produced by the SMPL generator at runtime (see comments above).

Is there a way you know of to make that work?

[ypflll]

@radvani I tried your code. I see that you use FbxGeometryConverter to triangulate the mesh which I didn't use. Instead I get 6900 quads and save each quad into 2 triangles manually. The index is got by GetPolygonVertex.
I'm not very sure which is the right way to triangulation, but maybe you can have a try with my method.

[Lotayou]

@ypfill This result actually looks promising. However I wonder if it will work on synthesized 3D mesh too, besides the SMPL template. Do you wanna share a converted .obj file or a processing script so I can acquire the aforementioned triangulation result? You can either attach it here or send me an email via lingbo@pku.edu.cn. Many thanks.

[ypflll]

@Lotayou Sorry that I can't share code or data for some reason. Any discussion is welcomed.

[Dorniwang]

@ypflll Hi, your job seems good! I wonder which coordinate system of body vertices is used to be as the uv position map's pixel rgb value. Cause i use coordinate system of image and get the same uv position map as radvani get. Maybe you use the world coordinate system, i guess?

[Dorniwang]

@radvani Hi, I have get the same uv position map as your firstly got, and i use it to get almost right coordinates of body in image. But the uv position map does seems weird.

[ypflll]

@Dorniwang We use coordinates aligned in the image plane.

[Lotayou]

@ypflll It's all right. It just that something doesn't add up according to your descriptions:

1. There are two different ways to split a quad into two triangles, I wonder how you determine that?
2. I got the same 6900 quads as you, but 6900 * 2 =13800 > 13776. How do you manage to get exactly 13776 faces?

BTW, did you use my script for UV map generation or you write it on your own?

[Dorniwang]

@Lotayou Hi, some quads (about 24 ) in SMPL only has three vertices, so the total number of triangles after split is just 13776. But I has the same question about how to split it, because I had done this half month ago, but i did not get the right triangles, and the index of triangles is only for control point, which is model vertices, and has no enough information about map between vertex and uv.

[ypflll]

@Lotayou
1.I don't think that how to cut the quad matters. Say the vertice index is (0,1,2,3), I use (0,1,2) and (2,3,0).
2. As @Dorniwang said, not all 6900 faces are quads, some are triangles.
Besides, the mapping mode I get is eByPolygonVertex.

[Lotayou]

@ypflll @radvani @Dorniwang Good news guys, I've successfully parsed SMPL original UV maps! The trick is to use Blender software for fbx2obj conversion, which is way easier than that cryptic FBX-SDK. After that, run data_utils/triangulation.py to acquire correctly triangluated UV texture map. Feel free to try it out guys:)

[radvani]

Nice job! I tried it out and it worked on my end.

[Dorniwang]

@Lotayou Amazing! I'll try it!

[Dorniwang]

@Lotayou Usinig the new uv map, I have got the right uv position map! That's greate!

[xinghuokang]

@ypflll @radvani @Dorniwang Good news guys, I've successfully parsed SMPL original UV maps! The trick is to use Blender software for fbx2obj conversion, which is way easier than that cryptic FBX-SDK. After that, run data_utils/triangulation.py to acquire correctly triangluated UV texture map. Feel free to try it out guys:)

Hi, when I run data_utils/triangulation.py, it meets "KeyError: 'joint_regressor'"
model = SMPLModel(device=device,
model_path='basicmodel_m_lbs_10_207_0_v1.0.0.pkl',
data_type=data_type,
simplify=True)
so, I can not generate smpl_fbx_template.obj