.mtl file support

[ikostrikov]

Thanks for adding support for the .obj file format! Do you have any plans on adding .mtl support as well?

[yuvaltassa]

What do imagine MTL support will look like?

Right now textures are specified directly in the MJCF as in this example. Other than the filename of the texture the MTL doesn't contain much, mostly material properties which are mostly not supported by the MuJoCo renderer. Do you have a particular example of a textured OBJ that you cannot load using this pattern?

We do have a long term plan to add more material properties to MuJoCo, not necessarily supported by the native renderer, but mostly so that they could be forwarded to more powerful renderers like Unity's. When we do that it would make sense to read and save the material properties specified in MTL.

[ikostrikov]

Some .obj files have several textures and materials specified in accompanying .mtl files. In theory, it's possible to split them into several geometries and assign a different texture for each geometry; however, mujoco does not support geometries with zero volume, which can often happen after splitting. And it adds processing step that might be difficult to do for large collections of objects (for example, ShapeNet).

[JeanElsner]

Edit: Actually, just what @ikostrikov said. However I think you can avoid unnecessary inertia/convex hull computations using the compiler options.

What works for us so far is to bake the material (diffuse channel) of more complex models into textures. Both the MTL and MuJoCo native renderer shader models are quite simple, so I don't see a huge benefit to supporting MTL as such. However, if the OBJ and MTL formats were fully supported it would greatly ease the workflow of bringing complex visual meshes into MuJoCo. This is especially true if your mesh consists of multiple objects or polygon groups that refer to different materials.

Take this robot model for example:

In the original visual meshes all of the coloured details are actually polygon groups referring to different materials, there are no textures. We can prepare this for MuJoCo by either baking into a texture or separating all the polygon groups into meshes and assigning each a corresponding material in MJCF. If you just naively import the visual meshes you will end up with monochrome robot links.

Bottom line is you can get decent results with a little effort but if you could just import the OBJ/MTL files directly that would be really nice to have.

[yuvaltassa]

Hi Jean,
Thanks for the explanation. One of the main reasons we didn't include support for polygon groups and multiple textures is that we couldn't find good examples of redistributable models that make use of this feature. If you can point us to an example of such a model, we could try to add in this functionality.
Cheers

[ikostrikov]

I can try to find several examples. What do you mean by redistributable models?

[yuvaltassa]

I mean models that are under permissive licenses. E.g. APACHE 2

[ikostrikov]

ShapeNet (see terms of use) is a massive dataset of objects that many objects with several textures per .obj. So, for example, I could find these:

02747177/1ce689a5c781af1bcf01bc59d215f0
04099429/7f8f5ba0ab543cd99cc7053a5669450f

And there are many more.

Also, Sapien (terms of use), another large dataset of articulated objects, has several things. For example:

door/9288
refrigerator/10068

Both of the datasets can be extremely useful for RL research. And it will be great if objects from these datasets can be easily imported into mujoco.

[yuvaltassa]

No cigar. As far as I can tell, neither of these databases qualify as permissive. They just allow you to use their objects for research, and that's it. Terms of use are not a proper license. (Note the MIT license at the bottom of the Sapien terms of use refers to the code which drives the database, not to the objects contained within it.)

@JeanElsner, what about your Franka model above, does it use OBJ groups?

[ikostrikov]

sketchlab has 300k 3d models under CC-BY license (it's listed as a permissive license, if I understand correctly). I found this one that has an .mtl with several textures:
https://sketchfab.com/3d-models/pony-cartoon-885d9f60b3a9429bb4077cfac5653cf9

Does it look good? If necessary, I can find more models.

[yuvaltassa]

Thanks @ikostrikov this model is good in the sense that the licensing is good and indeed the OBJ has several differently-textured sub-parts. It's not ideal in the sense that it's a cartoon model, not exactly the thing that MuJoCo was designed to simulate. So while definitely usable, it's not super motivating 🙂

What we'd really appreciate is a model of a robot that uses this feature. If you can find one, that would be great. That said, this cartoon model is definitely enough for us to start looking into MTL support, so thank you for that.

[yuvaltassa]

This has now been resolved by @kevinzakka's obj2mjcf, see #340 for details.