Ghost Texture

[czw0078]

Hi, I test this wonderful renderer with a coffee mug model and it looks great! The model is built by software Blender by myself.

I use this script run on PyTorch version to load the coffee mug and get this

Nice rendering work!

However, when I compare it with below result from OpenGL renderer, I realize that some "ghost" textures on the object on above neural renderer result:(the flip of texture is no big issue)

So I explored a little bit and also found the phenomenon on other white background objects(Both Chainer version and PyTorch version) when I test other models.

Any idea how to fix this bug?

[nkolot]

I suspect that it might have to do with the fact that the renderer does not support vertex normals when loading an .obj file. So most likely is an .obj loading issue and not a renderer bug.

Can you test with an obj file that does not have vertex normals and see if you are getting the desired results? Another thing to try is to load the mesh using neural_renderer, save it with save_obj and try to renderer it with OpenGL to see if it renders properly.

I'm quite busy and I won't have too much time this week to do proper tests.

[czw0078]

Can you test with an obj file that does not have vertex normals and see if you are getting the desired results? Another thing to try is to load the mesh using neural_renderer, save it with save_obj and try to renderer it with OpenGL to see if it renders properly.

Good strategy to identify the bug. I try it. Thank you for the advice!

[czw0078]

Thank you for the debugging advice!! I followed your guide and tried what you suggested, and here is my brief: (Hope it can help you narrow down your search for the solution)😄

(1). First, the "save_obj.py" is broken (so I use Chainer version instead). When I try to use the PyTorch version of save_obj.py, I get a pure white object and has no textures on the saved obj. I look into the save_obj.py source code and see this 1's image. I guess the save_obj.py is still incomplete, right?

(2). Since the PyTorch version does not work, I use the Chainer version save_obj function to save the loaded mesh (coffee mug), and here is how the saved model looks like in the OpenGL (Preview software on Mac):

The result looks correct, except (a) the blurry texture (I think due to the texture_size_out=16 not as large as the size of textures tensor texture_size x texture_size x texture_size = 4 x 4 x4 = 64 ); and (b) the minor flipping issue. But we do not see any ghost problem here.

Here is the Chainer code I used to save_obj:

import chainer
import numpy as np
import scipy.misc
import neural_renderer

renderer = neural_renderer.Renderer()

vertices, faces, textures = neural_renderer.load_obj( '/home/chengfei/projects/test_coffe_mug/straight.obj', load_texture=True, texture_size=16)

neural_renderer.save_obj('/home/chengfei/projects/test_coffe_mug/chainer_save.obj', vertices, faces, textures)

(3). Here is how the same model renders via your PyTorch version of neural renderer (with varying texture_size)

texture_size = 4

texture_size = 8

texture_size = 16

You can notice that the larger the texture_size is, the sharper the textures look, also the more obvious ghosts appear. Hope it can give you some insights on this bug.

(4). I also investigated your " vertex normals loading" doubt, and trace this part of the code.

If we give input .obj as the below:

f 4/5/4 10/10/17 11/12/18

then the extracted values are v0 = 5, v1 = 10, v2 = 12, the vertex normals i.e. those /4, /17, /18
(i.e. the numbers after the second slash [1] are not being read).

So currently I think the renderer (as you said) does not support normals.

@nkolot Do you have any ideas how "not supporting normals" relate to this ghost texture issues?

(5) As far as I've tested, this ghost problem only happens with image-texture 3D models (it works fine with textures composed of solid colors).
At least 5 image-texture .obj I have tested exhibit this ghost problems.

@nkolot Do you have any thoughts on how to go about it? Thank you so much!

[1] file format of .obj file

f  v1/vt1/vn1   v2/vt2/vn2   v3/vt3/vn3 . . .

Polygonal geometry statement.

Specifies a face element and its vertex reference number. You can
optionally include the texture vertex and vertex normal reference
numbers.

The reference numbers for the vertices, texture vertices, and
vertex normals must be separated by slashes (/). There is no space
between the number and the slash.

v is the reference number for a vertex in the face element. A
minimum of three vertices are required.

vt is an optional argument.

vt is the reference number for a texture vertex in the face
element. It always follows the first slash.

vn is an optional argument.

[czw0078]

I believe the ghost problem is solved now, we can close this issue. I have already sent a pull request. Note: My fix also considered the divided by zero issue.

[logicsh]

When I use save_obj.py:
image = image.cpu().numpy()
RuntimeError: CUDA error: an illegal memory access was encountered
I want to ask how to solve this problem.