Support of rendering multiple channels

[jacksoncsy]

Hi,

Thanks a lot for open sourcing the project, it is awesome!
After a few tries, it seems that the current rasterise function only supports 1 or 3 channels, I wonder if it is possible to rasterise more channels? For example, rasterizing texture and normals together without calling them separately.
I could also modify the codes myself, if you can kindly show me which part to change.

Many thanks,
Shiyang

[pmh47]

For now you need to do it in multiple separate calls, with three or one channels each time, as in the samples.

In principle, it is possible to change the renderer code to support arbitrarily-many channels in a single call, and this should be faster than using multiple calls. However, it requires a bit of work, as OpenGL is designed mainly to render three-channel (RGB) images.

The fiddly OpenGL aspects are:

• add more input/output variables to the forward_vertex and forward_fragment shaders (to pass the interpolated attributes through from vertices to pixels)
• create corresponding textures and color-attachments in RasteriseOpGpu::reinitialise_framebuffer (to retrieve the new image channels)
• change the vertex layout in RasteriseOpGpu::initialise_per_thread_objects (to allow passing the additional attributes into the vertex shader)

Then you'd also have to deal with the relevant book-keeping in setting up the output tensors, passing the additional vertex and background channels in, and including them in the gradient calculation.

[jacksoncsy]

Thanks for the quick response!
May I know if this is on your plan? If not, I will probably go into the codes sometime next week. Please do not close the issue, as I will definitely need your help for this part.

Many thanks,
Shiyang

[pmh47]

I don't plan to do it in the near future, given that one can achieve the same with multiple calls. However, I'm happy to help if you're interested in working on it.

I should also warn you that the gradient calculation in the texturing sample gives inaccurate results along edges where one face occludes another (it should be correct on silhouette edges where a face occludes the background). Solving that problem may impact how the multi-channel implementation has to work (it may mean the deferred shading pass has to be more tightly linked to the rasterise op). I'll think about this properly when I have time, and get back to you.

[jacksoncsy]

No problem as long as I could get some help from you.
Regarding the gradient calculation, are you going to fix that problem soon? PS: I am still not sure what exactly is the issue though.

[pmh47]

Note that c04ec96 adds support for rendering with arbitrary numbers of channels, but only in the python frontend (i.e. it still uses a loop over multiple calls to the Rasterise op, but the user does not need to care about this). Implementing it in the C++ backend instead may give better performance.

More importantly, 47185ce adds new functions rasterise_deferred and rasterise_batch_deferred for deferred rendering (e.g. texturing), that fix the issues with gradients that I hinted at above. In short, you should always use these new functions when using deferred shading, as otherwise gradients through the vertex locations will be incorrect. This is because the spatial-filter-based approximation for gradients wrt vertex locations must be calculated on the final, shaded image, not on the raw G-buffer channels.