[TRACKER] What it takes to create a photorealistic character?

[mysticfall]

Godot version:
master / 8b231b9

OS/device including version:
Manjaro Linux 18.0.4

Issue description:
I've been trying to create a photorealistic character in Godot but have stumbled upon many obstacles in the process, some of which I believe to be issues of the engine while others might be something I've yet to learn how to do it properly.

Either way, it is true that it is quite difficult to create a realistic looking humanoid character in Godot at this stage, and with the upcoming release of 3.1 it is expected more users would be interested in doing something similar.

So I thought it may be not inappropriate to create a tracker issue where you can check what actual engine limitations remaining to achieve a photorealistic humanoid character in Godot, or see what common caveats exist for inexperienced users like myself.

[FIXED] 1. Difficulty to use a roughness, transmission texture

• Related issue: Mesh looks too glossy when using a roughness texture #18880, Spatial material roughness/env map issues #21680

Although #18880 is marked as 'closed', it's still almost impossible to use a roughness texture created by Substance Painter to create a realistic looking human character.

Unless you post process the texture to make it look extremely bright (to such an extent that it loses all its details), your character will look like it was soaked in oil even with roughness set to 1.0.

For instance, if you use a roughness texture like this:

You will get the result as this, even with maxed out roughness of 1.0:

The workaround I found was to convert it to a shader material and use sqrt value of the input texture.

2. Rather primitive support for subsurface scattering effect(SSS)

• Related issue: Subsurface scattering does not support transmittance. #11398

Currently, SSS in Godot only creates a blurry effect without transmittance as described in #11398. It can be faked to some extent by using transmission effect, but it's also problematic since it will make the whole mesh glow like a light bulb if you overuse it and it's almost impossible to use a transmission texture because of the issue previously mentioned (a workaround with sqrt can be applied too, however).

Also, it seems that SSS texture only affects the amount of the effect without altering the diffuse color. It'd be helpful to create a more realistically looking texture if it was supported in Godot.

3. Visual artifacts when creating a hair mesh with overlapping hair strips

• Related issue: Blocky artifacts appears when a transparent mesh has overlapping regions #26785

In order to create a realistic looking hair asset, it is often required to design a mesh with overlapping hair strips. But when the material is transparent, it creates blocky visual artifacts like shown below:

The only workaround that I found is to make your hair strips opaque where it overlaps and limit the overall glossiness by lowering specular and boosting roughness.

4. Anisotropy doesn't seem to work

I need a confirmation from others to report it as an issue. But I found that anisotropy does have any effect on my hair mesh.

For now, I had to fake it with a shader, but it's something needed to create a realistic looking hair asset.

5. Lack of dedicated skin and hair shaders

• Related issue: Provide dedicated skin and hair shaders for creating realistic human characters #27283

Both Unity and Unreal provides dedicated shaders to render realistic human skin and hair, and without such a support it is quite difficult to achieve a similar result in Godot. It'd be great if Godot could support such a workflow as described in this link, for instance:

• https://docs.unrealengine.com/en-us/Resources/Showcases/PhotorealisticCharacter

6. Artifacts with glossiness and shadow when viewed from a certain angle

• Related issue: Glossy area recieves highlight even when it's shadowed due to lack of specular occlusion #19561

This issue was once reported but closed as an expected behaviour. However, I still have great trouble trying to prevent shadowed areas from looking extremly glossy, and I think it's the biggest roadblock to approaching a realistic render result.

It seems that the glossiness is mainly coming from the GI probe, and from its HDRI sky. But I haven't been able to mitigate the problem without making the GI influence to almost an inconsequential level.

My model looks like this when it faces a directional light:

But shadowy patches and highly glossy areas begin to be more noticeable when it faces away from a light source:

I was once told that I have set specular or indirect light too high, but I could not solve the problem by adjusting either of them.

7. Considerations when using blend shapes

• Related issue: Animating blend shapes produces visual artifacts #19880, Random jerkiness with animating blendshapes #22361, and also this, in case you are still using Better Collada and have a lot of blendshapes.

Probably it does not really belong to this tracker as it's not a rendering issue. But if you need to create a realistic human character, changes are high that you'll have at least a few blend shapes to animate expressions or visemes. So I decided to include it here.

Firstly, If you have a lot of blend shapes in your model, chances are that Better Collada may fail to export it. Godot's Bender exporter(.escn) doesn't have such a problem, so you may want to use it instead of Better Collada.

There is another caveat regarding blend shapes that can be easily overlooked. In Godot, a resource is shared between all nodes and a mesh (.mesh) is no exception to the rule. It may not be too obvious that blend shapes modify the actual mesh resource, so if you have more than one character you need to make sure the mesh is not shared between them, otherwise it would cause problems mentioned in #19880 and #22361.
.....
I will try to update this if I found any more issues or workarounds of them in future. My goal is to make this issue a place for the discussion regarding the things can be improved in Godot to allow its users to create photorealistic human characters with a quality on par with other popular game engines.

And it's more than likely that the above list is missing such things that are essential in achieving photorealism but I have no idea if they exist because I'm not really a 3d artist. In that case, please feel free to leave a comment and I'll add them to the list.

[Ace-Dragon]

The fact you needed to do workarounds with sqrt() suggests to me that Godot maps the roughness value linearly rather than on the more intuitive curve, much like what Blender Cycles did until recently.

I think these are great topics to tackle for the Vulkan project (which should hopefully begin soon as 3.1 is almost out).

---

On another note, the Godot docs. do have a note to be careful with the generic specular value, as it's not actually energy conserving. I do not know why there's not a separate energy-conserving control for tweaking the specular effect, but it would be good to have.

[mysticfall]

I just added no. 7 regarding blendshape related issues. I was hesitant at first to include it on the list because it's not entirely a rendering issue.

But I suppose creating a realistic character is quite pointless if you can't make it blink its eyes or change expressions, so I decided to mention it.

[mysticfall]

It's quite tangential to the issue and could be a mere anecdote, but I found that using ACES tonemapper with subdued colors (lowering saturation) instead of using other alternatives provides a better result when rendering a realistic character, possibly because it helps counter rather plastic looking skin due to lack of dedicated shaders:

(Please ignore the parts below the neck, as it's still a WIP)

[mysticfall]

The overly glossy area around the neck in the last screenshot turned out to be the same issue as #19561.

And from the discussion there, it apparently is an expected behaviour and we'd need a dedicated skin shader to solve such a problem. So I opened another issue to request such a feature in #27283.

[ConnorBP]

ACES is a good workflow for linear. I would highly recommend you learn how linear workflow works as it may help you with setting your values right.

Here is a great VFX channel i watch that explains the linear and ACES workflows. (watch linear first as ACES is basically the Linear Workflow ++)
Linear Workflow
ACES Workflow

[mindinsomnia]

The eyes could use some work, "the eyes are the window to the soul" as they say so it'd be worth devoting a lot of time to them. Right now they just look like white balls with black dots painted on them.

[mysticfall]

@mindinsomnia I agree with you. The original model had much more realistic (and complex) eye structure which involves a transparent region. I thought it could incur performance problems when there are many characters in a scene so I replaced it with a simpler model.

Maybe it was a premature optimization. But for now, other parts of my character don't look too realistic either and it's already suffering from low fps. So I think I better wait until I could make it more realistic to revisit the eye part.

At least, it's not a problem of Godot but that of my lacking artistic skills, so probably we better fix other problems involving actual bugs or limitations of the engine itself.

Thanks for the criticism, by the way!

[mysticfall]

Finally, 1 has been fixed with #21682, so the workaround mentioned in the description is no longer necessary with the latest build of Godot.

I think it makes #19561 to be the biggest roadblock in creating semi-realistic human characters now.

[Lexpartizan]

About blendshapes...
If I understand correctly, you are also working on a character editor.
Maybe you can give me a solution to my problem.
This is related to the sharing of mesh resources you mentioned.

I would like to get a simple mesh without shapekeys from a large mesh with a hundred shapekeys.
That is, adjust the character with sliders and generate a simple mesh.
because with local resources a character with a hundred shapekeys takes up a lot of video memory.
10 character -1 Gb videomemory without textures.
So, I would like to generate a simple mesh with a specific character already prepared from a mesh common to all characters. Is it possible?

I can get a specific shape from the array, but how to apply all the keys and get the character, I do not understand.
And in the documentation there is practically no even an example with the establishment of the values of the shapekey.

[Anutrix]

2. Rather primitive support for subsurface scattering effect(SSS)
   Related issue: Subsurface scattering does not support transmittance. #11398

@mysticfall This one seems to be fixed now.

[mysticfall]

@Anutrix Thanks for the head up. I haven't been able to run my project on the current master for a while. But I'll try to find a way to sort the problems out and test it.

[Anutrix]

@mysticfall Any updates? Were you able to run your project in a recent master? #27283 was closed. But it's probably not fixed yet and would require a Godot Proposal.
Sidenote: An MRP for #26785 would be helpful.

[Calinou]

Also, #40364 should greatly help with hair rendering.

[mysticfall]

@Anutrix Sorry, I got a bit frustrated with exporter problems so I decided to put my project on hold to explore a more Blender centric approach(UPBGE) instead.

I still think that Godot is a great game engine, and I may even return to use it in future. But at the moment, I don't feel motivated to migrate my old code to test this issue, especially when most of my other issues got closed without being resolved.

[Calinou]

Closing in favor of #19561, since this is the only issue still open in the tracker (and this tracker is unlikely to be extended since feature proposals have moved).