Add parallax mapping to bevy PBR

[nicopap]

Objective

Add a parallax mapping shader to bevy. Please note that
this is a 3d technique, NOT a 2d sidescroller feature.

Solution

• Add related fields to StandardMaterial
• update the pbr shader
• Add an example taking advantage of parallax mapping

A pre-existing implementation exists at:
https://github.com/nicopap/bevy_mod_paramap/

The implementation is derived from:
https://web.archive.org/web/20150419215321/http://sunandblackcat.com/tipFullView.php?l=eng&topicid=28

Further discussion on literature is found in the bevy_mod_paramap README.

Limitations

• The mesh silhouette isn't affected by the depth map.
• The depth of the pixel does not reflect its visual position, resulting
  in artifacts for depth-dependent features such as fog or SSAO
• GLTF does not define a height map texture, so somehow the user will
  always need to work around this limitation, though an extension is in
  the works

Future work

• It's possible to update the depth in the depth buffer to follow the
  parallaxed texture. This would enable interop with depth-based
  visual effects, it also allows discarding pixels of materials when
  computed depth is higher than the one in depth buffer
• Cheap lower quality single-sample method using offset limiting
• Add distance fading, to disable parallaxing (relatively expensive)
  on distant objects
• GLTF extension to allow defining height maps. Or a workaround
  implemented through a blender plugin to the GLTF exporter that
  uses the extras field to add height map.
• Quadratic surface vertex attributes to enable parallax
  mapping on bending surfaces and allow clean silhouetting.
• noise based sampling, to limit the pancake artifacts.
• Cone mapping (GPU gems, Simcity (2013)). Requires
  preprocessing, increase depth map size, reduces sample count greatly.
• Quadtree parallax mapping (also requires preprocessing)
• Self-shadowing of parallax-mapped surfaces by modifying the shadow map
• Generate depth map from normal map link to slides, blender question

2023-03-07.23-02-45.mp4

---

Changelog

• Add a depth_map field to the StandardMaterial, it is a grayscale
  image where white represents bottom and black the top. If depth_map
  is set, bevy's pbr shader will use it to do parallax mapping to
  give an increased feel of depth to the material. This is similar to a
  displacement map, but with infinite precision at fairly low cost.
• The fields parallax_mapping_method, parallax_depth_scale and
  max_parallax_layer_count allow finer grained control over the
  behavior of the parallax shader.
• Add the parallax_mapping example to show off the effect.

[nicopap]

Demo available at https://nicopap.github.io/bevy_mod_paramap/

[nicopap]

A few limitations to take into considerations:

• The height map is inverted compared to what usually people would expect
• Self Shadowing is not implemented, which results in awkward looks with shadow-enabled light sources. Not sure how hard it is to implement it.
• The earth example looks awkward at the poles, not sure why and if it is something that will generally show up

[schrottkatze]

The height map is inverted compared to what usually people would expect

Godot has the height map inverted too, but solves the confusion by simply calling it a depth map.

Self Shadowing is not implemented, which results in awkward looks with shadow-enabled light sources. Not sure how hard it is to implement it.

Godot doesn't seem to have that either, they suggest just using normal maps for that.

The earth example looks awkward at the poles, not sure why and if it is something that will generally show up

From what the parallax mapping that I've seen in other places so far, that seems to be a general limitation of parallax mapping if used on spheres.

I'd love if this gets some more attention since parallax mapping generally looks amazing if it's used correctly!

[JMS55]

Haven't looked at the PR beyond quickly skimming the code, but I'd like more documentation from the user's perspective. I'm not familiar with parallax mapping - it's some kind of normal map like technique? The StandardMaterial fields should have some documentation explaining what parallax mapping is, briefly how it differs from normal maps, and when/why you would want to use it :)

[superdump]

Mostly minor changes to clean things up. One minus sign that seems out of place in bitangent calculation. And maybe making the number of Relief Mapping steps configurable.

[superdump]

For those following along, this stalled a bit as I wanted to understand from where the -B was coming in the transformation of the view vector to camera space. And, I wanted to properly understand the details of the algorithm.

As I dug into it, I understood that the -B was because our bitangent calculation follows our right-handed y-up convention in tangent space, which means the bitangent points up. However, in texture coordinates (UVs) the V direction goes down. I modified this to use B but then flip the tangent space view vector's y component when calculating the delta_uv.

Also, I felt it would be more intuitive for understanding the algorithm to have the tangent space view vector pointing into the surface and step along it through the depth layers. By passing -Vt into the parallax_uv function and adjusting the calculation of delta_uv to divide by view_steepness (which is abs(Vt.z) instead of the signed Vt.z, the rest of the code could then be modified to step forward (as in + delta_uv is forwards) along the tangent-space view vector that points from the camera to the fragment.

And finally, because the texture we use has a value of 0 meaning no depth below the geometric surface, and 1 meaning maximum depth, I felt it appropriate to rename height to depth everywhere in the code. Also renamed parallax_depth to parallax_depth_scale to make it clearer that this parameter scales the depth effect.

PR here: nicopap#1

[superdump]

@robtfm I feel like requiring jpeg is better than using more bytes in the repo, but it's not a particularly strong opinion. Good idea about '.linear.jpeg' and I guess '.linear.png', though it's a hack and I hope the asset rework will include metadata files that will make this no-longer a problem. :)

[mockersf]

@robtfm I feel like requiring jpeg is better than using more bytes in the repo, but it's not a particularly strong opinion.

Not requiring a feature would be great... and worth the extra 100KB for me. Most of the size increase comes from the first one, we could make it smaller if that's still an issue for you

Here are pngs that aren't a lot bigger:

[superdump]

Also the examples failed on windows saying something about the lip falling to complete after 1009 iterations. When testing locally I did notice a little stuttering but I was doing other stuff on my laptop so I didn’t think anything of it. Though I did wonder if it was related to tracing rays at glancing angles.

[nicopap]

@mockersf I have the clean base assets I can generate better pngs. Note that when I run cargo run --example parallax_mapping, cargo just refuses to run the example and prints out in the terminal the following:

error: target `parallax_mapping` in package `bevy` requires the features: `jpeg`
Consider enabling them by passing, e.g., `--features="jpeg"`

So in terms of user experience, this is fine IMO, as the solution is directly given to the user. And in any case, teaching them to use the proper bevy features is better in an environment where failure to do so results in the program not running rather than assets mysteriously not loading.

But you are probably worrying about compilation time of examples right? How much of a worry is this?

@superdump Ahah! I happen to only have a very weak Windows machine with a very not modern GPU on hand, so it's going to be hard for me to test out. It seems possible to trigger nagga compilation for Dx12 on linux though? Just for checking? How would I go about that?

[mockersf]

Note that when I run cargo run --example parallax_mapping, cargo just refuses to run the example and prints out in the terminal the following:

error: target `parallax_mapping` in package `bevy` requires the features: `jpeg`
Consider enabling them by passing, e.g., `--features="jpeg"`

So in terms of user experience, this is fine IMO, as the solution is directly given to the user. And in any case, teaching them to use the proper bevy features is better in an environment where failure to do so results in the program not running rather than assets mysteriously not loading.

But you are probably worrying about compilation time of examples right? How much of a worry is this?

Yup it's mostly fine... but from a user point of view, it means they can't simply try one example after the other, they will have recompilation when switching between examples with different required features.

From a CI point of view, it means we'll have to add special case when building examples for tests in CI, or for building the website example page. Or for https://rparrett.github.io/prototype_bevy_example_runner/

Unless really necessary, not requiring a feature for any example is simpler.

[nicopap]

You make a compelling case. I'll swap the assets for pngs.

[nicopap]

Todo when/if this gets merged: Open a tracking issue on possible parallax mapping improvements, ordered by complexity, from easiest to most complex.

1. Distance fading
2. offset limiting method (inaccurate, fast and predictable)
3. Depth buffer update (SSAO, proper shadow maps, proper silhouette)
4. Depth map generation from normal map
5. glTF extras non-standard parallax mapping extension.
6. cone mapping method & cone map generation
7. Quadratic surface vertex attribute generation & usage (parallax mapping on curved surfaces)