Object Space Normal Maps?

[mlimper]

Hi,

we were wondering if there were any requests for object-space normal maps, and whether it could make sense to include those in the next version of glTF.

Why / where could we need those?
There can be different reasons to favor object space normal maps instead of tangent space ones.
Some are outlined and illustrated in this forum post, but also in this documentation from the CryEngine (scroll down for an example comparing the effect on seams / edges, where it says "Drawbacks of Tangent Space Lighting").

In general, tangent space normals offer better compression and re-usability of textures for different parts of the same object. But some of our customers, for example, are using our optimization pipeline on organic, scanned 3D objects, and we bake maps for Web-ready simplified versions of those. For use cases like this, the re-usability argument for tangent space maps becomes irrelevant. The remaining advantage, better compression, has to be weighted against the drawbacks of tangent space maps, such as visible seams due to filtering issues on edges, and the usual issue that the rendering tool will need to use exactly the same tangent spaces as the baking tool everywhere, in order to obtain the best (correct) result. The latter can be really annoying, also since many commonly used formats such as OBJ are lacking the ability to store tangents. Some tools export the mesh to OBJ and generate a tangent space normal map, but without providing actual tangents... it is then always up to the rendering crew to figure out which tangents were originally used for baking. And some renderers may even ignore stored tangents and compute them in the shader (#1252).

To give a live example from the real world: Smithonian's 3D viewer, showing 3D scans, uses object space normal maps.

So, there could be reasons to favor object space normals over tangent space maps, for certain kinds of workflows / applications.

Now, regarding glTF integration:
For the core PBR material we discussed this aspect earlier (#699 (comment)). The outcome was that we were unsure and we wanted to keep the spec simple and straightforward, so in the end we checked what Three.JS did, and they didn't support object space normals at that point - so we excluded them.

The core PBR material's normal maps already have a scale factor as an additional attribute. So, allowing for object space normals would mean just another additional boolean attribute, I guess. Implementation-wise, on the client side, it should be rather simple (I recently did it myself for BabylonJS: BabylonJS/Babylon.js#3706). And for content generation tools, especially when we are dealing with data from 3D scanning (no artist creating maps, no re-use of maps), having support for object space normals could simplify workflows and even provide better visual results (since a lot of error sources related to tangent space incompatiblities are elimintated). However, as always, I don't want to advertise anything too strongly, but rather get the opinion of the community on this topic.

With the first experiences made in practice with the 2.0 PBR material and associated workflows, has anyone heard about similar requests for supporting object space normal maps? What do you think about the trade-off between utility and implementation complexity?

Thanks in advance!

[pailhead]

@mlimper

With the first experiences made in practice with the 2.0 PBR material and associated workflows,

Out of curiosity why are you tying this to a lighting model (PBR) and a material description? I always saw this as not really part of a material, but rather geometry. If an artist or software bakes some high poly mesh into a low poly mesh using a normal map, they opt for either tangent or object space. The geometry is most likely going to be different in both cases since UVs can be laid out differently.

Imagine you have a gold statue. The PBR Material describing the gold should have the same properties in both:

1. Artist simplified, mapped and baked tangent space
2. Software scanned, decimated baked object space

Both have normal maps, and normal maps can be of different types / encoding.

so in the end we checked what Three.JS did, and they didn't support object space normals at that point

Three.JS does not support baked tangent space normal maps for that matter either, never has.

(are object space normal maps ever generated without the baking process?)

[emackey]

Baked tangent-space normal maps are known to work in ThreeJS. Demo.

[pailhead]

Maybe i'm using the term "baked" wrong? This is a hemi-sphere thats projected onto a plane, i think that this is a subset of the problem and this can work regardless of the tangent space mismatch. I'm not sure if this counts as baking, or most of what baking does. Maybe if "baking" is limited to "meshes with smoothing", "meshes with smoothing and seams".

If you try to bake a map onto something more complex than a two triangle quad you will get artifacts.

Artifacts with tangent space, no artifacts with object space:
http://dusanbosnjak.com/test/webGL/three-normalmap-space/

Artifacts with three's tangent space, no artifacts with synced tangent space:
http://192.241.199.119:8080/dev/testwp/

Demo.

A proper example for baking could possibly be high poly sphere -> low poly sphere not sphere -> quad.

Texture baking is the process of transferring details from one model to another. link

In the spheres demo above, the baking process correctly transferred this detail to the other mesh, and appropriate textures (tangent and object space normal maps) it's just that three.js, is unable to render the tangent ones.

I think the origin of the normal map may be just as important as space/encoding stuff, since it couples the geometry to the texture.

[mlimper]

Out of curiosity why are you tying this to a lighting model (PBR) and a material description? I always saw this as not really part of a material, but rather geometry

You are perfectly right! And the normal map is not part of the PBR material, in fact (similar to occlusion map and emissive map):
https://github.com/KhronosGroup/glTF/tree/master/specification/2.0#additional-maps

We didn't try to tie it to the PBR model, in fact, but the topic was discussed at the same time (that's why that issue was called "Additional Maps")... my post above was a bit mistakeable w.r.t. that topic. Normals are describing mesh geometry, independent from a particular material model.

[emackey]

This is a hemi-sphere thats projected onto a plane

Not to split hairs, certainly this isn't a perfect example of baking, but I did "bake" this normal map. Specifically, I built an actual hemisphere polygon mesh in Blender, and then used Substance Painter's "Bake Textures" option to bake it onto a quad. Some artifacts did result from this, I think because I used an actual quad where in retrospect I should have triangulated it first. But my goal here was just to test a high number of different directions for texture-supplied normal vectors from a single polygon normal vector, so I thought a hemisphere on a quad would be a good way to get that test case.

Certainly, real-world use of baking is much more complex than just one shape onto a quad, but this was meant to be a reduced test case. Someday I should go back and re-bake it to see if I can get rid of the remaining artifacts.

[pailhead]

Specifically, I built an actual hemisphere polygon mesh in Blender, and then used Substance Painter's "Bake Textures" option to bake it onto a quad.

I meant no offense but i think that it is actually very important to split hairs here.

Just because the button had the word "bake" in it, i don't think that the process that ensues necessarily counts as "baking". Or at least not related to the family of problems that you usually encounter while baking.

My logic here is that a baker needs to consider tangent spaces, cages, uv splits, smoothing splits, etc. hence we have all these specialized tools within blenders, 3d studios, etc, and even some standalone ones like x-normal that is completely built around the concept of baking.

This sphere onto a plane you could have done in a few different ways without requiring a baker. It's possible to set up a scene and some geometry and light it in such a way that a regular render results in a normal map. Various renderers (like vray) can output a normal map channel out of any rendering.
A tool for Photoshop such as nDO can convert a black and white image of a blob into the same normal map you generated.

I'm not entirely sure what artifacts you are referring to, but chances are that you may remove them in this simple test case, but as soon as you try baking a higher resolution sphere onto a lower resolution sphere it will fail.

With all of this being said i'm still curious how (and maybe why) this should be part of glTF?

[pailhead]

While an obvious solution may be to have something like normalSpace on some texture, the usage of tangent vs object normal maps results in different meshes. One is a mesh that requires no normals (since they're completely derived from an object space normal map) another is the one that not only requires normals, but most likely additional attributes (at least tangent) or a hint at them.