tr_shader: revamp the stage collapse code

[illwieckz]

• improve readability
• reduce cpu complexity
• make it extendable
• fix bug when surface with reflectionmap is not normalmapped
• fix lightmap being applied twice and glow map being lightmapped (note: lightmap alphaGen blend not yet supported)
• reduce the lighting stage type count to two: Phong (any combination including diffuse and something that is not material, including specular) and PBR (any combination including diffuse and material)
• fix the bug when a reflection type is applied to shader lighting type

[slipher]

You're confident that shader stages should never have any order dependency and can be arbitrarily shuffled? The old version only merges adjacent stages, while the proposed one merges them from anywhere.

[illwieckz]

You put the feet on it.

Basically, the initial quake3 material stages were order dependent. And there was no thing like a glow map, you just write a stage with a color map, then a stage that blend lightmap, then a stage that blend a color map. First color map is a diffuse map because you written it before lightmap, and second color map is a glow map because you written it after.

Once you use stage keywords like diffuseMap or glowMap, you're not only loading a color map, you're telling the engine when to load them. Also, the diffuseMap stage has an implicit lightmap stage, that's why this new code looks for explicit lightmap stage to collapse it instead. To render things properly, engine has to reorder things the way the engine is working internally and this must takes precedence over the way the material itself is written. Also the things like normalMap and specularMap are not order-relevant, they are sidecars for the diffuseMap.

Basically you set a diffuseMap file, a specularMap file, a normalMap file, a glowMap file, optionally a lightmap stage to configure the lightMap stage, and the engine knows what to do and in which order.

A glow map rendered before a diffuse map is wrong (diffuse map will be painted over and hide it), a glow map rendered before a light map stage is wrong (shadows will be painted over the glow map, but the purpose of a glow map is to not be affected by shadows), a light map before a diffuse map is wrong (diffuse map with be painted over and hide it).

The only situation were order would be required, is that if we were able to load two diffuseMap stage, or two specularStage, or two normalMap stage, or two glowMap stage, for example to blend two glowMap stages as one. I really doubt the engine supports such things, as the code seems to send one diffuse/specular/normal/glow map per pixel to glsl shaders. If I'm right the material language is able to load as map the result of another material, this another material may be able to blend. I'm not sure if this work, but that would be the correct way to merge glow maps for example.

Basically the somethingStage thing is deeply changing the way material works: people just fill the blank with required stuff, the engine knows how to use them and when. It's just a form-based material. It's still possible to do complex materials by somewhat programming them with staging and blend operations, but the somethingStage is designed from the start to do it the fill the blank way.

Part of this PR is to ensure stages are reordered the way the engine expects them, that's the only way to fix some bugs in any way.

So, if someone can prove me that we can put two diffuseMap stage in a material, please talk, it looks to have been designed against that. :-)

[illwieckz]

note that this is still ordered: the existing stages that are not collapsed together are still before/after the collapsed one.

The question is: does it makes sense to set multiple time the same stage: for diffuse/normal/specular/light/glow: it does not make sense.

We seen in the code that reflection map may be blended with a normal map, and diffuse map may be blended with a normal map, current code assumes it's the same, perhaps it's wrong and we can set two normal maps, one for reflection, one for diffuse, in that case I would look at it, and that would be the only exception.

[illwieckz]

Also one note: the heightmap in normalMap stage must be applied on both diffuse, specular and glow maps, there is no such things as ordering in that process (and there must not be).

[slipher]

Can we get a warning if the shader contains more than one of any of these kinds of stage?

[illwieckz]

That's doable, the original code was already looking for only one kind of these stage and ignore others once first was found.

[illwieckz]

I added the warning.

I also reduced the light stage merge to just a disablement (as there is already an implicit lightmap stage within diffuse stage), the problem is that I don't know how to pass the blend option. So it fixes like 90% (or more) of the double-lightmap and lightmap-over-glowmap issues. To fix the remaining ones (if they exist), more underlying design may have to be done and that would be better to do it in another PR. The current disablement fixes many bugs but does not add some, it just lefts some.

[illwieckz]

I have to figure out if there is more than one way to blend lightmap. Perhaps not, perhaps all the rare variations I see are just mistake or equivalences.

If I look at the 924 lightmap stages in xonotic maps, they all (but some) use the same stage:

{
	map $lightmap
	rgbGen identity
	tcGen lightmap
}

There is 4 differences in map_erbium.shader but they all are mistakes, for example this one:

{
	map $lightmap
	map textures/map_erbium/fern
	blendFunc GL_SRC_ALPHA GL_ONE_MINUS_SRC_ALPHA
	rgbgen identity
}

is a mistake that mixes two stage in one, that must never happen. The blendFunc is for the texture, not for the lightmap.

Then there if I remove the faulty erbium ones, I fond 17 lightmap stage that use a blendFunc, with those operations: blend (2 of them), add (4 of them), GL_DST_COLOR GL_ZERO (11 of them). So on a corpus like Xonotic, the number of custom blendFunc in lightmap stage that may be legit are around 1%.

This is an example of them (in liquids_water.shader):

textures/liquids_water/water0_cubemap_alphamod
{
	qer_trans 20
	surfaceparm nomarks
	surfaceparm trans
	surfaceparm water
	surfaceparm nolightmap
	cull none
	q3map_globaltexture
	//tessSize 256
	qer_editorimage textures/liquids_water/water0c_reflect.tga
	{
		map textures/liquids_water/water0c.tga
		tcMod turb 0 0.4 0 0.08
		blendfunc GL_SRC_ALPHA GL_ONE
		alphaGen vertex
	}
	dpreflectcube cubemaps/default/sky
	{
		map $lightmap
		blendfunc add
		tcGen lightmap
	}
	dp_water 0.1 0.9  3 3  1 1 1  1 1 1  0.4
}

The dpreflectcube is similar to our reflectionMapBlended. The three other shaders with blendFunc add are water shaders written the same (in same .shader file or in map_solarium.shader). And they all use a color map and reflection cubemap. Basically the engine does not need to parse the lightmap, it already knows enough. The remaining question is: is dæmon does the right thing with reflectionMapBlended and lightmaps or not. Notice the tcGen lightmap.

Then the other example is this one (in glassx.shader):

textures/glassx/hexglass
{
	q3map_bouncescale 3
	qer_editorimage textures/glassx/base/hexglass.tga
	dpreflectcube cubemaps/default/sky
	dpnoshadow
	surfaceparm trans
	surfaceparm alphashadow
	q3map_lightmapSampleSize 64
 	{
		map textures/glassx/base/hexglass.tga
		blendfunc blend
		rgbgen lightingDiffuse
	}
	{
		map $lightmap
		blendfunc GL_DST_COLOR GL_ZERO
		rgbGen identity
	}
}

It's a very similar shader to the water one, with a color map and a blended reflection map. I don't know what does blendfunc GL_DST_COLOR GL_ZERO at this point. Notice the rgbGen identity.

Then we have this (in model_trak.shader):

models/trak/servera
{
	dpreflectcube cubemaps/default/sky
 	{
		animMap 5 models/trak/servera1.tga models/trak/servera2.tga models/trak/servera3.tga models/trak/servera4.tga 
	}
	{
		map $lightmap
		blendfunc GL_DST_COLOR GL_ZERO
		rgbGen identity
    }
}

I don't know what it does, there is only two of them. There is one similar (but a bit different) in model_crate02.shader and another similar (but a bit different) in model_desertfactory.shader.

Then in terrain01x.shader we have 6 terrain shaders that looks almost the same, like this one:

textures/terrain01x/blends-mars-rock01-sand04
{
	qer_editorimage textures/terrain01x/blends/mars-rock01-sand04.tga
	
	q3map_bounceScale  0.5
	dpoffsetmapping - 2 match8 229

	q3map_lightmapSampleOffset 8
	q3map_nonplanar
	q3map_shadeangle 95

	surfaceparm dust

	q3map_alphaMod dotproduct2 ( 0 0 0.95 )

	{
		map textures/terrain01x/rock/mars01.tga	// Primary
	}

	{
		map textures/terrain01x/ground/sand04.tga	// Secondary
		blendFunc GL_SRC_ALPHA GL_ONE_MINUS_SRC_ALPHA		
		alphaGen vertex
	}

	{
		map $lightmap
		blendFunc GL_DST_COLOR GL_ZERO
	}
}

If those variations are legit I have a prototype that disables the implicit lightmap stage within diffuse stage and sort the stages the way an eventual diffuse stage is always before the lightmap stage and an eventual glow stage is always after the lightmap stage. With proper tests this would only match on those custom shaders and 90% of them will use the implicit lightmap stage of the diffuse one which is very good for performances. The real question is:are those custom lightmap stage legits? Adding those tests and the resorting is adding much noise to the code, but is doable.

On a side note that the terrain blending shader uses two color maps, no diffuse stage. It's the only supported way at this time and we hence don't support normalmap, parallax and other feature on such shader.

We also have a lot of those variants, much more than xonotic: 161 / 1222 (305 custom lightmaps + 917 diffuse maps), with blendfunc filter in parpax_custom.shader for example:

textures/parpax_custom/elebutton
{
	qer_editorImage textures/parpax_custom_src/elebutton_d
	surfaceparm nomarks

	{
		map textures/parpax_custom_src/elebutton_d
	}
	{
		map $lightmap
		blendfunc filter
	}
}

or in vega_custom.shader:

textures/vega_custom/vega_banner
{
	qer_editorImage textures/vega_custom_src/vega_banner_b
	qer_alphaFunc greater .5

	surfaceparm nomarks
	surfaceparm nonsolid

	polygonOffset
	sort decal
	noShadows

	{
		map textures/vega_custom_src/vega_banner_b
		alphafunc GE128
		depthwrite
		rgbGen identity
	}
	{
		rgbGen identity
		map $lightmap
		depthfunc equal
		blendFunc filter
	}
}

Or this in shared_vega.shader (another glass materiale):

textures/shared_vega/glass01
{
	qer_editorImage textures/shared_vega_src/glass01_b
	qer_trans .7

	cull none

	{
		map textures/shared_vega_src/glass01_env
		blendFunc gl_dst_color gl_one
		tcGen environment
	}
	{
		map textures/shared_vega_src/glass01_b
		blend filter
	}
	{
		map $lightmap
		blendFunc gl_dst_color gl_one
	}
}

Or this two others in thunder_custom.shader (glass and terrain):

textures/thunder_custom/nexusglass
{
	qer_editorImage textures/thunder_custom_src/nexusglass_d
	qer_trans .4
	surfaceparm trans
	cull none
	{
		map textures/thunder_custom_src/nexusglass_d
		blendfunc add
		rgbGen const ( .01 .01 .01 )
	}
	{
		map $lightmap 
		blendfunc gl_dst_color gl_src_alpha
		rgbGen identity
		tcGen lightmap 
	}
}

textures/thunder_custom/ter_rocksand_xy
{
	qer_editorImage textures/thunder_custom_src/ter_rocksand_p
	q3map_nonplanar
	q3map_shadeAngle 90
	q3map_tcGen ivector ( 256 0 0 ) ( 0 256 0 )
	q3map_alphaMod dotproduct2 ( 0 0 .75 )

	{
		map textures/shared_pk02_src/rock01_d
		rgbGen identity
	}

	{
		map textures/shared_pk02_src/sand01_d
		blendFunc GL_SRC_ALPHA GL_ONE_MINUS_SRC_ALPHA
		alphaFunc GE128
		rgbGen identity
		alphaGen vertex
	}

	{
		map $lightmap
		blendFunc GL_DST_COLOR GL_ZERO
		rgbGen identity
	}
}

I also found this on station15_custom.shader (no terrain, no glass):

textures/station15_custom/pipe_nonsolid
{
	surfaceparm nonsolid
	surfaceparm noimpact
	surfaceparm nomarks
	qer_editorImage textures/station15_custom_src/e6bmetal_d
	{
		map textures/station15_custom_src/e6bmetal_d
		rgbGen identity
	}
	{
		map $lightmap
		rgbGen identity
		blendfunc filter
	}
}

So, we can clearly see patterns for some various usages and that looks legit. Our shaders seems to make the effort to not use diffuseMap with those custom lightmaps, that's also why our maps are less likely than xonotic ones to help us to make dæmon better because they already avoid the traps.

It means we don't support normal/specular/parallax on shaders that need custom lightmaps while DarkPlaces seems to.

[illwieckz]

It looks like the common lightmap stage is:

{
	map $lightmap
	rgbGen identity
	tcGen lightmap
	blendfunc filter
}

For some reason the default rgbGen is CGEN_BAD if not defined so I don't know what to do with lightmap stages that miss it. I may already test for blendFunc not being filter to trigger the disablement of implicit lightmap in diffuseMap and so on.

Edit: there is a special code to set to CGEN_IDENTITY or CGEN_IDENTITY_LIGHTING if undefined:

Daemon/src/engine/renderer/tr_shader.cpp

Lines 2509 to 2520 in e920d3a

	// if cgen isn't explicitly specified, use either identity or identitylighting
	if ( stage->rgbGen == colorGen_t::CGEN_BAD )
	{
	if ( blendSrcBits == 0 || blendSrcBits == GLS_SRCBLEND_ONE || blendSrcBits == GLS_SRCBLEND_SRC_ALPHA )
	{
	stage->rgbGen = colorGen_t::CGEN_IDENTITY_LIGHTING;
	}
	else
	{
	stage->rgbGen = colorGen_t::CGEN_IDENTITY;
	}
	}

So it looks like only “filter” as to be tested.

[illwieckz]

So I added test for custom lightmap and then makes light stage and glow stage standalone and ordered after collapsed diffuse stage.

Since most of the light stages are common ones and since most of custom stages comes with stuff like dpreflectmap (reflectionMapBlended) that do not collapse light stage in any way, it will be very rare to walk the diffuse collapsed stage + standalone light stage + standalone glow stage path in real life, so there is no worry to have about performances.

[illwieckz]

Note that I don't like this code at all and I would prefer to see the lightmap blending being done in the diffuse collapsed stage instead.

[illwieckz]

After some deep thoughts I think that a lot of the tr_shader code may be revamped as many problems are just there because of design issue. This whole collapse thing is just a workaround for a design issue. A better design would be that diffuseMap/specularMap etc. would not be stages, but component of a stage. If those keywords are found outside of a stage, they are added to a default stage and then would not require any collapse at all.

Basically if that is found:

textures_src/something
{
	diffuseMap texture/something_d
	normalMap texture/something_n
	specularMap texture/something_s
	glowMap texture/something_g
}

It would be interpreted as:

textures_src/something
{
	{
		stage collapseDiffuseLighting
		diffuseMap texture/something_d
		normalMap texture/something_n
		specularMap texture/something_s
		glowMap texture/something_g
	}
}

It would not only obsolete the collapse computation, it would make possible to do one day a terrain shader this way:

textures/thunder_custom/ter_rocksand_xz
{
	qer_editorImage textures/thunder_custom_src/ter_rocksand_p
	q3map_nonplanar
	q3map_shadeAngle 90
	q3map_tcGen ivector ( 256 0 0 ) ( 0 0 256 )
	q3map_alphaMod dotproduct2 ( 0 0 .75 )

	{
		stage collapseDiffuse
		diffuseMap textures/shared_pk02_src/rock01_d
		normalMap textures/shared_pk02_src/rock01_n
		heightMap textures/shared_pk02_src/rock01_h
		specularMap textures/shared_pk02_src/rock01_s
	}
	{
		stage collapseDiffuse
		diffuseMap textures/shared_pk02_src/sand01_d
		normalMap textures/shared_pk02_src/sand01_n
		heightMap textures/shared_pk02_src/sand01_h
		specularMap textures/shared_pk02_src/sand01_s

		blendFunc GL_SRC_ALPHA GL_ONE_MINUS_SRC_ALPHA
		alphaFunc GE128
		alphaGen vertex
	}
	{
		stage lightMap
	}
}

But this is not for today, let's be iterative and the current code is good enough even if I dream of something better. In any way the glsl code is not yet ready for such dream. 😁

[illwieckz]

To illustrate the double lightmap and overpainted glow map in Unvanquised, you can look at the example in #186

These is another example with a xonotic map, how it looks when glow map is explicitely disabled (there is double lightmap over diffuse maps):

This is how it looks with glow map enabled, it get repainted with the second lightmap (there is double lightmap, before and after glow map):

This is now with fixed collapse code, with glow map enabled (one lightmap before glow map):

This is an example of problem with previous stage collapsing code, this texture has both diffuse, normal, specular, height and reflection map, you'll notice that only the reflection map with the glassy holes is correctly applying reflection map itself, the normalmap and the heightmap. The other collapse stage (diffuse, normal, specular) is disabled for the solid part of that same shader:

This is how it looks when fixed:

Also notice how the whole map is also more brighter because of the double-lightmap being fixed.

Note that I don't exclude that the glassy holes may be a bug themselves, I don't see them in xonotic (but their materials have a lot of mistakes, and perhaps the cubemap reflection parallax feature is not in DarkPlaces), but at least it produces what the material says and we can confirm that reflection maps get properly normalmapped and parallaxed.