New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: DarkPlaces compatibility: loads extra textures using suffixes #159

Open
wants to merge 5 commits into
base: master
from

Conversation

Projects
2 participants
@illwieckz
Copy link
Member

illwieckz commented Feb 9, 2019

Work in progress, I plan to disable it by default and to add a cvar to optionally enable it.

The purpose is to load extra maps (normal map, specular map…) the DarkPlaces way, using the diffuse map name as a basename then looking for files with predefined suffixes see DarkPlaces' code.

Unfortunately it does not work. That's why I publish my code the early way in hope someone with extra eyes sees what's wrong.

@illwieckz illwieckz added this to To do in Xonotic via automation Feb 9, 2019

@illwieckz illwieckz force-pushed the illwieckz:extramap branch from e999390 to badf5a7 Feb 9, 2019

@illwieckz

This comment has been minimized.

Copy link
Member Author

illwieckz commented Feb 9, 2019

By not working I mean the engine reports to find the extra texture files, but unfortunately it only renders diffuse maps.

@illwieckz illwieckz force-pushed the illwieckz:extramap branch 6 times, most recently from a22a173 to 6165bc8 Feb 9, 2019

Show resolved Hide resolved src/engine/renderer/tr_image.cpp Outdated
Show resolved Hide resolved src/engine/renderer/tr_image.cpp Outdated
Show resolved Hide resolved src/engine/renderer/tr_shader.cpp Outdated
Show resolved Hide resolved src/engine/renderer/tr_shader.cpp Outdated

@illwieckz illwieckz force-pushed the illwieckz:extramap branch from 6165bc8 to 0ad44e0 Feb 9, 2019

@illwieckz

This comment has been minimized.

Copy link
Member Author

illwieckz commented Feb 9, 2019

I think there is a special command to list opened files, does someone remember it?

@illwieckz

This comment has been minimized.

Copy link
Member Author

illwieckz commented Feb 9, 2019

For information this is how it looks on console:

Debug: looking for extra maps for color map: 'textures/phillipk2x/computer/pk02_switches01c' 
Debug: found extra glow map 'textures/phillipk2x/computer/pk02_switches01c_glow' 
Debug: found extra normal map 'textures/phillipk2x/computer/pk02_switches01c_norm' 
Debug: found extra specular map 'textures/phillipk2x/computer/pk02_switches01c_gloss' 

That's why it seems to work: the files are found. But what's painted on screen obviously proves that only diffusemap is rendered.

@illwieckz

This comment has been minimized.

Copy link
Member Author

illwieckz commented Feb 9, 2019

At this point the engine code claims to turn this:

textures/phillipk2x/computer-pk02_switches01c
{
    qer_editorimage textures/phillipk2x/computer/pk02_switches01c

    dpoffsetmapping - .68120712951256245813 match8 89.89403100000000000000
    dpglossintensitymod 3
    dpglossexponentmod 4

    q3map_bouncescale 1.25
    {
        map textures/phillipk2x/computer/pk02_switches01c
    }
    {
        map $lightmap
        rgbgen identity
        tcgen lightmap
        blendfunc filter
    }
}

into this (dp* keywords are ignored with an warning):

textures/phillipk2x/computer-pk02_switches01c
{
    qer_editorimage textures/phillipk2x/computer/pk02_switches01c

    q3map_bouncescale 1.25
    {
        map textures/phillipk2x/computer/pk02_switches01c
    }
    {
        map $lightmap
        rgbgen identity
        tcgen lightmap
        blendfunc filter
    }
    glowMap textures/phillipk2x/computer/pk02_switches01c_glow
    normalMap textures/phillipk2x/computer/pk02_switches01c_norm
    specularMap textures/phillipk2x/computer/pk02_switches01c_gloss
}

Note that the code runs after everything else is parsed, hence right before the ending }.
It means the code runs after the second stage (map $lightmap), perhaps that's the problem ?
I verified that glowMap, normalMap and specularMap are generated as stage 2, 3 and 4 (original ones being stages 0 and 1).

I also verified those files are loaded:

dds/textures/phillipk2x/computer/pk02_switches01c.dds
dds/textures/phillipk2x/computer/pk02_switches01c_gloss.dds
dds/textures/phillipk2x/computer/pk02_switches01c_glow.dds
dds/textures/phillipk2x/computer/pk02_switches01c_norm.dds

@illwieckz illwieckz force-pushed the illwieckz:extramap branch from 0ad44e0 to 628a323 Feb 9, 2019

@illwieckz

This comment has been minimized.

Copy link
Member Author

illwieckz commented Feb 9, 2019

To get a testbed, people may copy (or symlink) xonotic-20170401-maps.pk3 into a brand new directory, then run this code within that directory:

p="$(ls 'xonotic-'*'-maps.pk3' | tail -n 1)"
b='xonotic-maps'

listMaps () {
    unzip -l "${p}" \
    | tail -n +4 \
    | head -n -2 \
    | egrep '\.bsp$' \
    | sed -e 's|^.* maps/||;s|\.bsp$||'
}

filterSystem () {
    egrep -v '^_'
}

filterInvalid () {
    egrep -v '_'
}

ln -sfv "${p}" "${b}_0.dpk"

for m in $(listMaps | filterSystem | filterInvalid)
do
    d="map-${m}_0.dpkdir"
    mkdir -pv "${d}"
    echo "${b}" > "${d}/DEPS"
done

then run dæmon this way:

daemon -pakpath /path/to/xonotic/pkg \
    -set logs.logLevel.default debug \
    -set language en \
    -set developer 1 \
    +set g_neverEnd 1 \
    +devmap solarium

@illwieckz illwieckz force-pushed the illwieckz:extramap branch 3 times, most recently from c4f39bf to f206219 Feb 9, 2019

@illwieckz illwieckz force-pushed the illwieckz:extramap branch from f206219 to 1a857ad Feb 9, 2019

@illwieckz illwieckz referenced this pull request Feb 9, 2019

Open

Add list files command #160

R_FindImageLoader: walk the image loader list only once
I see no need to priorize dds/%s.dds over other formats in anyway,
it's a compatibility layer after all

@illwieckz illwieckz force-pushed the illwieckz:extramap branch from 1a857ad to ce34661 Feb 9, 2019

@illwieckz

This comment has been minimized.

Copy link
Member Author

illwieckz commented Feb 9, 2019

When I print numActiveStages the in CollapseStages() I get the expected amount of stages, for example 6 for this shader:

textures/exomorphx/light-metal_light03
{
	qer_editorimage textures/exomorphx/light/metal_light03
	dpoffsetmapping - 5.05836666682224888188 match8 160.48228200000000000000
	dpglossintensitymod  2
	dpglossexponentmod  4
	dpreflectcube env/exomorph/exomorph
	surfaceparm metalsteps
	q3map_bouncescale 1.50

	{
		map textures/exomorphx/light/metal_light03
	}

	{
		map $lightmap
		rgbGen identity
		tcGen lightmap
		blendfunc filter
	}
}

produces:

Warn: unknown general shader parameter 'dpoffsetmapping' in 'textures/exomorphx/light-metal_light03' 
Warn: unknown general shader parameter 'dpglossintensitymod' in 'textures/exomorphx/light-metal_light03' 
Warn: unknown general shader parameter 'dpglossexponentmod' in 'textures/exomorphx/light-metal_light03' 
Warn: unknown general shader parameter 'dpreflectcube' in 'textures/exomorphx/light-metal_light03' 
Debug: looking for extra maps for color map: 'textures/exomorphx/light/metal_light03' 
Debug: found extra glow map 'textures/exomorphx/light/metal_light03_glow' 
Debug: found extra normal map 'textures/exomorphx/light/metal_light03_norm' 
Debug: found extra specular map 'textures/exomorphx/light/metal_light03_gloss' 
Debug: active stage count: 5 

Five stages because of the two original stages plus the three computed ones:

glowMap textures/exomorphx/light/metal_light03_glow
normalMap textures/exomorphx/light/metal_light03_norm
specularMap textures/exomorphx/light/metal_light03_gloss

@illwieckz illwieckz force-pushed the illwieckz:extramap branch from ce34661 to 1c740d5 Feb 9, 2019

@illwieckz

This comment has been minimized.

Copy link
Member Author

illwieckz commented Feb 9, 2019

If I hack stuff by disabling the color map stage and creating a brand new diffuse map stage instead with the same texture path, it works, but it's a hack since we lose all the other keywords that may be there (like tcMod and so on), but it helped me to validate we have to turn the colormap into a diffusemap to get things working.

Note that the lighting is wrong because Xonotic uses sRGB lightmaps and Dæmon does not support them yet.

without hack

xonotic extramap

xonotic extramap

xonotic extramap

with hack

xonotic extramap

xonotic extramap

xonotic extramap

@illwieckz

This comment has been minimized.

Copy link
Member Author

illwieckz commented Feb 9, 2019

Note that since Xonotic ships a heightmap with almost all its normalmap, I enabled temporarily parallax on all shaders just to have a look. It works (and I haven't experienced preformance issue) but I discovered that xonotic heightmaps are upside down, or our engine is.

Perhaps it's a trick to not have to detect the presence of heightmap hidden in alpha channel of normalmap, as a missing alphachannel would mean a flat surface at the good level. Currently our engine displays a flat surface at the wrong level if we enable parallax on a normalmap without heightmap in alpha channel.

xonotic upside-down parallax

xonotic upside-down parallax

xonotic upside-down parallax

xonotic upside-down parallax

@illwieckz illwieckz referenced this pull request Feb 9, 2019

Open

Parallax mapping #30

0 of 4 tasks complete
@slipher

This comment has been minimized.

Copy link
Contributor

slipher commented Feb 10, 2019

If you actually write the second shader in this comment in a file, does it do what you want?

@illwieckz

This comment has been minimized.

Copy link
Member Author

illwieckz commented Feb 10, 2019

I have not tried but I'm now almost sure it won't work because this is a colorMap stage:

	{
		map textures/exomorphx/light/metal_light03
	}

and a diffuseMap stage looks like this:

	diffuseMap textures/exomorphx/light/metal_light03

or

	{
		stage diffuseMap
		map textures/exomorphx/light/metal_light03
	}

and the engine code is explicit: normalMap, glowMap, specularMap and others only work with diffuseMap stage, not colormap stage.

// try to merge diffuse/normal/specular/glow
if ( stages4.diffuseStage && stages4.normalStage && stages4.specularStage && stages4.glowStage )
{
tmpCollapseType = collapseType_t::COLLAPSE_lighting_DBSG;
shaderStage_t tmpStage = *stages4.diffuseStage;
tmpStage.type = stageType_t::ST_COLLAPSE_lighting_DBSG;
tmpStage.bundle[ TB_NORMALMAP ] = stages4.normalStage->bundle[ 0 ];
tmpStage.bundle[ TB_SPECULARMAP ] = stages4.specularStage->bundle[ 0 ];
tmpStage.specularExponentMin = stages4.specularStage->specularExponentMin;
tmpStage.specularExponentMax = stages4.specularStage->specularExponentMax;
tmpStage.bundle[ TB_GLOWMAP ] = stages4.glowStage->bundle[ 0 ];
stages[stagesWritten++] = tmpStage;
j += 4;
}

So what we need is to transform the colorMap into a diffuseMap. I don't know why but setting the diffuseMap bits in a colorMap stage does not do the trick. The hacky way that works is to disable the colorMap and load a brand new diffuseMap that is activated instead. That's how I produced those pictures above, the problem is that sometime those colorMap stages ship more information that are then lost.

For example the colorMap of the solarium water is this one:

	{
		map textures/map_solarium/water4/water4.tga
		tcmod scale 0.3 0.4
		tcMod scroll 0.05 0.05
		blendfunc add
		alphaGen vertex
	}

If I create a diffuseMap using the textures/map_solarium/water4/water4.tga path and disable the original colorMap, I just lost everything else that came after the name: wobbling, transparency…

Edit: I'm like 99.99% sure of what I just wrote in this comment.

@illwieckz

This comment has been minimized.

Copy link
Member Author

illwieckz commented Feb 10, 2019

I also think that changing the stage type bits later does not work because the image is already loaded, and I guess the bits must be set before the image loading. It's a strong guess but a guess. There is still things I have not read or not understood enough to say it's the issue, but there is a very high probability that the issue is there.

@illwieckz illwieckz force-pushed the illwieckz:extramap branch 2 times, most recently from 60466d5 to b5790c6 Feb 10, 2019

@illwieckz

This comment has been minimized.

Copy link
Member Author

illwieckz commented Feb 10, 2019

So, I pushed a new attempt. Instead of waiting the end of the shader parsing (and having to store the diffuse map name somewhere), the code does a quick lookup on the current stage before parsing it, if it finds a map without stage, it looks for extra maps, if it finds extra maps, it load them and set the current stage as a diffusemap one, then it parses the stage, so no one data is deleted and the stage is loaded as a diffuse one from the start.

This is how it looks now, it's not bad. Remind the lighting must be wrong since we don't have sRGB support yet. Also, it's hard to know whats wrong things comes from potential mistakes in the code or from unsupported features.

That's the best result I got without doing nasty stuff.

It's important to know we can get things that looks better but doing things that are wrong on purpose. I got things that looks better by disabling entire non-picture stages (i.e. disabling features), that's obviously wrong and can't count as reference.

xonotic extramap

xonotic extramap

xonotic extramap

xonotic extramap

xonotic extramap

xonotic extramap

xonotic extramap

xonotic extramap

xonotic extramap

xonotic extramap

xonotic extramap

xonotic extramap

xonotic extramap

@illwieckz illwieckz force-pushed the illwieckz:extramap branch from b5790c6 to ad9c323 Feb 10, 2019

@illwieckz

This comment has been minimized.

Copy link
Member Author

illwieckz commented Feb 10, 2019

So, currently this:

textures/phillipk2x/computer-pk02_switches01c
{
    qer_editorimage textures/phillipk2x/computer/pk02_switches01c

    dpoffsetmapping - .68120712951256245813 match8 89.89403100000000000000
    dpglossintensitymod 3
    dpglossexponentmod 4

    q3map_bouncescale 1.25
    {
        map textures/phillipk2x/computer/pk02_switches01c
    }
    {
        map $lightmap
        rgbgen identity
        tcgen lightmap
        blendfunc filter
    }
}

is turned into this:

textures/phillipk2x/computer-pk02_switches01c
{
    qer_editorimage textures/phillipk2x/computer/pk02_switches01c

    q3map_bouncescale 1.25
    normalMap textures/phillipk2x/computer/pk02_switches01c_norm
    glowMap textures/phillipk2x/computer/pk02_switches01c_glow
    specularMap textures/phillipk2x/computer/pk02_switches01c_gloss
    {
        stage diffuseMap
        map textures/phillipk2x/computer/pk02_switches01c
    }
    {
        map $lightmap
        rgbgen identity
        tcgen lightmap
        blendfunc filter
    }
}

and this:

textures/map_solarium/water4
{
	qer_editorimage textures/map_solarium/water4/water4.tga
	qer_trans 20
	surfaceparm nomarks
	surfaceparm trans 
	surfaceparm water
	surfaceparm nolightmap
	cull none
	q3map_globaltexture
	tessSize 256

	{
		map textures/map_solarium/water4/water4.tga
		tcmod scale 0.3 0.4
		tcMod scroll 0.05 0.05
		blendfunc add
		alphaGen vertex
	}
		dpreflectcube cubemaps/default/sky
	{
		map $lightmap
		blendfunc add
		tcGen lightmap
	}
	dp_water 0.1 1.2  1.4 0.7  1 1 1  1 1 1  0.1
}

into this:

textures/map_solarium/water4
{
	qer_editorimage textures/map_solarium/water4/water4.tga
	qer_trans 20
	surfaceparm nomarks
	surfaceparm trans 
	surfaceparm water
	surfaceparm nolightmap
	cull none
	q3map_globaltexture
	tessSize 256

	normalMap textures/map_solarium/water4/water4_norm
	specularMap textures/map_solarium/water4/water4_gloss
	{
		stage diffuseMap
		map textures/map_solarium/water4/water4.tga
		tcmod scale 0.3 0.4
		tcMod scroll 0.05 0.05
		blendfunc add
		alphaGen vertex
	}
	{
		map $lightmap
		blendfunc add
		tcGen lightmap
	}
}

The dpreflectcube keyword being a stage we can't expect this shader to be properly rendered since that stage line is entirely removed because of the unknown keyword.

@illwieckz illwieckz force-pushed the illwieckz:extramap branch from ad9c323 to cf9addd Feb 10, 2019

@illwieckz illwieckz changed the title WIP: DarkPlaces compatibility: loads extra textures using suffixes [does not work yet] WIP: DarkPlaces compatibility: loads extra textures using suffixes Feb 16, 2019

illwieckz added some commits Feb 9, 2019

R_UploadImage: detect heightmap in normalmap alphachanel
detect heightmap in normalmap alpha channel
and enable parallax

@illwieckz illwieckz force-pushed the illwieckz:extramap branch from bc3f6c3 to ee0763d Feb 17, 2019

SetUniform_DepthScale: reverse parallaxDepthScale hack
HACK: reverse parallaxDepthScale since it's
mistakenly reversed
@illwieckz

This comment has been minimized.

Copy link
Member Author

illwieckz commented Feb 17, 2019

I implemented the detection and auto enablement of heightmap in normalmap alpha channel.

I added a hack that reverses r_parallaxDepthScale, it looks to be an engine bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment