Vulkan Ray Traced Pipeline

[pollend]

I was considering what the interest would be if i were to add a some hybrid lighting model. I'm not sure what that would be like though and what technologies would be best. I currently have this working for Amnesia the Dark Descent but had to gut the opengl backend because it coupled very deeply into how the graphics part of the codebase worked. something like this would probably need to move a lot of the data to a bindless architecture. So ill have to bind all the textures/meshes into a couple large ssbo. assuming this was built with opengl in mind there is a lot of state driven logic so i don't think a conversion like this will be straightforward.

In terms of the methods for direct lighting im using Restir and a surfel radiance cache for indirect lighting. Not sure if something else would be better wouldn't mind looking into some different methods to achieve a lighting model that fits better with this
engine.

I attached a couple of images of this working in my other project. I don't want to promise anything since this is a fairly complex process that will take quite a while to get something stable. How this fits in with the rest of the tooling is a question i would need to explore more. I just wanted to share my thoughts and see if there is any interest.

[dpogue]

The game and its content were originally designed for a DirectX 8 pipeline with no ray tracing or per-pixel shading support, so a lot of the lighting is baked as vertex colours into the geometry data without corresponding in-game light sources. During the development of the Metal and OpenGL rendering pipelines, per-pixel lighting was tested but found to yield different rendering results than the expected per-vertex lighting.

We wouldn't be opposed to a Vulkan rendering pipeline, but the expectation would be for it to visually match the existing Direct3D9 rendering output.

[colincornaby]

The pipeline is also very much not bindless right now. D3D9 doesn't support bindless rendering. The Metal renderer has bindless rendering on a branch - and I'm hoping to get it integrated but it has not been so far. Even then - the bindless pipeline is limited to just the material portions of rendering. I'm hoping to get an architecture defined to allow bindless rendering of both meshes and materials together. But the pipeline has several tricks it plays that modify a lot of resources on the fly which makes it trickier to support bindless. And the pipeline doesn't have a clean linkage of materials and meshes that provide an easy storage point for bindless resources.

Ray tracing has not been considered for the bindless Metal pipeline because of the light baking in the textures - as @dpogue mentioned.

[pollend]

so the light data is just baked into the vertex data, so the lights are essential compiled out right? something like this would probably just discard that and rebuild new authored lights ...

[dpogue]

There are a few lamps in the game data to light dynamic objects like the avatar, but static objects by and large do not participate in runtime lighting.

[colincornaby]

Im just assuming the lightmap are done similar to quake where you have some repeated texture and a packed texture that has the baked light data. Amnesia does the same thing i just had to gut the whole thing to make it even possible. The restir style singe bounce lights look incredible.

I'm not sure if the artists followed any hard and fast rules there. Do you know @dpogue if baked lighting is generally encoded down into the base texture or if it's a separate layer?

(Ah, I see in the meantime you've responded.)

I feel like I've seen it mostly baked down but I can't think of specific examples. My guess is they would baked it down to better support cards with limited texture units. The background is that Plasma supported cards with up to eight texture/blend units - but also supported cards with much less. So they needed to be economical with how many textures and blends they used for a single material.

There's a few other baked light setups that I'm aware of:

• God rays using baked textures
• Light maps that are overlays but do not have a light source. (An example is the light in the link in area in Gahreesen. There is technically a light caster in the engine but it renders as a projected texture.)
• Shadow map overlays that have no real source (for example: Kadish Tolesa has shadows cast by branches that do not have any real geometry in the scene.)

If you're interested in working on the engine, having Vulkan rendering working at all would be a good first step, or perhaps contributing to the GL renderer. The Metal renderer has been the first renderer that was not fixed function and it was a painful transition. I'm always excited to get more support in the engine for doing things like bindless rendering.

One place ray tracing could be workable is in the water shaders. They use cubic texture maps right now that approximate reflections, but I could imagine them looking nice with some ray tracing. But that would require a new renderer unless you were interested in exploring that in Metal.

[pollend]

I'm thinking of writing some tooling to unpack the compiled assets and rework the maps but might breaks backwards compatibility. I'll probably explore my options on the weekend and share anything i do. I don't expect any of these changes would be able to go back to this repository.

At minimum i could contribute a vulkan backend. (https://github.com/TeamForbiddenLLC/warfork-qfusion/blob/master/source/ref_nri/ri_prelude.h) not sure what things are applicable to this project but warfork is based on quake so there is a lot of fixed function logic. I basically back all the opengl state to some backing then hash that to work out if I've seen a pipeline. I found it impractical to hand author a bunch of vulkan pipelines.

I don't really want to promise anything until i have a look at the code more.

[colincornaby]

I basically back all the opengl state to some backing then hash that to work out if I've seen a pipeline.

This is how the Metal pipeline works.
https://github.com/H-uru/Plasma/blob/master/Sources/Plasma/FeatureLib/pfMetalPipeline/plMetalPipelineState.h
https://github.com/H-uru/Plasma/blob/master/Sources/Plasma/FeatureLib/pfMetalPipeline/plMetalPipelineState.cpp

It would be nice to have this generalized so the same hash logic could be used in multiple engines.

I'm thinking of writing some tooling to unpack the compiled assets and rework the maps but might breaks backwards compatibility.

Also keep in mind there is still new content being authored by the community. But ti would be an interesting experiment.

[pollend]

here is the vulkan backend/linux #1901