Shaders

Shaders are regular HLSL 2021 files that live alongside C++ source code in the solution.

They are compiled to platform bytecode by EsotericaReflector.exe (invoked via the Esoterica.Scripts.Reflect project or RunReflection.bat) — DXIL for D3D12, SPIR-V for Vulkan, and so on.

The engine creates root signatures and pipelines from the bytecode at initialization.

Your First Shader

We start with the simplest possible surface shader — Downsample.esf — to show the complete example before diving into each piece.

#include "Base/Render/RHI.esh"
#include "Engine/Render/Shaders/CommonSamplers.esh"
#include "Engine/Render/Shaders/FullscreenTriangle.esh"

#ifndef __cplusplus
ESF_RESOURCE_TABLE struct DownsampleResourceTable
{
    Texture2D                   m_inputTexture;
    SamplerState                m_inputSampler;
};
#endif

#include "Engine/_Module/_AutoGenerated/ShaderReflection/Downsample.esh"

#ifndef __cplusplus

ESF_PARAMETER string ShaderType = "Surface";

#define DownsampleRootSignature    EE_BEGIN_ROOT_SIGNATURE \
                                   DownsampleResourceTableRootConstant( "b0", "space=0" )

EE_DECLARE_ROOT_CONSTANTS( DownsampleResourceTableData, b0, space0 );

[RootSignature( DownsampleRootSignature )]
float4 PS_main( FullscreenTriangleVertexOutput input ) : SV_Target
{
    DownsampleResourceTable resourceTable = CreateDownsampleResourceTable( RootConstants );

    return resourceTable.m_inputTexture.SampleLevel( resourceTable.m_inputSampler, input.m_uv, 0 );
}

#endif

Every .esf follows this structure:

Includes — RHI.esh (shared CPU/GPU types), helper headers, and the auto-generated reflection header.
Resource table — declared with ESF_RESOURCE_TABLE, defines the inputs the shader consumes. Wrapped in #ifndef __cplusplus because it's HLSL-only.
ESF_PARAMETER — the only mandatory one is ShaderType. Tells the reflector which pipeline to build.
Root signature — wires the resource table's packed data into a root constant slot.
Entry point — PS_main here; MaterialShaderMain, VS_main, or CS_main for other shader types. Calls Create<Name>ResourceTable once at the top to decode the packed data.

This is the pattern for every shader in the engine. The rest of this document explains each piece in detail.

Shader Compilation — How and When

Shader compilation is separate from C++ compilation. After changing a .esf or.esh:

First-time or after structural changes — run RunReflection.bat (or build Esoterica.Scripts.Reflect in Visual Studio). This generates reflection headers and compiles bytecode.
Iterating while the engine is running — run CompileShaders.bat. This recompiles changed shaders and hot-reloads them into the live engine without restarting.

When EsotericaReflector.exe runs, it performs these steps for each .esf (as part of its broader C++ reflection pass):

Parse — scans the file for ESF_PARAMETER, ESF_RESOURCE_TABLE, and ESF_CUSTOM_PARAMETERS declarations.
Generate reflection headers — writes a .esh file under Code/Engine/_Module/_AutoGenerated/ShaderReflection/<Name>.esh containing the packed data struct, Create<Name>ResourceTable, and Load<Name>ResourceTable functions.
Compile HLSL — invokes the DirectX Shader Compiler (DXC) to produce platform bytecode — DXIL for D3D12, SPIR-V for Vulkan — for each entry point and permutation.
Store bytecode — writes the compiled bytecode blobs to Build/x64_*/.
Export metadata — writes Build/ShaderStageMetadata.xml containing raw DXC compilation commands for each shader and permutation. Used by the shader stats toolchain (see Profiling VGPR Pressure ).

The generated headers are checked into source control so that C++ code can reference them without running the reflector. Engine shader files live under Code/Engine/Render/Shaders/; game shader files live under Code/Game/Render/Shaders/.

Shader compiler errors appear in the output of EsotericaReflector.exe. When run via the Esoterica.Scripts.Reflect project, errors appear in the Visual Studio output window. When run via RunReflection.bat, errors appear in the console window.

Workflow for Adding a Shader

Create the .esf file under Code/Engine/Render/Shaders/ or Code/Game/Render/Shaders/.
Add it to the project via the Visual Studio UI (right-click the project, Add → Existing Item). If Visual Studio incorrectly tries to compile it as HLSL, edit the .vcxproj to change the item to <None Include="YourShader.esf" />.
Set ShaderType via ESF_PARAMETER — this is the only mandatory parameter.
Write your entry point — MaterialShaderMain for materials, VS_main / PS_main for surfaces, CS_main for compute.
Run RunReflection.bat to compile the shader and generate reflection headers.
Look up the shader at runtime with RenderSystem::FindMaterialShader, FindSurfaceShader, or FindComputeShader by StringID.

For subsequent shader edits while the engine is running, use CompileShaders.bat to recompile and hot-reload without restarting.

Shader File Types

We use two file extensions for shader code:

Extension	Purpose
`.esf`	Esoterica Shader File — the entry point. Compiled to platform bytecode (DXIL, SPIR-V, etc.).
`.esh`	Esoterica Shader Header — shared between C++ and HLSL via `#ifdef __cplusplus`.

How .esh Files Work

We include .esh files from both HLSL and C++. The CPU and GPU share the same type definitions, resource table layouts, and root constant structs. The #ifdef __cplusplus guard splits the two paths:

HLSL side (#ifndef __cplusplus): macros like HLSL_ALIGN_STRUCT expand to nothing. HLSL_PASS_BY_REFERENCE(X) becomes inout X (HLSL's pass-by-reference). EE_BEGIN_SHADER_TYPES_NAMESPACE / EE_END_SHADER_TYPES_NAMESPACE expand to nothing. Structs and functions are compiled as HLSL.
C++ side (#ifdef __cplusplus): HLSL_ALIGN_STRUCT becomes alignas(8). HLSL_PASS_BY_REFERENCE(X) becomes X&. The namespace macros expand to namespace EE::Render::ShaderTypes { }. The C++ compiler sees standard C++20.

This dual compilation ensures that struct layouts, enum values, and constants are identical on both the CPU and GPU — eliminating an entire class of layout-mismatch bugs without paying the runtime cost of parameter lookup by name.

// Inside a custom .esh file — shared between C++ and shader code
struct HLSL_ALIGN_STRUCT MyCustomData
{
    float4  m_color;
    float   m_roughness;
    float   m_metallic;
    uint    m_flags;
};

HLSL_ALIGN_STRUCT forces 8-byte alignment on C++ structs to match HLSL defaults. Misaligned C++ structs will produce corrupted data on the GPU with no compiler warning — always use HLSL_ALIGN_STRUCT on shared structs.

Resource Tables

Resource tables declare shader inputs — they can contain both bindless resource handles and regular parameters like floats or ints. The autogenerated code translates resource handles to HLSL descriptors and packs everything into a single root constant struct. See Bindless Resource System for how handles work.

A resource table is declared with ESF_RESOURCE_TABLE in the .esf file:

#ifndef __cplusplus
ESF_RESOURCE_TABLE struct PostProcessResourceTable
{
    Texture2D   m_depthTexture;
    Texture2D   m_HDRTexture;
    Texture3D   m_tonemapLUT;
    float       m_exposure;
    int         m_toneMapMode;
    float4      m_tintColor;
};
#endif

The autogenerated code handles all conversion between the CPU and GPU representations. Resource fields are resolved from bindless handles; regular parameters are decoded from their packed form. The shader receives the struct exactly as declared — no handles, no bit packing.

The shader compiler rearranges all members and packs them into uint slots internally, this avoids layout-mismatch bugs between CPU and GPU by letting the compiler own the layout.

The cost is a small decode step when Create unpacks the data. Declare fields in any order; the compiler handles alignment.

At build time the shader compiler parses the struct and writes a corresponding .esh file under Code/Engine/_Module/_AutoGenerated/ShaderReflection/<Name>.esh.

The generated file contains three things:

A packed data struct (PostProcessResourceTableData) — the autogenerated C++ counterpart. Resource fields get setters that accept RHI handles; regular parameters get setters that encode the value. You populate this on the CPU and pass it as root constants.
A Create<Name>ResourceTable function — decodes the packed data and resolves all handles, returning a fully usable HLSL resource table with raw types. Call this once at the top of your shader.
A Load<Name>ResourceTable function — loads the packed data from a ByteAddressBuffer at a given offset, then calls Create. Used by material shaders where parameter instances live in the shader data buffer.

HLSL Side

The .esf includes the auto-generated header and wires the data struct into the root signature. At the start of the shader we create the resource table:

#include "Engine/_Module/_AutoGenerated/ShaderReflection/PostProcess.esh"

#define PostProcessRootSignature    EE_BEGIN_ROOT_SIGNATURE \
                                    PostProcessResourceTableRootConstant( "b0", "space=0" )

EE_DECLARE_ROOT_CONSTANTS( PostProcessResourceTableData, b0, space0 );

[RootSignature( PostProcessRootSignature )]
float4 PS_main( FullscreenTriangleVertexOutput input ) : SV_Target
{
    PostProcessResourceTable resourceTable = CreatePostProcessResourceTable( RootConstants );

    float3 hdrColor = resourceTable.m_HDRTexture.SampleLevel( pointClampSampler, input.m_uv, 0 ).rgb;
    // ...
}

PostProcessResourceTableRootConstant (auto-generated) declares the root constant slot. EE_DECLARE_ROOT_CONSTANTS exposes it as RootConstants.

CreatePostProcessResourceTable resolves every handle in one call — the shader never touches descriptor handles directly.

C++ Side

On the CPU we populate the packed data struct with setters and pass it as root constants at draw time:

ShaderTypes::PostProcessResourceTableData rootConstants = {};
rootConstants.SetTonemapLUT( pTonemapLUT );
rootConstants.SetHDRTexture( resourceStates, RHI::PipelineStage::PixelShader, hdrTexture );
rootConstants.SetDepthTexture( resourceStates, RHI::PipelineStage::PixelShader, depthTexture );
rootConstants.SetExposure( 1.0f );
rootConstants.SetToneMapMode( 0 );
rootConstants.SetTintColor( float4( 1.0f, 1.0f, 1.0f, 1.0f ) );

resourceStates.FlushBarriers( pCommandBuffer );

RHI::CmdSetPipeline( pCommandBuffer, m_pPipeline );
RHI::CmdSetRootConstants( pCommandBuffer, 0, &rootConstants, sizeof( rootConstants ) );
RHI::CmdDraw( pCommandBuffer, 3, 0 );

Each Set<Field> for resources has two overloads: one without state tracking, and an optional one that tracks resource state. When using the tracked overload, set all resources then call FlushBarriers once before the draw call.

Non-resource fields use a single setter that encodes the value.

Root Signatures

Every shader declares its own root signature with two macros:

EE_BEGIN_ROOT_SIGNATURE starts the root signature definition. Each line names a root parameter — typically the auto-generated RootConstant macro that matches the resource table data struct.
EE_DECLARE_ROOT_CONSTANTS binds the packed data struct to a register and exposes it as RootConstants in the shader body. There are matching macros for every root parameter type: EE_DECLARE_ROOT_CBV, EE_DECLARE_ROOT_SRV, and EE_DECLARE_ROOT_UAV.

The resource table auto-generates only the root constant macro (PostProcessResourceTableRootConstant). You still write the root signature manually because a shader often needs additional root parameters beyond the resource table.

Material shaders, for example, bind both the resource table's packed data (via EE_DECLARE_ROOT_CONSTANTS) and engine global parameters (via EE_DECLARE_ROOT_CBV for GlobalParameters). This explicit wiring keeps the root signature layout visible in the shader source rather than hidden behind code generation.

In practice, resource tables have replaced direct SRV/UAV root parameters — you will mostly use EE_DECLARE_ROOT_CONSTANTS (for resource table packed data) and EE_DECLARE_ROOT_CBV (for engine global parameters like GlobalParameters).

Shader Types

We have three shader types, declared via ShaderType: Material, Surface, and Compute.

All pipelines are created during engine initialization. Creating pipelines at runtime won't pass code review — this is the leading cause of stuttering.

Material Shaders

A material shader is part of the GPU-driven Material Rendering Pipeline.

It carries the material concept end-to-end: a material is a named instance of a shader's parameter struct — a handle into the global shader data buffer.

There is no intermediate material graph and no shader-permutation generator sitting between the artist and the GPU. The custom parameter struct you write in HLSL is what the material editor exposes in the UI.

When an artist changes a texture or color, the edit writes directly to the corresponding shader parameter in a global GPU buffer — there is no intermediate layer.

This is a deliberate design trade-off. Material graphs generate large numbers of shader permutations, which are fundamentally incompatible with the GPU-driven pipeline. While node-based workflows can feel accessible to artists, the cost in pipeline count, shader compilation time, and runtime stutter scales poorly with content.

Esoterica instead provides a small number of hand-written material shaders (on the order of 100–200), each with artist-editable parameters exposed through the material editor.

For feature variation, we use dynamic branching on uniform flags when VGPR pressure permits, or create a separate shader and material when it doesn't — see Shader Writing Guidelines .

The material shader combines a mesh shader and a pixel shader into a single file that participates in the instance culling, cluster culling, bucket resolve, and indirect draw pipeline.

The entry point is MaterialShaderMain:

#include "Engine/Render/Shaders/Renderer/DefaultMeshShader.esh"
#include "Engine/Render/Shaders/Renderer/MaterialShaderPBR.esh"

ESF_PARAMETER string ShaderType = "Material";

ESF_CUSTOM_PARAMETERS struct CustomParameters : ShaderParameters
{
    SamplerState    m_textureSamplerLinear  ESF_DEFAULT_VALUE( "Linear" );
    Texture2D       m_albedoTexture         ESF_DEFAULT_VALUE( "data://Render/Textures/DefaultColor.texture" );
};

#include "Engine/_Module/_AutoGenerated/ShaderReflection/DefaultPBR.esh"

MaterialShaderOutput MaterialShaderMain( in MaterialShaderInput materialInput )
{
    ByteAddressBuffer    shaderDataBuffer = GetBuffer( RootCBV.m_shaderDataBuffer );

    CustomParameters     customParameters = LoadCustomParameters( shaderDataBuffer, materialInput.m_primitive.GetShaderParametersOffsetIn32ByteBlocks() );
    MaterialShaderOutput materialOutput   = MaterialShaderOutput::New( materialInput );

    float4 albedoSample = customParameters.m_albedoTexture.Sample( customParameters.m_textureSamplerLinear, materialInput.m_vertex.m_uv.xy );
    materialOutput.m_albedo  = albedoSample.rgb;
    materialOutput.m_opacity = albedoSample.a * customParameters.m_opacity;

    return materialOutput;
}

Materials and the Global Parameter Buffer

An engine material — the asset an artist creates and edits in the material editor — is a named, UI-editable instance of a custom shader parameter struct.

The editor uses reflection to expose the struct fields. Changing a texture or color writes directly to the corresponding shader parameter.

All material instances — from every material in the scene — live in a single global shader data buffer. LoadCustomParameters looks up the correct block by handle.

A shader can access any material's parameters as long as it knows the handle. This enables material blending: include a handle to another material in the custom parameter struct, load both parameter sets and blend the results.

The shader must know the struct layout of both materials — either both use the same shader, or the blending shader includes the other's struct definition.

Every material instance is a self-contained blob of packed parameter data in the global buffer. There is no parent-child inheritance chain — the ESF_CUSTOM_PARAMETERS struct inherits from ShaderParameters to establish the shader relationship, and overriding a field means setting it to a different value in the editor or from code.

On the CPU side, materials managed through the editor use the same mechanism as code-driven parameter instances (MaterialShaderParametersInstance). The underlying representation is identical — a packed data struct in a global buffer, resolved by handle.

We plan to add more material features to the stock shaders — material blending, UV transforms, and a flexible forward decal system.

Shader Permutations

Material shaders support a fixed set of runtime pixel shader permutations. The mesh shader compiles once; only the pixel shader varies.

Permutation selection happens entirely on the GPU with no CPU involvement:

The GPU inspects runtime shader parameters and writes draw commands into different indirect draw buffers — one per permutation. There is no CPU readback.
Per-cluster effects are the primary use case. For example, translucent cutouts around the player: only clusters in the affected area use the alpha test or alpha blend pipelines, while the rest of the scene renders with the cheaper opaque pipeline.
LOD dither cross-fading also uses permutation selection but works at the instance level rather than per-cluster.
The cost of these effects is temporary and area-limited; only select clusters pay it.

The permutation set is fixed. The base permutations are hardcoded in the engine and adding new ones is unlikely to pass code review. This keeps the total pipeline count low.

Prefer dynamic branching on uniform flags for feature variation (unless VGPR increase is prohibitive).

When a feature would explode VGPR, create a new shader and material. See Shader Writing Guidelines for the trade-off analysis.

The source detects which permutation is active via preprocessor macros:

Permutation	Macro	Notes
Default	(none)	Opaque shader, writes color, no `clip()`
Depth only	`PERMUTATION_DEPTH_ONLY`	Only when `AllowBufferWritesInPixelShader=true`, otherwise null PS
Alpha test	`PERMUTATION_ALPHA_TEST`	Always hide `clip()` under this macro — important performance optimization

The renderer creates a set of pipelines from these permutations:

Pipeline	Bytecode combination	Notes
Opaque	MS + PS[Default]	Default opaque rendering
Depth only	MS + null PS	Depth prepass. Null PS unless `AllowBufferWritesInPixelShader` is set.
Depth only alpha test	MS + PS[DepthOnly+AlphaTest]	Depth pass with texture-sampled alpha testing
Alpha blend	MS + PS[Default]	Alpha blended objects, uses default pixel shader, blend state set in pipeline

AllowBufferWritesInPixelShader is not yet implemented.

Material shader pipelines are renderer-specific. The world renderer creates them, not the global renderer system, because pipeline parameters depend on what a particular world renderer is doing.

Custom Shader Parameters

ShaderParameters is the base struct that every material shader carries.

It encodes common material metadata — shader ID, double-sided flags, and other properties needed by system shaders.

Because ESF_CUSTOM_PARAMETERS requires that custom structs inherit from ShaderParameters, any shader that loads a material's parameter block (culling, bucket resolve, etc.) can read these common fields from any material regardless of its custom struct layout.

Most material shaders use ESF_CUSTOM_PARAMETERS to extend the base struct with artist-editable properties — the fields you declare are what the material editor exposes to artists.

The struct must inherit from ShaderParameters; the shader compiler will error if it doesn't. If you omit custom parameter declaration, the shader will use the base struct implicitly, with no artist-facing controls.

ESF_CUSTOM_PARAMETERS struct CustomParameters : ShaderParameters
{
    SamplerState    m_textureSamplerLinear  ESF_DEFAULT_VALUE( "Linear" );
    Texture2D       m_albedoTexture         ESF_DEFAULT_VALUE( "data://Render/Textures/DefaultColor.texture" );
};

A custom parameter struct is an extension of a resource table: everything from the Resource Tables section applies.

Memory is allocated in 32-byte increments, and all instances, from every material, live in one global shader data buffer.

Annotations control how each field appears in the material editor:

Annotation	Purpose
`ESF_DEFAULT_VALUE( X )`	Sets the default value shown in the editor. For textures this is a resource path.
`ESF_MIN_VALUE( X )`	Lower bound for numeric sliders in the editor UI.
`ESF_MAX_VALUE( X )`	Upper bound for numeric sliders in the editor UI.

These macros are no-ops at the HLSL level — the shader compiler reads them during reflection and the material editor uses them to provide safe defaults and clamped ranges to the artist.

On the CPU side we use MaterialShaderParametersInstance, which holds a handle to a struct instance in the global shader data buffer and a pointer to the shader reflection for field lookup by string ID:

class DissolveEffect
{
public:
    void Initialize( RenderSystem* pRenderSystem, Material* pMaterial )
    {
        m_parameters     = pRenderSystem->CreateShaderParameters( pMaterial->GetShaderIndex() );
        m_dissolveTex    = m_parameters.FindParameter( "m_dissolveTexture" );
        m_dissolveAmount = m_parameters.FindParameter( "m_dissolveAmount" );
    }

    void Shutdown( RenderSystem* pRenderSystem )
    {
        pRenderSystem->DeleteShaderParameters( eastl::move( m_parameters ) );
    }

    void Update( RenderSystem* pRenderSystem, float deltaTime )
    {
        m_currentDissolve += deltaTime * 0.5f;

        m_parameters.SetTexture( m_dissolveTex, RHI::GetTexture2DHandle( m_pDissolveTexture, RHI::DescriptorTypeFlags::Texture, 0 ) );
        m_parameters.SetScalar( m_dissolveAmount, m_currentDissolve );

        pRenderSystem->QueueShaderParametersUpdate( m_parameters );
    }

private:
    MaterialShaderParametersInstance    m_parameters;
    MaterialShaderParameterHandle       m_dissolveTex;
    MaterialShaderParameterHandle       m_dissolveAmount;

    RHI::Texture*                       m_pDissolveTexture = nullptr;
    float                               m_currentDissolve  = 0.0f;
};

Parameter lookup by name is relatively expensive. Resolve all handles once at initialization and reuse them for subsequent updates.

Parameter memory is persistent on the GPU — you don't have to update it every frame if it's not needed.

Surface Shaders

A surface shader combines a vertex or mesh shader with a pixel shader.

Most surface shaders use FullscreenTriangle.esh as the vertex shader to draw a single triangle covering the viewport, then run the pixel shader for image filtering or post-processing — but a custom vertex or mesh shader works just as well.

You create surface shader pipelines manually — the parameters depend on what the render pass is doing.

ESF_PARAMETER string ShaderType = "Surface";

float4 VS_main( uint vertexID : SV_VertexID ) : SV_Position
{
    // Vertex shader code, or just #include "FullscreenTriangle.esh"
}

[RootSignature( DownsampleRootSignature )]
float4 PS_main( FullscreenTriangleVertexOutput input ) : SV_Target
{
    DownsampleResourceTable resourceTable = CreateDownsampleResourceTable( RootConstants );
    return resourceTable.m_inputTexture.SampleLevel( resourceTable.m_inputSampler, input.m_uv, 0 );
}

Compute Shaders

Compute shaders need no special treatment. Declare them with ShaderType = "Compute"and the renderer system creates their pipelines immediately at initialization.

ESF_PARAMETER string ShaderType = "Compute";

ESF_RESOURCE_TABLE struct MyComputeResourceTable
{
    RWTexture2D<float4> m_outputTexture;
    Texture2D<float>    m_inputTexture;
    uint                m_dispatchWidth;
};

#include "Engine/_Module/_AutoGenerated/ShaderReflection/MyCompute.esh"

#define MyComputeRootSignature  EE_BEGIN_ROOT_SIGNATURE \
                                MyComputeResourceTableRootConstant( "b0", "space=0" )

EE_DECLARE_ROOT_CONSTANTS( MyComputeResourceTableData, b0, space0 );

[RootSignature( MyComputeRootSignature )]
[numthreads( 64, 1, 1 )]
void CS_main( uint3 dispatchThreadID : SV_DispatchThreadID )
{
    MyComputeResourceTable resourceTable = CreateMyComputeResourceTable( RootConstants );

    if ( dispatchThreadID.x >= resourceTable.m_dispatchWidth ) { return; }

    float value = resourceTable.m_inputTexture[dispatchThreadID.xy];
    resourceTable.m_outputTexture[dispatchThreadID.xy] = float4( value, value, value, 1.0 );
}

Debug Draw

Every shader has access to a global debug draw system for visualizing shader internals in real time. The system is built on AppendBuffer .

Warning — pixel shader debug draw has significant performance cost.

Requires setting AllowBufferWritesInPixelShader to true, which silently disables HiZ.
Append buffer memory is unbounded per invocation. Pixel shader invocations count can easily be in the millions — an unguarded append can exhaust GPU memory.
Turn the flag off when done. Do not ship with it enabled.

Setup

The standard rendering pipeline passes a ShaderDebugDrawBuffer handle via the RootCBV global parameters (RootCBV.m_shaderDebugDrawBuffer).

For custom render passes, get the buffer handle from the render viewport:

#if EE_DEVELOPMENT_TOOLS
RHI::BufferHandle debugBufferHandle = RHI::GetBufferHandle( pRenderViewport->m_shaderDebugDrawBuffers[frameIndex], RHI::DescriptorTypeFlags::Buffer );
#endif

Pass this handle to your shader via root constants or a root CBV.

Initialize at the top of the shader, then call any drawing function:

#include "Engine/Render/Shaders/DebugDraw.esh"

ESF_PARAMETER string ShaderType = "Compute";

EE_DECLARE_ROOT_CONSTANTS( DebugVisualizeRootConstants, b0, space0 );

[RootSignature( DebugVisualizeRootSignature )]
[numthreads( 64, 1, 1 )]
void CS_main( uint3 dispatchThreadID : SV_DispatchThreadID )
{
    ShaderDebugDraw debugDraw = InitializeShaderDebugDraw( RootConstants.m_shaderDebugDrawBuffer );

    // All threads can write — be mindful of memory usage
    if ( dispatchThreadID.x == 0 )
    {
        // Opaque line with interpolated colors
        debugDraw.DrawLine( startPos, endPos, startColor, endColor );

        // Transparent box
        debugDraw.DrawBox( transform, halfSize, float4( color.rgb, 0.5 ), thickness );

        // Always-visible point (depth testing disabled)
        debugDraw.DrawPoint( position, color, false /* depthTest */ );
    }
}

For material shaders, debug data is initialized automatically and available in the material shader input struct.

Conventions

Shader debug draw follows the same conventions as engine-side debug draw but with shader-appropriate defaults:

All positions are in world space.
Positions accept float3 or float4. With float4, the W component is thickness (default 1.0).
Colors accept float3 or float4. With float4, the W component is transparency (default 1.0).
Lines interpolate color between start and end.
The last parameter on each draw call controls depth testing against the main camera depth buffer.

As a side effect of the CPU readback, draws can occasionally flicker for one frame while the buffer resizes.

Shader Writing Guidelines

With the mechanics of shader authoring covered, here are the constraints imposed by the GPU-driven pipeline.

Keep the number of unique shaders low

The pipeline scales poorly with shader count.

Do more work in one complex shader instead of creating many simple ones — the GPU prefers it this way too. The system works well with 100–200 unique shaders but falls apart with 100k or more.

Use dynamic branching, but profile VGPR pressure

Toggle features with dynamic branches on uniform flags instead of creating separate shaders and materials — when the combined register footprint is acceptable.

If a feature would significantly reduce occupancy, create a separate shader and material instead.

The engine's 100–200 shader target gives you headroom; PSO stutter from large permutation spaces is a significantly harder problem to solve. See Profiling VGPR Pressure for more details.

Merge post-processing passes.

Combine post-processing steps into a single shader where practical.

Profiling VGPR Pressure

We have a toolchain for profiling shader register usage across GPU architectures:

EsotericaReflector.exe outputs Build/ShaderStageMetadata.xml — raw DXC compilation commands for each shader entry point and permutation.
Code/Scripts/ShaderStats.py consumes that XML and invokes RGA (Radeon GPU Analyzer) to compile and analyze all shaders for different AMD GPU architectures. Requires the RGA_PATH environment variable pointing to rga.exe — RGA is not bundled with the engine.
ShaderStats is the auto-generated output — a table of VGPR, SGPR, occupancy, LDS, and scratch memory for every shader, broken down by GPU architecture and permutation.

Run ShaderStats.py after significant shader changes to check register pressure. When a shader pushes into high VGPR allocation, that's the signal to either simplify the shader or split into a separate shader and material.

RGA gives real hardware-level numbers rather than DXC estimates, and checking the output into the repo makes register pressure visible in code review.

Common Patterns

Passing Per-Frame Data to a Compute Shader

// CPU side — populate the packed data struct each frame
ShaderTypes::MyComputeResourceTableData rootConstants = {};
rootConstants.SetInputTexture( resourceStates, RHI::PipelineStage::ComputeShader, inputState );
rootConstants.SetOutputTexture( resourceStates, RHI::PipelineStage::ComputeShader, outputState );
rootConstants.SetDeltaTime( deltaTime );
resourceStates.FlushBarriers( pCommandBuffer );

RHI::CmdSetPipeline( pCommandBuffer, m_pPipeline );
RHI::CmdSetRootConstants( pCommandBuffer, 0, &rootConstants, sizeof( rootConstants ) );
RHI::CmdDispatch( pCommandBuffer, dispatchX, dispatchY, 1 );

Using a Shared Helper in Multiple Shaders

Create a .esh file with shared functions, structs, or constants:

// MySharedHelpers.esh — included by both HLSL and C++
EE_BEGIN_SHADER_TYPES_NAMESPACE

struct HLSL_ALIGN_STRUCT MySharedData
{
    float4x4 m_viewProjection;
    float3   m_cameraPosition;
    float    m_padding; // explicit, to avoid HLSL surprises
};

EE_END_SHADER_TYPES_NAMESPACE

Include it in your .esf like any other header:

#include "Engine/Render/Shaders/MySharedHelpers.esh"

Accessing a Material Parameter from Another Shader

A material shader can access parameters from another material by storing a parameter-block offset in its custom parameter struct. The shader loads both parameter sets from the global shader data buffer and blends the results. Both materials must use the same shader, or the blending shader must include the other material's struct definition.

// The primary material's parameter struct — note the m_overlayOffsetIn32ByteBlocks field
ESF_CUSTOM_PARAMETERS struct BlendedParameters : ShaderParameters
{
    Texture2D   m_baseAlbedo        ESF_DEFAULT_VALUE( "data://Render/Textures/DefaultColor.texture" );
    Texture2D   m_overlayAlbedo     ESF_DEFAULT_VALUE( "data://Render/Textures/DefaultColor.texture" );
    uint        m_overlayOffsetIn32ByteBlocks     ESF_DEFAULT_VALUE( "~0" );
    float       m_blendFactor       ESF_DEFAULT_VALUE( "0.5" ) ESF_MIN_VALUE( "0.0" ) ESF_MAX_VALUE( "1.0" );
};

// Inside MaterialShaderMain — load both parameter sets and blend
MaterialShaderOutput MaterialShaderMain( in MaterialShaderInput materialInput )
{
    ByteAddressBuffer    shaderDataBuffer   = GetBuffer( RootCBV.m_shaderDataBuffer );

    // Load the primary material's parameters
    BlendedParameters    primaryParams      = LoadCustomParameters( shaderDataBuffer, materialInput.m_primitive.GetShaderParametersOffsetIn32ByteBlocks() );

    MaterialShaderOutput output             = MaterialShaderOutput::New( materialInput );

    float4 baseSample = primaryParams.m_baseAlbedo.Sample( GetLinearClampSampler(), materialInput.m_vertex.m_uv.xy );
    output.m_albedo  = baseSample.rgb;
    output.m_opacity = baseSample.a * primaryParams.m_opacity;

    // Load a second material's parameters if an overlay is set
    if ( primaryParams.m_overlayOffsetIn32ByteBlocks != ~0U )
    {
        BlendedParameters overlayParams = LoadCustomParameters( shaderDataBuffer, primaryParams.m_overlayOffsetIn32ByteBlocks );

        float4 overlaySample = overlayParams.m_overlayAlbedo.Sample( GetLinearClampSampler(), materialInput.m_vertex.m_uv.xy );
        output.m_albedo  = lerp( output.m_albedo, overlaySample.rgb, primaryParams.m_blendFactor );
        output.m_opacity = max( output.m_opacity, overlaySample.a * overlayParams.m_opacity );
    }

    return output;
}

The offset is in 32-byte blocks. ~0U is the invalid/unset sentinel (same convention as RHI handles). Editor support for setting the offset is work in progress.

Compared to Other Engines

Esoterica's shader system makes a deliberate trade-off: it sacrifices some artist-facing abstractions (material graphs, visual editors) for GPU-driven predictability and less performance surprises.

If you're coming from Unreal, Unity, or Godot, the shift is from a high-level shader workflow to hand-written HLSL with explicit resource binding.

Concept	Unreal	Unity	Godot	Esoterica
Shader language	Material graph editor	ShaderLab + HLSL / `.shader`	Godot Shading Language (GLSL-like)	HLSL 2021
Shader ↔ material	Material + Material Instance	Shader + Material	Shader + ShaderMaterial	Material = handle to shader parameter instance
Resource binding	Param nodes in material graph	`Properties` block	`uniform` keyword	Resource tables
Variants / permutations	Static Switch Param, dynamic branching	`#pragma multi_compile` / `shader_feature`	`render_mode` annotations	Fixed set: opaque, alpha test, depth, alpha blend
Shader compilation	Automatic during cook	Automatic on save	Automatic on save	`RunReflection.bat` (codegen) + `CompileShaders.bat` (live reload)
CPU/GPU struct sharing	Manual C++ ↔ HLSL sync	Manual C# ↔ HLSL sync	Manual GDScript ↔ shader sync	Dual-compilation `.esh` — mismatches are compile errors
Debug tooling	Unreal Insights	Frame Debugger, Profiler	Built-in Profiler	Shader debug draw

What's better: Compile-time CPU/GPU layout verification (.esh), shader debug draw without captures, cross-material parameter blending by handle, and zero hidden shader variants — PSO count is known at authoring time.

What you lose: No shader/material separation (new behavior = new .esf + new material), and a manual two-step compilation workflow we plan to streamline.

See Material Rendering Pipeline for the GPU-driven pipeline that material shaders plug into, and RHI for the bindless resource system underneath.

Coding guidelines

Rendering

Shaders

Shaders

Your First Shader

Shader Compilation — How and When

Workflow for Adding a Shader

Shader File Types

How .esh Files Work

Resource Tables

HLSL Side

C++ Side

Root Signatures

Shader Types

Material Shaders

Materials and the Global Parameter Buffer

Shader Permutations

Custom Shader Parameters

Surface Shaders

Compute Shaders

Debug Draw

Setup

Conventions

Shader Writing Guidelines

Keep the number of unique shaders low

Use dynamic branching, but profile VGPR pressure

Merge post-processing passes.

Profiling VGPR Pressure

Common Patterns

Passing Per-Frame Data to a Compute Shader

Using a Shared Helper in Multiple Shaders

Accessing a Material Parameter from Another Shader

Compared to Other Engines

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally