Skip to content

Shaders in Axmol3

Jump to bottom
halx99 edited this page Jul 9, 2026 · 1 revision

Shader Migration Guide: GLSL → HLSL 5.1 (WIP)

axmol PR #3233 — Migration of all engine and test shaders from GLSL to HLSL 5.1

Overview

This document describes the complete shader migration from GLSL (OpenGL ES 3.0) to HLSL 5.1 (D3D11/D3D12/Vulkan/GLES) for the axmol engine. The migration covers:

  • 57 builtin engine shaders (axmol/renderer/shaders/)
  • 44 cpp-tests shaders (tests/cpp-tests/Source/shaders/)
  • 2 spine shaders (extensions/spine/shaders/)
  • 19 Live2D shaders (extensions/Live2D/.../shaders/)

Code Structure Changes

New vs Old File Extensions

Old New
.vert _vs.hlsl
.frag _ps.hlsl
.fsh _ps.hlsl
.vsh _vs.hlsl
.vs _vs.hlsl
.fs _ps.hlsl

Stage detection: axslcc now detects shader stage only from file stem suffix (_vs, _ps, _cs). Legacy extension-based detection removed.

AXSLCC.cmake

# Before: supported GLSL + HLSL extensions
axslcc_option(AXSLCC_SOURCE_FILE_EXTENSIONS ".hlsl;.vert;.vsh;.vs;.frag;.fsh;.fs")

# After: only .hlsl
axslcc_option(AXSLCC_SOURCE_FILE_EXTENSIONS ".hlsl")

Key Files Involved

File Role
axmol/renderer/shaders/base.hlsli Common macros, builtin sampler declarations
axmol/cmake/Modules/AXSLCC.cmake Shader build rules
axslcc/src/ Offline shader compiler

Shader Conversion Patterns

Vertex Shader: GLSL → HLSL

// BEFORE (GLSL)
#version 310 es
layout(location = POSITION) in vec3 a_position;
layout(location = TEXCOORD0) in vec2 a_texCoord;
layout(location = TEXCOORD0) out vec2 v_texCoord;
layout(std140, set = 0, binding = 0) uniform vs_ub {
    mat4 u_MVPMatrix;
};

void main() {
    v_texCoord = a_texCoord;
    gl_Position = u_MVPMatrix * vec4(a_position, 1.0);
}

// AFTER (HLSL)
#include "base.hlsli"

struct VS_IN {
    float3 a_position : POSITION;
    float2 a_texCoord : TEXCOORD0;
};

struct VS_OUT {
    float2 v_texCoord : TEXCOORD0;
    float4 position : SV_Position;
};

cbuffer vs_ub : register(b0, space0) {
    float4x4 u_MVPMatrix;
};

VS_OUT main(VS_IN input) {
    VS_OUT output;
    output.position = mul(u_MVPMatrix, float4(input.a_position, 1.0));
    output.v_texCoord = input.a_texCoord;
    return output;
}

Fragment Shader: GLSL → HLSL

// BEFORE (GLSL)
#version 310 es
precision highp float;
layout(location = TEXCOORD0) in vec2 v_texCoord;
layout(set = 1, binding = 0) uniform sampler2D u_tex0;
layout(std140, set = 0, binding = 1) uniform fs_ub {
    vec4 u_color;
};
layout(location = SV_Target0) out vec4 FragColor;

void main() {
    FragColor = texture(u_tex0, v_texCoord) * u_color;
}

// AFTER (HLSL)
#include "base.hlsli"

struct PS_IN {
    float2 v_texCoord : TEXCOORD0;
};

cbuffer fs_ub : register(b1, space0) {
    float4 u_color;
};

Texture2D u_tex0 : register(t0, space1);

float4 main(PS_IN input) : SV_Target0 {
    return u_tex0.Sample(LinearClamp, input.v_texCoord) * u_color;
}

Shadertoy / Fullscreen PS (no varying inputs)

#include "base.hlsli"

struct PS_IN {
    float4 gl_FragCoord : SV_Position;
};

cbuffer fs_ub : register(b1, space0) {
    float2 center;
    float2 resolution;
    float4 u_Time;
};

float4 main(PS_IN input) : SV_Target0 {
    float2 fragCoord = input.gl_FragCoord.xy;
    // ... shadertoy body ...
}

Type Translation Table

GLSL HLSL
vec2 float2
vec3 float3
vec4 float4
mat4 float4x4
mat3 float3x3
mix(a,b,t) lerp(a,b,t)
fract(x) frac(x)
mod(x,y) fmod(x,y)
texture(sampler, uv) sampler.Sample(LinearClamp, uv)
texture(samplerCube, dir) samplerCube.Sample(LinearClamp, dir)
gl_Position output.position (SV_Position)
gl_FragCoord.xy input.gl_FragCoord.xy (SV_Position)
FragColor = val; return val;
layout(location=SV_Target0) out vec4 FragColor; float4 main(...) : SV_Target0

Semantic Mapping

GLSL Location HLSL Semantic
POSITION (0) POSITION
TEXCOORD0 TEXCOORD0
TEXCOORD1 TEXCOORD1
NORMAL NORMAL
COLOR0 COLOR0
BINORMAL BINORMAL
BLENDINDICES BLENDINDICES
BLENDWEIGHT BLENDWEIGHT

Uniform Buffer Layout

// Vertex shader UBO — set 0 → space 0, binding 0
cbuffer vs_ub : register(b0, space0) { ... }

// Fragment shader UBO — set 0 → space 0, binding 1
cbuffer fs_ub : register(b1, space0) { ... }

Texture and Sampler Bindings

// Texture at t0 — set 1 → space 1
Texture2D u_tex0 : register(t0, space1);
Texture2D u_tex1 : register(t1, space1);
TextureCube u_cubeTex : register(t0, space1);

// Builtin samplers are declared in base.hlsli
// Sampling: use LinearClamp, LinearWrap, PointClamp, etc.
u_tex0.Sample(LinearClamp, uv)

Legacy Cleanup

base.hlsli — Removed Macros

// REMOVED — these were GLES2 legacy
#undef TANGENT
#undef BINORMAL
#undef BLENDINDICES
#undef BLENDWEIGHT
#define TANGENT TEXCOORD6
#define BINORMAL TEXCOORD7
#define BLENDINDICES COLOR1
#define BLENDWEIGHT COLOR2

base.hlsli — Semantic Fix

// BEFORE: SPOTLIGHT overlapped with POINTLIGHT array (TEXCOORD2–5)
#ifndef SPOTLIGHT
#define SPOTLIGHT TEXCOORD4
#endif

// AFTER: moved to TEXCOORD6 (after POINTLIGHT's 4-element range)
#ifndef SPOTLIGHT
#define SPOTLIGHT TEXCOORD6
#endif

Sampler12.h — Deleted

The entire axmol/rhi/d3d12/Sampler12.h file was removed. Its BuiltinSamplers string contained sampler declarations that are now handled by base.hlsli.

colorUtils.hlsli — Matrix Multiply Fix

// BEFORE: DXC rejected implicit float3x3 constructor
return float3x3(colorTransform[0].xyz, colorTransform[1].xyz, colorTransform[2].xyz) * YUV;

// AFTER: explicit matrix construction
float3x3 m = { colorTransform[0].xyz, colorTransform[1].xyz, colorTransform[2].xyz };
return mul(m, YUV);

Tool Changes

axslcc — New Options

Option Description
--dxil Windows only: compile SPIRV-Cross HLSL output to DXIL bytecode via DXC
--dxc-reflect Windows only: use DXC reflection for REFL chunk (stub, validation only)

axslcc — Compilation Pipeline

axslcc performs per-target front-end compilation to enable source-level target macros:

for each target:
  HLSL source + -DAXSLC_TARGET_<LANG>=1
    → glslang → SPIR-V
      → SPIRV-Cross + attribute remap → target output (HLSL/ESSL/GLSL/MSL)
      → SPIR-V raw (for Vulkan, direct copy)
    → (+ --dxil) DXC → DXIL bytecode (for HLSL targets, optional)
    → (+ --reflect) REFL chunk from SPIR-V reflection
  → packed into .sc file (axslcc --sc)

axslcc — Key Fixes

Fix File
Attribute remap for HLSL output cross_compiler.cpp
Varying name normalize for GLES cross_compiler.cpp
HLSL source semantic parse for REFL utils.cpp + reflection.cpp
separate_images for HLSL Texture2D reflection.cpp
input. prefix strip from variable names utils.cpp
SEH crash handler + g_inDxcCompile flag main.cpp + compiler.cpp
/EHa for cross-DLL C++ exception catch CMakeLists.txt

Backend-Specific Notes

D3D11/D3D12

  • Engine RHI receives HLSL source code from SC files
  • D3D12 driver compiles HLSL → DXIL at runtime using DXC
  • Use --dxil option for offline DXIL pre-compilation

GLES (ANGLE / OpenGL ES 3.0)

  • SPIRV-Cross generates ESSL 300 from SPIR-V
  • Varying names normalized to strip input. prefix
  • GLES linker matches varyings by name

Vulkan

  • SPIR-V binary directly embedded in SC file
  • No cross-compilation needed

Metal (MSL)

  • Not tested yet
  • Metal Y-flip handled by AXSLC_TARGET_MSL macro

Common Pitfalls

PS_IN / VS_OUT Member Declaration Order

Symptom: D3D12 CreateGraphicsPipelineState fails:

D3D12 ERROR: ID3D12Device::CreateGraphicsPipelineState: Vertex Shader - Pixel Shader
linkage error: Signatures between stages are incompatible.
Semantic 'TEXCOORD' of the input stage has a hardware register component mask that
is not a subset of the output of the previous stage.
[STATE_CREATION ERROR #662: CREATEGRAPHICSPIPELINESTATE_SHADER_LINKAGE_REGISTERMASK]

Root Cause: The shader compilation pipeline does not preserve HLSL semantic names for varyings through the intermediate SPIR-V representation:

  1. glslang compiles HLSL to SPIR-V, assigning Location decorations to struct members in declaration order (ignoring the semantic name/index)
  2. SPIRV-Cross cross-compiles SPIR-V to HLSL for D3D targets, generating TEXCOORDn semantics from Location values
  3. If VS_OUT and PS_IN declare members in different orders, the cross-compiled semantics will mismatch

Example — incorrect:

// VS_OUT (from positionTextureColor_vs.hlsl):
struct VS_OUT {
    float4 v_color : COLOR0;       // location 0 → TEXCOORD0
    float2 v_texCoord : TEXCOORD0; // location 1 → TEXCOORD1
    float4 position : SV_Position;
};

// PS_IN (custom):
struct PS_IN {
    float2 v_texCoord : TEXCOORD0; // location 0 → TEXCOORD0 (receives v_color!)
    float4 v_color : COLOR0;       // location 1 → TEXCOORD1 (receives v_texCoord!)
};
// → D3D12: PS expects float4 at TEXCOORD1 but VS writes float2 → PSO rejected

Fix: Ensure PS_IN member declaration order matches VS_OUT:

struct PS_IN {
    float4 v_color : COLOR0;       // location 0 → TEXCOORD0 ✓
    float2 v_texCoord : TEXCOORD0; // location 1 → TEXCOORD1 ✓
};

Affected files: 9 custom PS shaders in tests/cpp-tests/Source/shaders/ paired with positionTextureColor_vs had reversed v_texCoord/v_color order:

File Status
example_Blur_ps.hlsl Fixed
example_Outline_ps.hlsl Fixed
example_Noisy_ps.hlsl Fixed
example_GreyScale_ps.hlsl Fixed
example_Sepia_ps.hlsl Fixed
example_EdgeDetection_ps.hlsl Fixed
example_Bloom_ps.hlsl Fixed
example_CelShading_ps.hlsl Fixed
example_Blur_winrt_ps.hlsl Fixed
example_Normal_ps.hlsl Fixed
example_HorizontalColor_ps.hlsl Fixed
Normal_ps.hlsl Already correct

This issue only affects PS shaders paired with a VS from a different file. When the VS and PS are written together (same prefix), the declaration order is naturally consistent.

Y-Flip Handling with AXSLC_TARGET_* Macros

After migration, the AXSLC_TARGET_* macros were removed from shader sources, but they are still needed because the Y-axis origin differs between platforms (GLSL/ESSL: bottom-left; HLSL/SPIRV/MSL: top-left).

axslcc now performs per-target front-end compilation, passing the appropriate AXSLC_TARGET_<LANG>=1 as a preprocessor define via glslang. This allows #include "base.hlsli" to resolve #ifdef conditionals correctly at source level.

base.hlsli provides two macros:

Macro Purpose
AXSLC_UV_TOP Flag: 0 = bottom-left (GLSL/ESSL), 1 = top-left (HLSL/SPIRV/MSL)
AX_Y_UP(v) Convert Y to bottom-left convention: (AXSLC_UV_TOP ? (1.0 - (v).y) : ((v).y))

vr_vs.hlsl uses AX_Y_UP for Y-up texcoord calculation:

output.v_texCoord = float2(input.a_texCoord.x, AX_Y_UP(input.a_texCoord));

11 fullscreen PS shaders use #if AXSLC_UV_TOP for gl_FragCoord Y-flip:

#if AXSLC_UV_TOP
    float2 fragCoord = float2(input.gl_FragCoord.x, u_screenSize.y - input.gl_FragCoord.y);
#else
    float2 fragCoord = input.gl_FragCoord.xy;
#endif
File Flip source
example_Heart_ps.hlsl u_screenSize.y
example_Julia_ps.hlsl u_screenSize.y
example_Flower_ps.hlsl u_screenSize.y
example_Mandelbrot_ps.hlsl u_screenSize.y
example_Monjori_ps.hlsl u_screenSize.y
example_Twist_ps.hlsl u_screenSize.y
example_LensFlare_ps.hlsl resolution.y
example_Plasma_ps.hlsl u_screenSize.y
shadertoy_FireBall_ps.hlsl u_screenSize.y
shadertoy_Glow_ps.hlsl u_screenSize.y
shadertoy_LensFlare_ps.hlsl u_screenSize.y

Migration Checklist

  • Builtin engine shaders (57 files)
  • cpp-tests shaders (44 GLSL → 50 HLSL)
  • spine shaders (2 files)
  • Live2D shaders (19 files)
  • AXSLCC.cmake: only .hlsl
  • axslcc: only _vs/_ps/_cs stage detection
  • D3D11/D3D12 basic rendering
  • Vulkan basic rendering
  • GLES (ANGLE) basic rendering
  • Metal testing
  • --dxc-reflect DXC reflection extraction
  • solid_circle/solid_capsule SDF shaders (orphan, no test references)

Axmol.dev / Docs / Discord / Reddit

INDEX

GENERAL

MIGRATIONS & UPDATES

FEATURES

BASIC TUTORIALS

ADVANCED TUTORIALS

ABOUT

Clone this wiki locally