Shader Translator

Shader translator

Wallpaper Engine ships its effects and materials as GLSL written in its own dialect. Open Wallpaper Engine translates them to Metal in a compile helper process: GLSL → SPIR-V (glslang) → MSL (SPIRV-Cross), then compiles the MSL with Metal. This page covers the pipeline, variants and combos, the caches, and when to bump ShaderVariantTranslator.revision.

The toolchain

• glslang and SPIRV-Cross are vendored as source in Vendor/ShaderToolchain and built into the app as a local Swift package. Nothing needs installing.
• InProcessShaderCompiler wraps them. glslang's global state isn't thread-safe, so calls are serialized inside the library.
• The app runs it in a long-lived helper process (HelperShaderCompiler, the app's own executable with --shader-compile-helper), so a crash or hang in glslang or SPIRV-Cross costs the helper, not the app. A request whose helper dies is retried once on a new one. Tests translate in process.
• Scripts/check-toolchain-symbols.sh (run by CI) checks the two static libraries for symbol clashes.
• The old command-line fallback compiler was removed.

From WE shader to Metal

Step	Component	What happens
1. Find the source	ShaderSource	Roots are searched in order, so a wallpaper's own copy of a shader wins over the WE assets. Includes are inlined; // [COMBO] {…} declarations and uniform annotations (// {"material":…}) are parsed.
2. Dialect shim	ShaderPrelude	WE writes GLSL with HLSL-isms (mul, frac, saturate, float3, CAST3…). They are mapped with preprocessor macros and helper functions, never text replacement. HLSL's implicit conversions (scalar splats, vector truncation, float %) are applied after preprocessing.
3. Engine-generated code	LightingV1Require, SceneEngineCombos	#require LightingV1 is generated per combo set, as WE generates it; the engine sets LIGHTS_*, shadow, cookie, fog, HDR and similar combos from the scene and the user's settings.
4. Geometry stages	GeometryShaderEmulation, HLSLStageRewrites, ParticleQuadExpansion	Metal has no geometry shaders: WE's .geom stages are folded into the vertex stage and drawn instanced (particles, ropes).
5. Pair rewrite	ShaderPairRewriter	Gives the pair a fixed interface: loose uniforms in one std140 block WEUniforms at buffer 0, g_TextureN at texture/sampler N, varyings matched by name, fixed vertex attribute locations.
6. Translate	InProcessShaderCompiler	glslang → SPIR-V → SPIRV-Cross → MSL, with reflection (attributes, uniform offsets, textures)
7. Build pipelines	Renderer, EffectPipelineArchive	MSL compiled by Metal; pipeline states recorded into a binary archive

ShaderPattern precompiles every regular expression the text passes use, since compiling them per call dominated translation time.

Variants and combos

A WE shader is a family: each [COMBO] is a preprocessor switch (MASK, BLENDMODE, LIGHTING…). A variant is one (vertex, fragment, combo values) combination.

• Variants are compiled lazily, the first time a scene needs one. Scenes use a few hundred of the hundreds of thousands possible.
• Combos come from the material, the effect's pass, user properties and the engine (SceneEngineCombos); defaults come from the shader's own declarations.
• A variant is keyed only on the combos its shaders actually name, so unrelated combos don't multiply the cache.

Caches

Cache	Where	Keyed on
Variant cache	~/Library/Caches/app.openwallpaperengine/shader-variants/<generation>	The inputs, ShaderVariantTranslator.revision and the toolchain fingerprint (compiler, versions, options)
In-memory	Per translator	Same key
Pipeline binary archive	EffectPipelineArchive, in the caches directory	GPU (name and registry ID), OS build, ShaderVariantTranslator.revision and the toolchain, and its own revision (not the app version, so an update keeps it)

• A generation directory holds everything one revision and compiler can produce. A change starts a new directory; other generations unused for a while are deleted (pruneStaleGenerations), so two builds used side by side each keep theirs.
• The binary archive lets a later launch skip the GPU backend compile of every pipeline it has seen. A file Metal refuses is replaced; writes go to a staging file renamed into place; every write serializes a fresh archive; writes are debounced to the end of a compile burst; a failed write isn't retried in the same session (a Metal bug makes retries crash).
• Settings › Diagnostics shows the compiler's versions and the number of translated variants.

Crash and hang protection

• Every compile runs on one dedicated ShaderCompileThread under a watchdog. glslang can't be interrupted, so a compile that overruns abandons the thread; later compiles that session fail fast.
• InProcessCompileCrashGuard records the shader being compiled. If the compiler crashes or hangs on it, that shader is quarantined: later compiles skip it and compile every other one.
• A source the compiler rejects is written to ~/Library/Caches/app.openwallpaperengine/FailedShaders (a folder only the user can read), one file per shader and stage, so the compiler's line numbers can be read against it.
• Failures are logged once in the ShaderTranslator category; no try? hides them.

ShaderVariantTranslator.revision

Important

Bump ShaderVariantTranslator.revision (in Scene/Shaders/ShaderVariant.swift) whenever translated output for the same input can change: prelude, rewriter, conversions, generated code. ShaderVariantCacheTests fails when the output changes without a bump. Bump EffectPipelineArchive.revision when what goes into the archive changes (e.g. descriptor fields).

This is the project's rule 6, "Caches are versioned": see Contributing.

HLSL and limits

• WE's Direct3D-only HLSL sources are not used; the GLSL sources are.
• All loose uniforms of a pair share one buffer (WEUniforms, buffer 0) rather than taking a buffer slot each.

User guide: Settings › Plugins, Permissions, Diagnostics, About