Skip to content

Scene Renderer

Jump to bottom
CaptSilver edited this page Jun 6, 2026 · 2 revisions

Scene Renderer

Scene wallpapers are drawn by wallpaper-scene-renderer — an open-source Vulkan renderer for Wallpaper Engine wallpapers on Linux. It is pulled in as a Git submodule of this plugin (src/) and built alongside the plugin.

Open source Vulkan scene renderer for Wallpaper Engine wallpapers on Linux.

Core capabilities:

  • Vulkan 1.1
  • Render graph for automatic pass dependencies
  • HLSL-to-GLSL shader translation
  • Async shader compilation with disk cache

The rest of this page is a feature checklist — useful when you're trying to figure out why a particular Workshop wallpaper doesn't look quite right.

Layers

  • ✅ Image layers
    • autosize: true resolves size from the first texture's sprite frame dimensions (or mapWidth/mapHeight for non-sprite pictures).
    • Script-referenced layers stay in the main render graph even when initially invisible (so SceneScript visibility toggles actually render — e.g. dino_run's jump sprite).
    • Dynamic-asset pool for thisScene.createLayer(asset) — pre-allocated pool of hidden scene nodes per engine.registerAsset(path) for both image and particle assets. Burst particle FX rearm via ParticleSubSystem::Reset on each pool pop and auto-hide once all particles die.
  • ✅ Composition / Fullscreen layers
  • ✅ Text (static rasterization via FreeType)
  • ✅ Text (dynamic via SceneScript — clocks, dates, counters)
  • ✅ Sound objects (looping playback via miniaudio)
  • ✅ Particle systems — see Particle System
  • ✅ 3D models — see 3D Models
  • ✅ Light objects (point lights, skylight)

Effects (generic shader pipeline)

Built-in effects work via the same shader pipeline — any effect that compiles to a valid fragment shader will render.

  • ✅ Animation: Scroll, Spin, Shake, Pulse, Water Flow, Water Ripple, Water Waves, Foliage Sway, Swing, Twirl, Cloud Motion, Iris Movement
  • ✅ Blur: Blur, Blur Precise, Motion Blur, Radial Blur
  • ✅ Interactive: Cursor Ripple, Depth Parallax, X-Ray
  • ✅ Color: Blend, Blend Gradient, Chromatic Aberration, Clouds, Color Key, Film Grain, Shimmer, Glitter, Tint, Opacity, VHS, Fire, Light Shafts, Nitro, Reflection, Water Caustics
  • ✅ Distortion: Fisheye, Perspective, Refraction, Skew, Transform
  • ✅ Enhancement: Edge Detection, God Rays, Local Contrast, Shine
  • ❌ Simulation: Advanced Fluid Simulation
  • ✅ ColorBlendMode (effectpassthrough for complex blend modes)
  • ✅ Mouse position with delay (g_PointerPosition)
  • ✅ Parallax (mouse-reactive layer offset)
  • ✅ PBR lighting
  • ✅ Global bloom
  • ✅ Ping-pong FBO pipeline
  • ✅ Per-effect visibility control
  • ✅ HDR-capable effect rendering

Camera

  • ✅ Orthographic (2D scenes)
  • ✅ Perspective (3D scenes)
  • ✅ Zoom
  • ✅ Shake (sum-of-sinusoids)
  • ✅ Path animation (keyframe interpolation)
  • ✅ Parallax (mouse-reactive camera offset)
  • ✅ Fade (transitions between camera paths)

Audio

  • ✅ BGM playback (loop)
  • ✅ Spectrum analysis (KissFFT — 16/32/64 bands, L+R channels)
  • ✅ Shader uniforms (g_AudioSpectrum{16,32,64}{Left,Right}, std140 padded)
  • ✅ System audio capture (PipeWire/PulseAudio monitor source)
  • ✅ Playback tap fallback (when system capture unavailable)
  • ✅ Runtime toggle between system capture and playback tap
  • ✅ Random sound playback (random track selection + inter-track delay)

Particle System

  • ✅ Renderers: Sprite, Rope, Sprite Trail, Rope Trail
  • ✅ Geometry shader renderers: GS Sprite, GS Rope, GS Sprite Trail
  • ✅ Emitters: Box, Sphere — rate-based, burst, one-per-frame, periodic, duration-limited
  • ✅ Initializers: colorrandom, hsvcolorrandom, colorlist, lifetimerandom, sizerandom, alpharandom, velocityrandom, rotationrandom, angularvelocityrandom, turbulentvelocityrandom, positionoffsetrandom, box, sphere, remapinitialvalue, mapsequencebetweencontrolpoints, mapsequencearoundcontrolpoint, inheritcontrolpointvelocity, inheritinitialvaluefromevent
  • ✅ Operators: movement, angularmovement, sizechange, alphafade, alphachange, colorchange, oscillatealpha, oscillatesize, oscillateposition, turbulence, vortex, vortex_v2, controlpointattract, maintaindistancebetweencontrolpoints, reducemovementnearcontrolpoint, remapvalue, controlpointforce, capvelocity, maintaindistancetocontrolpoint, inheritvaluefromevent, boids (Reynolds 1987 flocking)
  • ✅ Collision: collisionplane, collisionsphere, collisionbox / collisionbounds, collisionquad, collisionmodel (with restitution)
  • ✅ Control Points (up to 8) — mouse-follow (link_mouse), worldspace mode, per-instance offset overrides
  • ✅ Child particle systems (static, eventfollow, eventspawn, eventdeath)
  • ✅ Instance overrides (color, alpha, brightness, count, lifetime, rate, speed, size)
  • ✅ Sprite sheet animation (sequential / random frames)
  • ✅ Perspective rendering flag
  • ❌ Real-time particle lighting (sample SceneLights in particle shader)
  • ✅ Audio-responsive emission — bass-band-driven emit-rate scaling (FFT bins 0..3 averaged, exponentially smoothed with fast attack / slow decay, ~0.4× at silence and ~2.0× at full).

Puppet Warp

  • ✅ Multi-bone skeletal animation
  • ✅ Play modes: Loop, Mirror, Single
  • ✅ Animation blending (blend weights)
  • ✅ Bone frame interpolation
  • ✅ Animation layer system (visibility, rate)
  • ✅ Per-vertex skinning (blend indices + weights)
  • ❌ Blend shapes / shape animations
  • ❌ Inverse kinematics
  • ❌ Texture channel animations
  • ❌ Interactive puppet warps (SceneScript-triggered)
  • ❌ 3D perspective extrusion

3D Models

  • ✅ MDL format parsing (v13+, multi-submesh)
  • ✅ Vertex attributes: position, normal, tangent, texcoord, texcoord1, blend indices/weights
  • ✅ Per-submesh materials
  • ✅ Camera path animation
  • ✅ Depth occlusion & transparency
  • ✅ Planar reflection (Y=0 mirror)
  • ✅ Normal mapping
  • ✅ Lightmap UVs (secondary UV channel)
  • ✅ Point lights & skylight
  • ✅ Skybox (spherical environment mapping)
  • ✅ MSAA x4
  • ✅ Anisotropic filtering (16x)
  • ✅ HDR rendering (RGBA16F internal RTs)
  • ✅ HDR texture formats (BC6H, RGBA16F, RG16F, R16F, RGB565, RGBA1010102)
  • ❌ Shadow mapping
  • ❌ HDR bloom post-process
  • ❌ Spot lights
  • ❌ Volumetric lighting

Blend Modes

  • ✅ Translucent (standard alpha)
  • ✅ Translucent_PA (premultiplied alpha)
  • ✅ Additive
  • ✅ Opaque (screen blend for passthrough compose)
  • ✅ Normal
  • ✅ Disable (no blending)
  • ✅ Complex colorBlendMode via effectpassthrough (BlendColor, etc.)

Texture Formats

  • ✅ BC1 (DXT1), BC2 (DXT3), BC3 (DXT5), BC7
  • ✅ BC6H (HDR compressed)
  • ✅ RGBA8, RGB8, RG8, R8
  • ✅ RGBA16F, RG16F, R16F
  • ✅ RGB565, RGBA1010102
  • ✅ TEXB container (v1–v4)
  • ✅ FreeImage formats (PNG, JPEG, TGA, BMP, GIF, DDS, PSD, HDR, EXR, etc.)

User Properties

  • ✅ Default values from project.json
  • ✅ Runtime visibility bindings (conditional show/hide)
  • ✅ Runtime uniform bindings (shader value overrides)
  • ✅ Per-wallpaper config persistence (JSON)
  • ✅ Combo, slider, color, bool, text property types
  • ❌ User-defined custom shaders

Shader Pipeline

  • ✅ HLSL-to-GLSL translation (XSC-based)
  • ✅ GLSL-to-SPIR-V compilation (glslang)
  • ✅ Geometry shader support (VS+GS+FS linked compilation)
  • ✅ Async compilation with disk caching
  • ✅ #define injection (log10, etc.)
  • ✅ Implicit conversion fixes (bool() wraps, cross-stage varying upgrades)
  • ✅ Effect alpha preservation (component-write shaders that omit .a)
  • ✅ flat interpolation qualifier support
  • ✅ 13 texture slots (g_Texture0–g_Texture12) with resolution/rotation/translation metadata

SceneScript

The renderer implements most of Wallpaper Engine's authoring-time JavaScript API.

  • ✅ engine global (IEngine) — registerAudioBuffers, screenResolution, canvasSize, timeOfDay, frametime, runtime, userProperties, setTimeout / setInterval, device detection
  • ✅ thisScene global (IScene)
    • Scene property control (bloom, clear color, camera, lighting)
      • ❌ bloomEnabled runtime toggle (requires render-graph rebuild)
    • getLayer(name), enumerateLayers()
    • createLayer(asset) / destroyLayer(layer) — pool-backed for image and particle assets
    • Camera transforms
  • ✅ thisLayer global (ILayer) — position, angles, scale, visibility, parent/child hierarchy (cycle-checked), animation-layer access
  • ✅ input global — cursor position, cursorEnter / cursorLeave / cursorClick / cursorDown / cursorUp / cursorMove
  • ✅ shared global (inter-script data sharing)
  • ✅ Events: init, update, destroy, applyUserProperties, resizeScreen
  • ✅ Media integration (MPRIS D-Bus → mediaPlaybackChanged, mediaPropertiesChanged, mediaThumbnailChanged, mediaTimelineChanged, mediaStatusChanged)
  • ✅ ILocalStorage (in-memory key-value store)
  • ✅ Text-layer dynamic content
  • ✅ Color scripts (colorScript field parsed, compiled, evaluated at 30 Hz)
  • ✅ WEMath, WEColor, WEVector utility modules
  • ✅ Vec2, Vec3, Vec4 factories (full method sets)
  • ✅ Mat3, Mat4 matrix classes (translation/rotation/scale; view + projection helpers)
  • ✅ engine.colorScheme — Vec3 from the schemecolor user property
  • ✅ engine.openUserShortcut(name) — dispatched to MPRIS and to in-scene scene.on('userShortcut', …) listeners
  • ✅ getEffect(name).visible runtime per-effect toggle (dirty-tracked)
  • ✅ scene.on(event, fn) / scene.off(event[, fn]) — in-scene event bus (10 fixed event points + userShortcut)

Video texture

  • ✅ IVideoTexture — thisLayer.getVideoTexture() returns a proxy with getCurrentTime, setCurrentTime, duration, rate, play, pause, stop, isPlaying (mpv-backed decoder).

Material uniforms

  • ✅ IMaterial — thisLayer.getMaterial().setValue(name, value) / getValue(name); uniform writes go through a render-thread queue into customShader.constValues, dirty flag re-uploads on next frame. Reads are JS-side cached so they never race the renderer.

Sound

  • ✅ Sound layer control (enumerateLayers, play / stop / pause / isPlaying / volume)
  • ✅ console.log support (buffered → LOG_INFO flush)

Timeline Animations

Not yet implemented. Many wallpapers use SceneScript instead, so the gap mostly affects authored-without-script scenes.

  • ❌ Keyframe system with bezier curve interpolation
  • ❌ Animation modes: Loop, Mirror, Single
  • ❌ Animatable properties: origin, angles, scale, color, alpha, etc.
  • ❌ Auto-keyframing
  • ❌ SceneScript animation events

Not yet investigated

  • ❌ RGB hardware lighting integration (OpenRGB / liquidctl)
  • ❌ IParticleSystem / IParticleSystemInstance SceneScript control

Roadmap — unimplemented features

What it would take to finish each unchecked item above, roughly ordered by scope (smallest → largest).

Easy / mostly plumbing

  • bloomEnabled runtime toggle — bloom passes are baked into the render graph at scene load. Cheapest implementation is a "bypass mode" branch in the combine pass; the cleaner path is a graph rebuild on toggle.
  • IParticleSystem / IParticleSystemInstance — expose a stable nodeId → ParticleSubSystem map through __sceneBridge, then thread emission rate / pause / per-instance CP overrides through the existing ParticleModify queue.
  • User-defined customshader user-properties — accept a customshader property type and swap the layer's shader pass at runtime through the async compile cache.
  • RGB hardware lighting — pure plumbing: forward audio-spectrum + dominant-color outputs to OpenRGB / liquidctl over D-Bus.

Medium

  • Real-time particle lighting — particles use a separate sprite/rope shader path that ignores SceneLights. Add per-particle sampling of nearby point lights, broadcast positions via a small uniform array, reuse the PBR shader prelude.
  • Spot lights — extend SceneLight.hpp with cone angle, inner/outer falloff, and direction; add cone culling in shaders.
  • HDR bloom post-process — the combine_hdr variant exists, but the per-mip bloom convolution chain that feeds it isn't built.
  • Puppet: blend shapes — linear morph-target blending. Adds vertex streams for shape deltas and per-shape weight uniforms.
  • Puppet: 3D perspective extrusion — synthesize Z + normals on a 2D puppet so it picks up lighting and parallax.
  • Puppet: texture channel animations — per-bone texcoord offsets, separate from skin matrices.
  • Interactive puppet warps (SceneScript-triggered) — expose Puppet bone manipulation through a thisLayer.getPuppet() JS proxy.

Large

  • Shadow mapping — depth-only render pass per light, cascade for the skylight, PCF/PCSS sampler, integration into the PBR prelude.
  • Volumetric lighting — ray-march from camera through participating media; pairs naturally with shadow mapping.
  • Advanced Fluid Simulation effect — the renderer has no compute pipeline today. Would need VkComputePipeline plumbing, pressure/velocity field textures, and a Navier-Stokes solver split across passes.
  • Puppet: inverse kinematics — FABRIK/CCD per chain. Limited value until interactive puppet warps land.
  • Timeline animations (entire section) — Wallpaper Engine's authoring-time keyframe system. Requires extracting the timeline JSON in WPSceneParser, per-property bezier/Hermite interpolation, loop/mirror/single state machines, and integration with the existing dirty-update queues. Largest single missing system — comparable in scope to SceneScript itself.

License

The scene renderer is GPL-2.0, same as the plugin.

Repo · Scene Renderer · GPL-2.0

Getting Started

Reference

Troubleshooting

Maintenance

About

Clone this wiki locally