Skip to content

How OpenGlass renders glass

Jump to bottom
ALTaleX edited this page Aug 25, 2026 · 2 revisions

How OpenGlass renders glass

This page explains how to approach OpenGlass when the target Windows implementation is completely unfamiliar. It is not an offset-extraction guide. The goal is to understand the rendering problem well enough to discover new hook points after Microsoft renames classes, replaces command formats, or moves the compositor to a new architecture.

The implementation changes, but the division of responsibility is stable:

  • uDWM.dll and DirectComposition/Windows.UI.Composition are the control side (the logical client). They decide which windows and regions should have a visual, build the visual/resource graph, and submit updates.
  • dwmcore.dll is the rendering side (the logical server). It receives or realizes those resources, computes dirty and occlusion state, selects a desktop render target, and draws the visual tree.

These are logical roles inside DWM, not necessarily a process or security boundary. To add custom glass, OpenGlass must solve three connected problems:

  1. make the control side submit a durable marker for a glass region;
  2. recognize that marker on the rendering side at a point where the correct geometry and current desktop render target are available;
  3. integrate the custom draw with DWM's redraw and occlusion mechanisms so the result remains correct and efficient.

The end-to-end model

flowchart LR
    A[Window, theme and configuration state] --> B[uDWM / DComp control objects]
    B --> C[Native visual or render resource carrying a glass marker]
    C --> D[Channel command or composition resource update]
    D --> E[dwmcore resource and visual tree]
    E --> F[Dirty-region and occlusion processing]
    F --> G[Draw hook with geometry, transform, Z and current target]
    G --> H[Backdrop copy or sampling]
    H --> I[Blur, colorization, material and reflection]
    I --> J[Selected desktop swap chain]
Loading

The control side can either request Windows' native backdrop effect or submit a marker for a custom dwmcore renderer. If OpenGlass chooses the custom path, a control-side marker alone is insufficient: without render-side recognition it cannot select the exact geometry, current desktop target, or active/maximized state. Skipping dirty or occlusion integration produces stale pixels, clipping, tearing, halos, incorrect thumbnails, or excessive full-screen work.

Native backdrop blur is the simpler option

Windows 10 version 1607, build 14393, already introduced a mature composition backdrop mechanism. A Windows.UI.Composition CompositionBackdropBrush samples the content behind a visual and can feed that backdrop into an effect graph. Gaussian blur, tint, saturation, and similar effects can therefore be assembled entirely through the composition API.

The former OpenGlass dcomp branch and DWMBlurGlass use this general model. It is the pragmatic choice when the goal is to obtain a conventional backdrop blur with minimal private-renderer work. Windows owns the backdrop brush, effect graph, resource scheduling, and most compositor integration, so the implementation is smaller and usually easier to carry to another build.

OpenGlass deliberately chooses the harder custom-renderer path because its goals require more control:

  • Exact input geometry. OpenGlass can restrict backdrop work to the actual window-frame geometry. A composition effect normally processes the backdrop for the visual's boundary rectangle even when only thin frame regions are visible, causing the client/interior area to participate unnecessarily.
  • Custom shaders and effect order. OpenGlass can implement its own blur kernel, Vista/Windows 7 colorization, afterglow, material, reflection, HDR/scRGB handling, and future effects without being limited to the composition effect vocabulary or its ordering rules.
  • Selective redraw and bounded overhead. By integrating with dwmcore dirty-region and occlusion data, OpenGlass redraws only backdrop pixels that can affect visible glass instead of invalidating the visual's complete boundary rectangle. Bounded backdrop copies and reusable per-device resources further avoid work that a general-purpose composition graph must conservatively perform, which is especially important when only narrow frame bands need glass.

The tradeoff is responsibility. A native CompositionBackdropBrush delegates most correctness and lifetime behavior to Windows. A custom dwmcore renderer must correctly handle target selection, resource hazards, color space, redraw, occlusion, device loss, thumbnails, and shutdown itself. If a project only needs ordinary blur, using the composition API is usually the better engineering choice.

The name MILComp does not mean that OpenGlass delegates blur to Windows.UI.Composition. MILComp uses composition visuals as the control-side carrier and for reflection presentation, while the main blur and colorization remain custom rendering performed through dwmcore hooks.

Start from behavior, not names

On an unknown build, do not begin by searching for the old OpenGlass hook names. First reconstruct the native path:

  1. Find the object that owns the non-client background, client blur, accent, or equivalent surface.
  2. Follow how that object creates a visual, brush, geometry, clip, or effect resource.
  3. Follow the resource update across the uDWM/DComp-to-dwmcore boundary.
  4. Identify the server-side object created from that update.
  5. Trace that object through occlusion collection, dirty-region calculation, and final drawing.
  6. Identify the first reliable draw point that has both the marked region and the currently selected desktop render target.

The old class may be gone while the same responsibilities remain. Search for creation, serialization, resource lookup, visual-tree traversal, render-target selection, and final draw behavior. A new architecture is understood when the complete chain is known, not when one promising function has been found.

Problem 1: carry a glass marker from control to render

The marker tells dwmcore, "this native-looking region is not an ordinary solid fill; invoke the OpenGlass renderer here." It must survive the normal DWM update path.

A good marker is:

  • created through native control-side objects, so DWM transports and owns it normally;
  • recognizable after serialization or realization on the server side;
  • extremely unlikely to collide with legitimate Windows content;
  • able to carry the minimum required state, such as active and maximized;
  • bounded to the intended visual or geometry lifetime;
  • harmless if OpenGlass is removed before the resource is redrawn;
  • representable without raw cross-layer pointers or global assumptions.

Possible carriers include:

  • a native brush with a deliberately encoded property;
  • a resource type plus an otherwise-unused but valid state combination;
  • a visual with a distinctive clip or child-resource topology;
  • a command whose server-side realization can be associated with a tracked object;
  • a side table keyed by a native resource whose lifetime is hooked and cleaned up.

Do not invent an invalid command that only happens to pass one build. Prefer a valid native object whose unusual state can be normalized before Windows consumes it in an incompatible way.

The marker is not the glass effect. It is only the protocol between the control-side identification of a region and the render-side replacement draw.

Problem 2: choose a usable render interception point

The render hook must run late enough that DWM has selected the current drawing pass, but early enough that OpenGlass can replace or augment the native draw.

The ideal interception point exposes or can reliably reach:

  • the current CDrawingContext or equivalent;
  • the D2D device context and/or D3D device context;
  • the current device target, render-target view, texture, and shader-resource view;
  • the current desktop render-target information and color space;
  • the geometry or shape being drawn;
  • world and device transforms;
  • clip and visual bounds;
  • current Z/depth and occlusion context;
  • a point inside the active BeginDraw/render pass;
  • a way to suppress the ordinary solid draw or restore DWM state afterward.

A random ID2D1DeviceContext is insufficient. DWM owns multiple devices, off-screen surfaces, thumbnails, snapshots, and intermediate targets. OpenGlass must render against the target selected by the current desktop drawing context. The current implementation verifies the drawing context's device target and obtains the target bitmap or D3D render-target resources from that active path.

The hook must also preserve all D2D/D3D state it changes: transform, primitive blend, antialias mode, target, viewport, shaders, bindings, and resource hazards. Backdrop rendering often needs to read pixels that are also part of the current output; copying or sampling must avoid read/write aliasing and must use bounds derived from the current target.

Problem 3: participate in redraw correctly

Backdrop glass depends on pixels behind the glass. Its output can change even when the glass visual itself did not change. Examples include:

  • a window moving behind a stationary glass frame;
  • activation or maximization changing colorization;
  • the glass window moving, resizing, snapping, or changing DPI;
  • theme, power, transparency, material, or reflection settings changing;
  • desktop, display, device, or swap-chain replacement;
  • a live preview, clone, thumbnail, or transition drawing the visual elsewhere.

The blur kernel reads outside the nominal glass geometry. Dirty bounds therefore need expansion by the actual sampling radius. If DWM invalidates only the original rectangle, edge samples remain stale and appear as seams or trails. Expanding every desktop dirty rectangle unconditionally is correct but wasteful; the goal is to expand only work that can affect visible glass.

A reliable adaptation identifies:

  • where control-side visuals are marked dirty after their state changes;
  • where dwmcore computes optimized dirty rectangles;
  • how the blur expansion maps through world/device transforms;
  • how device loss and target replacement invalidate cached resources;
  • how cloned and off-screen visual trees inherit or reconstruct the marker.

The implementation should produce the smallest dirty region that is still sufficient for every sample read by the glass shader.

Problem 4: participate in occlusion correctly

Normal opaque content lets DWM discard rendering behind it. Glass is different: it is visually in front, but its result depends on the content behind it. If DWM treats a glass region as an ordinary opaque occluder, the backdrop may never be rendered. If OpenGlass disables occlusion globally, correctness improves at the cost of large and persistent GPU/CPU overhead.

The useful model is a depth-aware glass coverage set:

  • record the device-space region and Z/depth of marked glass geometry;
  • preserve background work that can contribute to that region;
  • distinguish full coverage, partial coverage, and no coverage;
  • expand coverage by the blur sampling radius where necessary;
  • prevent small-node or overlay optimizations from skipping required backdrop content;
  • remove coverage when the owning geometry, visual, or occlusion context is destroyed;
  • keep ordinary non-glass occlusion behavior unchanged outside affected regions.

This logic is responsible for more than blur visibility. It prevents clipping at glass boundaries, stale islands during movement, incorrect direct-scanout/overlay decisions, and unnecessary full-screen redraw.

OpenGlass also maintains a glass safety zone for cases where DWM's normal dirty and occlusion decisions are too narrow for backdrop sampling. This is a bounded compatibility mechanism, not a substitute for understanding the native occlusion pipeline.

Legacy implementation: mark a MIL draw command

The Legacy architecture uses the native MIL draw-stream path.

Legacy control side

GlassFrameHandler obtains the native non-client or client-blur geometry in uDWM. It replaces the ordinary instruction with native CDrawGeometryInstruction objects:

  • a CSolidColorLegacyMilBrushProxy identifies the main glass effect;
  • a CImageLegacyMilBrushProxy carries reflection rendering;
  • the native geometry still describes the exact frame or blur region.

The main glass brush encodes a small marker in the floating-point alpha bits through GlassKernel::AlphaChannelReinterpreter. The marker records that the brush is OpenGlass glass plus the active and maximized state. It is transported as an ordinary brush property through the MIL channel. The reflection marker uses an image-brush state that is recognizable in dwmcore—a null image source and empty viewbox—while its other properties carry reflection opacity and viewport.

Relevant code:

Legacy rendering side

CRenderData::TryDrawCommandAsDrawList sees the realized draw-geometry command and its brush/resource types. OpenGlass prevents the marked or potentially marked geometry from disappearing into a cached draw-list fast path, then intercepts the corresponding IDrawingContext::DrawGeometry call.

At that point the renderer can reach:

  • the realized geometry and rectangles;
  • the world transform and current Z;
  • the COcclusionContext;
  • the current D2D context;
  • the current IDeviceTarget or historical render target;
  • the selected target bitmap/texture, render-target view, and shader-resource view.

The solid brush's alpha marker is decoded. A valid marker selects blur/colorization/material rendering; an ordinary brush retains the native solid-color path. Reflection is recognized separately from the image-brush marker. ID2D1DeviceContext::FillGeometry is the final narrow point used to apply the custom realizer while preserving and restoring D2D/D3D state.

Relevant code:

Legacy redraw and occlusion

GlassIntegrity observes marked solid geometry during COcclusionContext processing and records depth-aware glass coverage. It expands dirty inputs according to the blur radius, prevents optimizations from dropping nodes that contribute to glass, and uses bounded safety-zone layers where required. The coverage set is tied to native occlusion objects and cleaned up with their lifetime.

Relevant code:

A useful failed prototype: ShapeVisual plus FillShapeWithBrush

The first MILComp design attempted to replace the non-client background with a Windows.UI.Composition ShapeVisual, assign a special brush, and intercept dwmcore's CDrawingContext::FillShapeWithBrush. This was attractive because the draw function can expose D2D/D3D resources and appears to provide a direct place to replace the shape fill. The investigation is recorded in issue #260.

The prototype was abandoned even though it could produce glass in selected cases:

  1. ShapeVisual had no occlusion implementation. The visual hierarchy exposes a virtual CVisual::CollectOcclusion method, but ShapeVisual does not override it; its vtable therefore reaches the empty base implementation. No shape coverage is contributed to DWM's occlusion pass, so DWM has no correct model of the backdrop-dependent glass region.
  2. Implementing CollectOcclusion required too much private layout knowledge. A replacement would need the actual shape owned by the ShapeVisual in order to submit meaningful occlusion information. Recovering that shape meant walking its private object graph and reading multiple additional members by offset. That expanded the projection surface for one feature and made every future DWM layout change more expensive and dangerous.
  3. FillShapeWithBrush was not a stable current-desktop hook. HDR could select draw-list rendering instead of the pure D2D path, and some passes rendered through an intermediate buffer. Reaching a D2D/D3D context did not prove access to the desktop back buffer selected for the final composition pass.
  4. A visually successful prototype would have hidden an architectural dead end. It optimized for one build's private layout instead of minimizing the assumptions that future builds must preserve.

This is an important counterexample: the hook with the most convenient parameters is not necessarily the best integration point. Count the private fields, vtables, state transitions, and exceptional render paths needed to make the complete design correct. A solution that needs many unrelated offsets to recover its own marker or geometry has poor long-term leverage, even when each offset can be reverse-engineered.

The current MILComp design therefore uses a simpler native rectangle visual plus a distinctive combined-geometry clip, tracks the association at resource update and clip assignment, hooks the ordinary color-brush draw path, and integrates occlusion through visual and brush collection. It still requires private projections, but the projections follow existing compositor responsibilities instead of reconstructing an entire ShapeVisual implementation.

MILComp implementation: mark a composition visual and clip

The build 28000+ architecture no longer offers the Legacy MIL brush pipeline used above. OpenGlass keeps the same client/server idea but selects a carrier native to the newer visual system.

MILComp control side

The uDWM/DComp side creates a native CSolidRectangleVisual inside the non-client background visual collection. Its rectangle and color follow the window state. The visual receives a CCombinedGeometryProxy clip built from the intended glass region and a null second geometry.

During CChannel::CombinedGeometryUpdate, OpenGlass uses the command's GeometryCombineMode field to carry the active/maximized bits associated with that glass geometry. On the server side the status is captured, then the command is patched to a valid union or intersection before native geometry processing. The native realized object therefore remains usable while OpenGlass retains an out-of-band association between that geometry and its glass state.

The distinctive topology is the marker: a sprite/color visual whose clip is the tracked combined geometry. Reflection remains a child Windows.UI.Composition surface visual and uses the underlying DComp visual proxy and compositor rather than pretending the removed Legacy image brush still exists.

Relevant code:

MILComp rendering side

CCombinedGeometry::ProcessUpdate records the marker state and normalizes the native command. CVisual::SetClip recognizes a CSpriteVisual using the combined geometry and tracks it as a glass visual. Object destructors remove the geometry and visual associations.

CColorBrush::Draw is the main interception point. It verifies that the current drawing context is drawing one of the tracked visuals, retrieves its clip geometry, transform, current Z, occlusion context, D2D context, and current device target, and then invokes the same shared glass realizers used by Legacy. Ordinary color brushes and visuals continue through the native draw.

Relevant code:

MILComp redraw and occlusion

Because the carrier is now a visual rather than a Legacy draw command, occlusion is integrated through visual collection and brush hooks:

  • CVisual::CollectOcclusion scopes the currently tracked glass visual;
  • CColorBrush::AddOcclusionInformation records its clip geometry into the glass coverage set;
  • combined-geometry status supplies active/maximized state;
  • COcclusionContext hooks expand dirty regions and preserve backdrop contributors;
  • optimized-rect, overlay, visual-tree, and safety-zone paths prevent narrow native decisions from producing glass artifacts.

The data structures differ from Legacy, but the invariant is identical: mark glass on the control side, recognize it at the server draw, and teach redraw/occlusion that the region is backdrop-dependent rather than ordinarily opaque.

What Legacy and MILComp have in common

Concern Legacy MILComp Stable idea
Control carrier MIL geometry instruction and brush proxy Solid rectangle composition visual and combined-geometry clip Use a native object that survives the control-to-render path
State marker Encoded solid-brush alpha bits Combined-geometry mode captured in a side table Carry only the minimum durable glass state
Server recognition Render command, brush type/state, geometry Tracked sprite visual and clip geometry Recognize the marker before replacing the draw
Main draw hook IDrawingContext::DrawGeometry / D2D fill CColorBrush::Draw / D2D fill Intercept where geometry and current target coexist
Render target Current Legacy drawing-context device target Current MILComp drawing-context device target Render only into the selected desktop pass
Occlusion source Geometry observed during COcclusionContext drawing Visual/brush occlusion collection Track depth-aware backdrop-dependent coverage
Reflection Marked Legacy image brush Windows.UI.Composition surface child Keep auxiliary effects native to the architecture

This table is the practical answer to "how do I port OpenGlass?" Do not port the left or middle column mechanically. Reconstruct the right column in the new compositor.

How to approach a third, unknown architecture

When neither current path survives, look for the following sequence:

A. Find the control-side region owner

Identify the object that knows the non-client/client blur bounds and window state. Confirm how it updates on resize, activation, maximize, DPI, theme, and destruction.

B. Find a native carrier

Create or repurpose a visual/resource that the new control layer already knows how to submit. Prove how its identity, geometry, and minimum state reach the rendering layer. Choose a marker that can be restored to valid native semantics before ordinary processing needs it.

C. Follow realization into dwmcore

Locate the channel/resource update, object constructor, resource table, clip assignment, or visual-tree insertion that realizes the carrier. Hook lifetime boundaries before storing side-table associations.

D. Find the current-target draw

Trace the realized carrier until DWM has selected the desktop drawing context. Reject hook points that expose only a device or an off-screen surface. Require geometry, transforms, Z/occlusion, and the active D2D/D3D target together.

E. Reconstruct dirty and occlusion paths

Find where the new architecture:

  • marks visuals dirty;
  • computes optimized dirty rectangles;
  • collects occlusion and overlays;
  • decides direct scanout or composition;
  • clones visuals for thumbnails and transitions;
  • replaces device and swap-chain resources.

Add glass coverage and blur expansion at the narrowest points that preserve correctness.

F. Integrate shared rendering policy

Only after the carrier and target are reliable should the new architecture call shared colorization, blur, material, and reflection policy. The shared realizer should not need to know how the marker crossed the compositor.

A practical first contribution

For a developer starting with OpenGlass:

  1. Build and debug a supported version first.
  2. Trace one existing glass frame from GlassFrameHandler to GlassRenderer and GlassIntegrity.
  3. Change only a diagnostic or marker observation before changing rendering.
  4. Verify the exact current D2D target, transform, geometry, and Z in the debugger.
  5. Disable the custom draw and prove the native fallback remains intact.
  6. Render a solid debug color only inside the intended marked geometry.
  7. Add backdrop sampling without blur.
  8. Add blur expansion and occlusion coverage.
  9. Add colorization/material/reflection one at a time.
  10. Test movement, resize, occlusion, thumbnails, DPI, device loss, and unload after every stage.

This order separates marker failures, target-selection failures, shader failures, and redraw/occlusion failures. Attempting the complete glass effect immediately makes all four look like the same black rectangle or DWM crash.

Correctness and performance checklist

Before calling a new renderer usable, verify:

  • the marker never collides with ordinary visuals or brushes;
  • active/maximized and other encoded state is updated and cleaned up;
  • the hook runs only for the intended visual and render pass;
  • the current desktop target and color space are known;
  • backdrop copies do not read and write the same resource illegally;
  • every D2D/D3D state mutation is restored;
  • blur sampling bounds are included in dirty and occlusion calculations;
  • translucent glass does not become an opaque occluder;
  • non-glass content keeps native occlusion efficiency;
  • moving content behind stationary glass redraws correctly;
  • moving glass leaves no trails or stale borders;
  • partial occlusion, multiple displays, DPI, RTL, HDR/scRGB, and rounded regions remain correct;
  • thumbnails, Peek, Snap, live preview, lock/unlock, and virtual desktops do not lose the marker;
  • per-device resources are reused and released on device loss;
  • render hot paths avoid network, symbol, registry, and repeated allocation work;
  • shutdown removes hooks, visual associations, marker maps, and GPU resources only after rundown.

Common mistakes

  • Hooking only uDWM and expecting a true backdrop effect.
  • Hooking a D2D function globally without proving which target is current.
  • Choosing a visual carrier whose geometry can only be recovered by projecting a large private object graph.
  • Using a global "current window" without a scoped render/validation lifetime.
  • Marking glass with an invalid native command that later builds reject.
  • Treating every solid brush or transparent visual as glass.
  • Blurring the nominal geometry without expanding dirty bounds.
  • Disabling all occlusion instead of tracking glass coverage.
  • Rendering after DWM has switched to a thumbnail, intermediate, or stale target.
  • Reusing Legacy brushes in an architecture that removed the MIL resource path.
  • Adding a beautiful shader before proving marker, target, redraw, and occlusion correctness.
  • Fixing trails with unconditional full-screen redraws or repeated ForceRender calls.

Related documentation

Clone this wiki locally