Fix line anti-aliasing: edge smoothing, minimum quad width, opaque blending with depth test

[Copilot]

Thin lines in _LineRenderStep partially disappear because the fragment shader always outputs alpha = 1.0 with no edge falloff, and sub-pixel quads vanish when zoom * 0.15 drops below one pixel.

Changes

• LineGS.h: Compute core half-width (max(zoom * 0.15 * 0.5, 0.5)) clamped to a minimum of 0.5px for sub-pixel visibility, then add a 1px AA margin outside the core for smooth edge fadeout. Pass coreEdge (ratio of core to total width) and edgeDist ∈ [-1, 1] to the fragment shader. No alpha reduction for sub-pixel lines — they render at full brightness with clamped minimum width.
• LineFS.h: Apply smoothstep(coreEdge, 1.0, abs(edgeDist)) so the core region stays at full brightness and only the AA margin fades. Uses gl_FragDepth = 1.0 for edge pixels so they don't occupy the depth buffer at normal depth, allowing the subsequent _TriangleRenderStep to overwrite dim line edges via GL_LESS. Core pixels keep gl_FragDepth = gl_FragCoord.z for proper occlusion.
• RenderStep.cpp: Keep opaque blend (GL_ONE, GL_ZERO) with depth testing (GL_LESS), matching _TriangleRenderStep. This prevents flickering caused by additive blending where overlapping line quads accumulate colors inconsistently as draw order changes between frames.

Key geometry shader logic

float coreHalfWidth = max(zoom * 0.15 * 0.5, 0.5);
float aaMargin = 1.0;
float totalHalfWidth = coreHalfWidth + aaMargin;
float coreRatio = coreHalfWidth / totalHalfWidth;

Key fragment shader logic

float edgeAlpha = 1.0 - smoothstep(coreEdge, 1.0, abs(edgeDist));
if (edgeAlpha < 0.99) {
    gl_FragDepth = 1.0;   // edge pixels don't block triangles
} else {
    gl_FragDepth = gl_FragCoord.z;  // core pixels block normally
}
FragColor = vec4(fragColor * edgeAlpha, 1.0);

---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. Learn more about Advanced Security.

[chrxh]

@copilot the lines seems now thinner (too thin) and are thus rendered to dark.

[Copilot]

@copilot the lines seems now thinner (too thin) and are thus rendered to dark.

Fixed in 1a5287a. The previous approach faded the outer 50% of the line quad width, which made lines appear thinner and darker.

The new approach keeps the core line at its original width (zoom * 0.15 * 0.5) and full brightness, then adds a 1px AA margin outside the core for smooth edge fadeout. The fragment shader uses smoothstep(coreEdge, 1.0, abs(edgeDist)) where coreEdge = coreHalfWidth / totalHalfWidth, so the core region remains at full intensity and only the extra margin pixels fade smoothly to zero.

[chrxh]

@copilot it is much better. but there is still a flickering. maybe one needs a z buffer as in the _TriangleRenderStep.

[Copilot]

@copilot it is much better. but there is still a flickering. maybe one needs a z buffer as in the _TriangleRenderStep.

Fixed in 50a30cf. Changed the blend mode from additive (GL_SRC_ALPHA, GL_ONE) to opaque (GL_ONE, GL_ZERO) with depth testing, matching _TriangleRenderStep. The flickering was caused by overlapping line quads additively accumulating colors — as draw order changed between frames, brightness varied. With opaque rendering + depth test, each pixel has a single clean depth-buffer winner. The AA edge fade still works because fragColor * alpha produces dimmer edge pixels that blend naturally against the dark background.

[chrxh]

@copilot at low zoom levels the lines are much too dark.

[chrxh]

@copilot The problem here is that before the LineRenderStep, the TriangleRenderStep is executed. Then the lines overwrite the triangles at the edges. But the boundaries of the lines are darker rendered at low zoom level. Maybe other blending function is needed or the color calculation for the lines is not optimal.

[Copilot]

Fixed in 5c4a7b8. The fragment shader now uses gl_FragDepth = 1.0 for AA edge pixels so they don't occupy the depth buffer at normal depth. This allows the subsequent TriangleRenderStep to overwrite dim line edges via GL_LESS. Core line pixels keep gl_FragDepth = gl_FragCoord.z to maintain proper occlusion. The blending stays opaque (GL_ONE, GL_ZERO).

[Copilot]

@copilot at low zoom levels the lines are much too dark.

Fixed in 5c4a7b8. Two root causes:

1. Sub-pixel alpha reduction was too aggressive — at zoom=1, coreHalfWidth=0.075 produced alpha=0.15, making lines 85% darker. Removed the alpha reduction entirely; sub-pixel lines now clamp to 0.5px half-width at full brightness.

2. Dark AA edge pixels blocked triangles via depth buffer — since lines draw first and triangles second with GL_LESS, dim edge pixels prevented triangles from overwriting. Now AA edge pixels write gl_FragDepth = 1.0 (far plane) so triangles at normal depth can overwrite them. Core pixels keep normal depth.