Implement vertex discard in shader translator

[benvanik]

//          add_sat oPts.__z_, -r5.wwww, r0.wwww
         src0 = -r[5].wwww;
         src1 = r[0].wwww;
         pv = src0 + src1;
         gl_PointSize.z = clamp(pv.z, 0.0, 1.0);
0(360) : error C1031: swizzle mask element not present in operand "z"

z in a vertex shader is a discard, though that's not easily possible in GLSL. May need to set the vertex position way out of bounds, or set point size to 0 (does that work?).

[hlide]

discard as the GLSL keyword discard; ? oh you mean it might be only possible for a fragment shader. You may be right. The best way is to use a geometry shader to handle discarded vertices. If I recall well, you can do the same thing as in a vertex shader, the only difference is that you need to access an element of an array of vertex instead of a vertex in the source. But I probably overlook the issues you might have.

EDIT: it was a long time ago I didn't use geometry shader. Not using EmitVertex will discard a primitive (the number of vertices depends upon what you're drawing and the generated geometry shader will indeed depend upon the primitive type while your vertex shader won't. So it may not be a right idea to do so).

[benvanik]

In this case, the vertex shader is discarding the vertex, not the fragment shader (the only place 'discard' works) or something that can happen prior to that (like in the geometry shader). It's a dynamic operation computed per vertex, so unless the entire vertex shader is run for every vertex inside of the geometry shader (or all vertex processing is moved to the geometry shader) that won't work :(

The best 'solution' I've found is to offset the position far out of the clip volume and let the clipping system take care of it.

[hlide]

If your solution works fine thanks to the clipping system, there is no reason to do otherwise :).

EDIT: seriously I'm out. Geometry shader is applied after Vertex shader and for some reason I thought it was the inverse.

[benvanik]

Moving the vertex shader to the geometry shader isn't a terrible idea (I
think) - I know I need to switch to custom vertex fetch logic at somepoint,
and maybe I'll do this as well. An unfortunate side effect is that the
shader must be generated for each primitive type it is drawn with, but
modular shader libraries in Vulkan/D3D12 could help with the reuse there.

On Wed, Dec 30, 2015 at 9:41 AM hlide notifications@github.com wrote:

geometry shader is done prior the vertex shader (so the latter can be
applied to the resulting set of vertices issued by the geometry shader). So
it would be a geometry shader doing the real vertex job (and the vertex
shader will only return the vertex as such). But yeah, if your solution
works fine thanks to the clipping system, there is no reason to do
otherwise :).

—
Reply to this email directly or view it on GitHub
#501 (comment).

[hlide]

Never mind I made a mistake and correct the previous post :).

[Triang3l]

Implemented in the Direct3D 12 backend. Using NaN SV_Position to discard if any discarded (seems to be well-defined on the current PC hardware), and SV_CullDistance to discard if all discarded. Banjo-Kazooie: Nuts & Bolts uses this functionality to remove grass that would otherwise be hovering in the air. However, we need to check how discarding should work with quads and point sprites, also depending on some registers (I remember seeing something about "quads as triangles" somewhere). Currently I'm treating them as whole primitives, culled entirely, but this may be wrong.