Tracking issue for new shader pipeline

[rdb]

Copy-pasting this here from Dropbox Paper so we have everything in one place.

Problem

Currently, we have a single Shader class that encompasses an entire shader pipeline, both handling GLSL and Cg shaders. In the case of GLSL, the shaders are compiled directly by the driver. This has a number of problems:

• We cannot reliably provide preprocessor definitions, an oft-requested feature
• Relies on driver side support for shader language (no Vulkan support for GLSL!)
• Relies on driver to accurately check code, for code to be portable
• Relies on driver to support GLSL version the user happens to be using
• Assumes that a shader will only be used by one particular graphics API
• Can’t know whether a shader compiles until it is used by a GSG

This means that it is currently impossible to use any but the most basic GLSL shaders in Panda3D without having to jump through hoops to make sure that the shaders work on every platform, with every driver, sometimes even needing multiple sets of shaders or custom preprocessing code.

Secondly, there are some issues with the monolithic design of the Shader class itself:

• Shader has a lot of code duplication handling different types of shader stages
• It’s hard to cache individual shader programs
• Using a custom shader generator is not straightforward and requires modifying the source
• There is no interface for passing compilation options
• Cg is interwoven with the base Shader class, so we can’t split out Cg into a separate plug-in, which means it’s hard to exclude proprietary code from user programs without recompiling libpanda.

I propose that we fix these issues by splitting up the shader pipeline into a front-end and a back-end, and introduce an abstraction for the intermediate representation.

Design

Front-end

The front-end ShaderCompiler is what turns shader code into an intermediate representation. We would offer a selection between different front-ends depending on the needs:

• Khronos’ glslang or Google’s libshaderc (which uses glslang) is preferred for GLSL and HLSL, which compiles to SPIR-V.
• Our own GLSL preprocessor can be kept as compatibility fallback if glslang is not enabled. The “compilation” here just means “preprocessing”; no validation can be done with this.
• Cg shaders would use the Cg compiler as front-end. are compiled via glslang's legacy HLSL compilation mode with a special preamble for handling all the library differences.
• For users who want to load SPIR-V directly, we would have a front-end that just validates it.

Using the glslang front-end for GLSL on all platforms ensures that shaders are compiled with the same compiler on all platforms; there are no more issues with some drivers being more lenient than others, or supporting a different set of GLSL versions. We can perform reflection, validation and any transformations on the resulting SPIR-V.

Since glslang supports HLSL, this gets us free support for HLSL, which is currently the most popular shading language in the industry. Since HLSL is very similar to Cg syntactically, we can use this opportunity to to compile Cg shaders through this HLSL with a special preamble to handle any library differences.

The front-end is not dependent on a particular GSG or driver and the compilation happens at shader load time, so if the shader fails to be compiled by the front-end, the application will know about it right away, the same way as with loading a model. Asynchronous compilation could be supported similar to loadModel.

We could introduce an CompilerOptions class (similar to LoaderOptions) to bundle various compilation options, such as search path for #include files, optimization options, and preprocessor definitions.

Intermediate representation

The front-ends produce a per-stage intermediate representation, stored in a ShaderModule class.
We could support several intermediate representations:

• SPIR-V (preferred)
• Preprocessed GLSL code (if glslang is not enabled). Only for compatibility.
• Cg object
• ARB assembly (hypothetical, if needed to support very ancient GPUs, OpenGL only)

The Shader class effectively becomes a container of ShaderModule objects; it does not contain the original source, since by the time that loader.loadShader returns, it will either have the intermediate representation or know that the shader has failed to compile.

This class solves several problems in one go:

• One module corresponds to one shader stage, reducing code duplication currently in Shader
• We can subclass ShaderModule for different implementations, fixing the fact that Shader currently handles all sorts of different languages, and making it possible to move CgShaderModule to a plug-in.
• We can cache the ShaderModule to disk, similar to how we cache loaded .egg files or compressed textures.
• It’s theoretically possible to eg. reuse the same vertex module in different shaders.

ShaderModule could also store reflection information about the inputs and outputs of this module.

ShaderModule is explicitly a serializable class, and we will aggressively cache the compiled shaders in the BamCache, similar to what we do for loaded models. This could happen as .spv files or even as a bam-ified Shader object in the form of an .sho file (cf. .txo for Texture object).

Deployment tools would not need to store the original GLSL source but only ship these intermediate representations, similar to how they already ship .bam instead of .egg. The front-end compiler plug-ins can therefore be excluded from the shipped game!

Back-end

The back-end compiler is what turns the intermediate representation into whatever the graphics API wants.

As for SPIR-V, OpenGL 4.6 and Vulkan can load it natively. In other situations, we would use Khronos’ SPIR-V Cross tool to cross-compile SPIR-V to various formats that can be consumed by the driver or graphics API (GLSL, GLSL ES, MetalSL, HLSL).

This means that in pre-GL 4.6 cases, the pipeline will look like GLSL → SPIR-V → GLSL → machine code, creating two intermediate substeps. However, I think the benefits are worth it, and we can mitigate the extra compilation time with async support and the aforementioned caching. The deployment tools can precompile shaders to SPIR-V to skip the first step.

To Do

• Implement DX9 support
• Get shadows working in DX9 - pending SPIRV-Cross PR:
  HLSL: Support depth comparison texture sampling in SM 2/3. KhronosGroup/SPIRV-Cross#1520
• Issues with marking shadow-compared sampler as depth sampler
• uniform bool does not work due to SPIR-V spec bug (workaround: disable validation for now, or perhaps only if use of OpTypeBool is detected, or automatically convert to float/int)
  Illegal to declare OpTypeBool variables as UniformConstant for OpenGL? KhronosGroup/SPIRV-Registry#72
• Get RenderPipeline to work
  • #pragma include doesn’t work in included shaders
  • glslang struggles with #pragma optionNV, strip out
  • Fix issue finding some includes
  • Macro concatenation issue (can be worked around in RP)
    Macro with concatenation used in #if causing preprocessor error KhronosGroup/glslang#2443
• Fix issue with textureLod with gl-force-glsl-version 120
  GLSL: Fix support for textureLod in legacy vertex shaders KhronosGroup/SPIRV-Cross#1528
• Test that old GLSL 110/120 shaders are still working, including #pragma include
• Fix Cg round() mapping to roundEven(), which isn’t supported by older targets
  HLSL: Support roundEven() in HLSL SM 4.0 and above KhronosGroup/SPIRV-Cross#1524
  GLSL: Provide round/roundEven for legacy GLSL KhronosGroup/SPIRV-Cross#1530
• Test geometry shaders
• Test tessellation shaders
• Test OpenGL ES 2 functioning
• Test HW skinning
• Fix GL_INVALID_OPERATION in default pview with shader without lighting
  • Also happens with gl-support-spirv and pollen
• Take pass over Capabilities and see if there is anything missing, eg.
  • basic shaders (always set if shaders are supported), obviates _supports_basic_shaders
  • depth sample of regular 2d shadow map
  • depth sample of cube shadow map
  • texture sample with LOD (sm3.0)
  • vertex texture fetch
  • other SM3.0/4.0 features?
• Implement instancing in DX9
• Fix bugs with optimize_unused_variables
• Revisit “integer” capability (and whether it rightfully prevents hw skinning from working on GLSL 120)
  • GLSL 1.20 appears to support indexing by casting to int
  • GLSL 1.20 has signed int types (only 16 bit guaranteed, very limited because spec allows them to be implemented as floats) but not unsigned
  • GLSL 1.20 has bool types + vectors as well
  • GLSL 1.30 adds uints and ability to declare attributes and varyings as int
  • GLSL 1.30 also adds int samplers
  • ESSL 1/3 and SM 3/4 situation is the same
  • emulating int varyings requires noperspective, which is GLSL 1.30 / SM4.0
  • Check if SPIRV-Cross automatically translates int attributes, if not, make PR
    -> It only checks and forbids unsigned ints
    GLSL/HLSL: Add legacy handling for int vertex attributes KhronosGroup/SPIRV-Cross#2086
  • Do not assign capability for signed int attributes or variables
  • Do assign capability for unsigned ints, bit ops, etc.
  • Add capabilities for flat interpolation and int samplers
• Add capability checking for gl_PointCoord and gl_PointSize
• Get point sprites working
• Add unit tests for capabilities
• Add CMake build support
• Implement SSBOs and runtime-sized arrays:
  SSBO support in shaderpipeline branch #1190
• Consider moving shader enums to shared base ShaderEnums (like GeomEnums)
• Review of ShaderType interface, ensure it is properly exposed to Python
• Add unit tests for reflection
• Review GSG shader-related query funcs, evaluate whether they still make sense
• Figure out whether rows/columns are swapped around
• Swap around arguments in as_scalar_type?
• Ability to configure compiler options and set defines (without breaking caching):
  Feature request: Programmable shader preprocessor #454
• Figure out what to do about async compilation
  • Return a future or return an uncompiled Shader object which is being compiled in the background?
  • Cg by default sync and glsl async, matching current expectations?
  • Deferred compilation flag to customize behavior?
• Implement caching of compiled SPIR-V modules
• Implement caching of compiled back-end code
• Implement auto injection of skinning and instancing
• Implement shader generator cache #1522
• Test default values of shader inputs
• Write unit tests for restoring shaders and shader inputs from bam files
• Consider whether we need alpha test injection, possibly on SPIRV-Cross side
  TransparencyAttrib::M_dual should only be respected for fixed-function pipeline #852
• Catch format mismatches for image load/store (see Inconsistent compute shader behavior #1576)

[DJs3000]

Nvidia Cg Shader compilator is a closed solution. Need to abandon the CG compiler.

[rdb]

@DJs3000 yes, this is already done, the issue text is outdated... Cg Toolkit is already removed from the shaderpipeline branch and all Cg shaders are compiled via glslang's legacy HLSL compiler instead, which turns out to work well enough as a compatibility solution.

[DJs3000]

There is also a good shader compiler that works on all platforms and architectures.
https://github.com/microsoft/DirectXShaderCompiler
It would be nice to have a group in discord for communication.

[rdb]

Yes, that may be interesting if we want to add support for HLSL shaders, but that's not currently a goal (feature parity first).

Panda3D does already have a Discord server, see here.